2024-01-21T11:14:43.670 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-01-21T11:14:43.670 DEBUG:teuthology.run:Teuthology command: teuthology --description rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2024-01-19_23:17:02-rados-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7524471 --verbose --name yuriw-2024-01-19_23:17:02-rados-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi -- /home/teuthworker/archive/yuriw-2024-01-19_23:17:02-rados-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7524471/orig.config.yaml 2024-01-21T11:14:43.719 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-21T11:14:43.815 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-01-19_23:17:02-rados-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7524471 branch: wip-yuri4-testing-2024-01-18-1257-pacific description: rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} email: yweinste@redhat.com first_in_suite: false job_id: '7524471' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/pacific v16.2.4 ' name: yuriw-2024-01-19_23:17:02-rados-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri4-testing-2024-01-18-1257-pacific ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri4-testing-2024-01-18-1257-pacific sha1: ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 8186 sha1: ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri4-testing-2024-01-18-1257-pacific suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec targets: smithi029.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAD9H+axVWmNAwZmUhyVHLDor2Zf4cNXo2bmuMxs2KDYnRaC8FwwIcOBWZqZHujp45PlN2c76oZOawacymQMAss= smithi134.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCiE3WmRvczd1PZH1a5QZEFf7ivJikogqRtkcplD6WARPP77b52Dqv/lgqxx2Urh+amBO3Qi1exgJcImtO7Vhnw= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - pexec: clients: - sudo modprobe -r ceph - sudo modprobe ceph disable_send_metrics=on - install: exclude_packages: - ceph-volume tag: v16.2.4 - print: '**** done install task...' - cephadm: avoid_pacific_features: true cephadm_branch: v16.2.4 cephadm_git_url: https://github.com/ceph/ceph image: docker.io/ceph/ceph:v16.2.4 roleless: true - print: '**** done starting v16.2.4' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 1 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay true - cephadm.shell: host.a: - ceph fs set cephfs inline_data true --yes-i-really-really-mean-it - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - kclient: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: cd45576300487d997e5a85abed65500b9f5d143b timestamp: 2024-01-19_23:17:02 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2024-01-21T11:14:43.815 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa; will attempt to use it 2024-01-21T11:14:43.816 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks 2024-01-21T11:14:43.816 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-01-21T11:14:43.818 INFO:teuthology.task.internal:Checking packages... 2024-01-21T11:14:43.841 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec' 2024-01-21T11:14:43.841 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-01-21T11:14:43.841 INFO:teuthology.packaging:ref: None 2024-01-21T11:14:43.841 INFO:teuthology.packaging:tag: None 2024-01-21T11:14:43.842 INFO:teuthology.packaging:branch: wip-yuri4-testing-2024-01-18-1257-pacific 2024-01-21T11:14:43.842 INFO:teuthology.packaging:sha1: ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec 2024-01-21T11:14:43.842 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri4-testing-2024-01-18-1257-pacific 2024-01-21T11:14:44.078 INFO:teuthology.task.internal:Found packages for ceph version 16.2.14-454.gea310589 2024-01-21T11:14:44.082 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-01-21T11:14:44.088 INFO:teuthology.task.internal:no buildpackages task found 2024-01-21T11:14:44.088 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-01-21T11:14:44.108 INFO:teuthology.task.internal:Saving configuration 2024-01-21T11:14:44.127 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-01-21T11:14:44.132 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-21T11:14:44.154 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi029.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-19_23:17:02-rados-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7524471', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-21 11:09:23.441867', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAD9H+axVWmNAwZmUhyVHLDor2Zf4cNXo2bmuMxs2KDYnRaC8FwwIcOBWZqZHujp45PlN2c76oZOawacymQMAss='} 2024-01-21T11:14:44.180 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi134.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-19_23:17:02-rados-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7524471', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-21 11:09:23.439875', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCiE3WmRvczd1PZH1a5QZEFf7ivJikogqRtkcplD6WARPP77b52Dqv/lgqxx2Urh+amBO3Qi1exgJcImtO7Vhnw='} 2024-01-21T11:14:44.180 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-01-21T11:14:44.186 INFO:teuthology.task.internal:roles: ubuntu@smithi029.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2024-01-21T11:14:44.186 INFO:teuthology.task.internal:roles: ubuntu@smithi134.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2024-01-21T11:14:44.186 INFO:teuthology.run_tasks:Running task console_log... 2024-01-21T11:14:44.259 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f276f49b160>, signals=[15]) 2024-01-21T11:14:44.259 INFO:teuthology.run_tasks:Running task internal.connect... 2024-01-21T11:14:44.265 INFO:teuthology.task.internal:Opening connections... 2024-01-21T11:14:44.265 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi029.front.sepia.ceph.com 2024-01-21T11:14:44.267 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T11:14:44.349 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi134.front.sepia.ceph.com 2024-01-21T11:14:44.350 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T11:14:44.422 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-01-21T11:14:44.430 DEBUG:teuthology.orchestra.run.smithi029:> uname -m 2024-01-21T11:14:44.453 INFO:teuthology.orchestra.run.smithi029.stdout:x86_64 2024-01-21T11:14:44.453 DEBUG:teuthology.orchestra.run.smithi029:> cat /etc/os-release 2024-01-21T11:14:44.516 INFO:teuthology.orchestra.run.smithi029.stdout:NAME="CentOS Stream" 2024-01-21T11:14:44.516 INFO:teuthology.orchestra.run.smithi029.stdout:VERSION="8" 2024-01-21T11:14:44.516 INFO:teuthology.orchestra.run.smithi029.stdout:ID="centos" 2024-01-21T11:14:44.516 INFO:teuthology.orchestra.run.smithi029.stdout:ID_LIKE="rhel fedora" 2024-01-21T11:14:44.516 INFO:teuthology.orchestra.run.smithi029.stdout:VERSION_ID="8" 2024-01-21T11:14:44.516 INFO:teuthology.orchestra.run.smithi029.stdout:PLATFORM_ID="platform:el8" 2024-01-21T11:14:44.516 INFO:teuthology.orchestra.run.smithi029.stdout:PRETTY_NAME="CentOS Stream 8" 2024-01-21T11:14:44.516 INFO:teuthology.orchestra.run.smithi029.stdout:ANSI_COLOR="0;31" 2024-01-21T11:14:44.516 INFO:teuthology.orchestra.run.smithi029.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-01-21T11:14:44.516 INFO:teuthology.orchestra.run.smithi029.stdout:HOME_URL="https://centos.org/" 2024-01-21T11:14:44.517 INFO:teuthology.orchestra.run.smithi029.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-21T11:14:44.517 INFO:teuthology.orchestra.run.smithi029.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-01-21T11:14:44.517 INFO:teuthology.orchestra.run.smithi029.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-01-21T11:14:44.517 INFO:teuthology.lock.ops:Updating smithi029.front.sepia.ceph.com on lock server 2024-01-21T11:14:44.544 DEBUG:teuthology.orchestra.run.smithi134:> uname -m 2024-01-21T11:14:44.567 INFO:teuthology.orchestra.run.smithi134.stdout:x86_64 2024-01-21T11:14:44.567 DEBUG:teuthology.orchestra.run.smithi134:> cat /etc/os-release 2024-01-21T11:14:44.628 INFO:teuthology.orchestra.run.smithi134.stdout:NAME="CentOS Stream" 2024-01-21T11:14:44.628 INFO:teuthology.orchestra.run.smithi134.stdout:VERSION="8" 2024-01-21T11:14:44.628 INFO:teuthology.orchestra.run.smithi134.stdout:ID="centos" 2024-01-21T11:14:44.628 INFO:teuthology.orchestra.run.smithi134.stdout:ID_LIKE="rhel fedora" 2024-01-21T11:14:44.628 INFO:teuthology.orchestra.run.smithi134.stdout:VERSION_ID="8" 2024-01-21T11:14:44.628 INFO:teuthology.orchestra.run.smithi134.stdout:PLATFORM_ID="platform:el8" 2024-01-21T11:14:44.629 INFO:teuthology.orchestra.run.smithi134.stdout:PRETTY_NAME="CentOS Stream 8" 2024-01-21T11:14:44.629 INFO:teuthology.orchestra.run.smithi134.stdout:ANSI_COLOR="0;31" 2024-01-21T11:14:44.629 INFO:teuthology.orchestra.run.smithi134.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-01-21T11:14:44.629 INFO:teuthology.orchestra.run.smithi134.stdout:HOME_URL="https://centos.org/" 2024-01-21T11:14:44.629 INFO:teuthology.orchestra.run.smithi134.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-21T11:14:44.629 INFO:teuthology.orchestra.run.smithi134.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-01-21T11:14:44.629 INFO:teuthology.orchestra.run.smithi134.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-01-21T11:14:44.629 INFO:teuthology.lock.ops:Updating smithi134.front.sepia.ceph.com on lock server 2024-01-21T11:14:44.652 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-01-21T11:14:44.661 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-01-21T11:14:44.666 INFO:teuthology.task.internal:Checking for old test directory... 2024-01-21T11:14:44.666 DEBUG:teuthology.orchestra.run.smithi029:> test '!' -e /home/ubuntu/cephtest 2024-01-21T11:14:44.669 DEBUG:teuthology.orchestra.run.smithi134:> test '!' -e /home/ubuntu/cephtest 2024-01-21T11:14:44.689 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-01-21T11:14:44.695 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-01-21T11:14:44.695 DEBUG:teuthology.orchestra.run.smithi029:> test -z $(ls -A /var/lib/ceph) 2024-01-21T11:14:44.732 DEBUG:teuthology.orchestra.run.smithi134:> test -z $(ls -A /var/lib/ceph) 2024-01-21T11:14:44.762 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-01-21T11:14:44.820 INFO:teuthology.run_tasks:Running task kernel... 2024-01-21T11:14:44.835 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-01-21T11:14:44.835 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-01-21T11:14:44.835 DEBUG:teuthology.orchestra.run.smithi029:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-21T11:14:44.836 DEBUG:teuthology.orchestra.run.smithi134:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-21T11:14:44.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-21T11:14:44.856 DEBUG:teuthology.orchestra.run.smithi134:> uname -r 2024-01-21T11:14:44.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-21T11:14:44.857 DEBUG:teuthology.orchestra.run.smithi029:> uname -r 2024-01-21T11:14:44.915 INFO:teuthology.orchestra.run.smithi134.stdout:4.18.0-489.el8.x86_64 2024-01-21T11:14:44.916 INFO:teuthology.task.kernel:Running kernel on smithi134: 4.18.0-489.el8.x86_64 2024-01-21T11:14:44.916 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum install -y kernel 2024-01-21T11:14:44.919 INFO:teuthology.orchestra.run.smithi029.stdout:4.18.0-489.el8.x86_64 2024-01-21T11:14:44.919 INFO:teuthology.task.kernel:Running kernel on smithi029: 4.18.0-489.el8.x86_64 2024-01-21T11:14:44.919 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum install -y kernel 2024-01-21T11:15:55.007 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 8 - AppStream 400 kB/s | 26 MB 01:07 2024-01-21T11:16:05.051 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 8 - AppStream 356 kB/s | 26 MB 01:15 2024-01-21T11:16:42.485 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 8 - BaseOS 246 kB/s | 10 MB 00:42 2024-01-21T11:16:44.510 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 8 - Extras 40 kB/s | 18 kB 00:00 2024-01-21T11:16:45.019 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 8 - Extras common packages 17 kB/s | 6.9 kB 00:00 2024-01-21T11:16:45.303 INFO:teuthology.orchestra.run.smithi134.stdout:Copr repo for python3-asyncssh owned by ceph 18 kB/s | 3.5 kB 00:00 2024-01-21T11:16:46.021 INFO:teuthology.orchestra.run.smithi134.stdout:Extra Packages for Enterprise Linux 26 MB/s | 16 MB 00:00 2024-01-21T11:16:49.721 INFO:teuthology.orchestra.run.smithi134.stdout:lab-extras 554 kB/s | 24 kB 00:00 2024-01-21T11:16:52.551 INFO:teuthology.orchestra.run.smithi134.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-01-21T11:16:52.551 INFO:teuthology.orchestra.run.smithi134.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-01-21T11:16:52.659 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-21T11:16:52.660 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-21T11:16:52.660 INFO:teuthology.orchestra.run.smithi134.stdout: Package Architecture Version Repository Size 2024-01-21T11:16:52.660 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-21T11:16:52.660 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-01-21T11:16:52.660 INFO:teuthology.orchestra.run.smithi134.stdout: kernel x86_64 4.18.0-535.el8 baseos 10 M 2024-01-21T11:16:52.660 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-01-21T11:16:52.661 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-core x86_64 4.18.0-535.el8 baseos 43 M 2024-01-21T11:16:52.661 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-modules x86_64 4.18.0-535.el8 baseos 36 M 2024-01-21T11:16:52.661 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:16:52.661 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-21T11:16:52.661 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-21T11:16:52.661 INFO:teuthology.orchestra.run.smithi134.stdout:Install 3 Packages 2024-01-21T11:16:52.661 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:16:52.661 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 89 M 2024-01-21T11:16:52.662 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 96 M 2024-01-21T11:16:52.662 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-01-21T11:16:56.656 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 8 - BaseOS 226 kB/s | 10 MB 00:46 2024-01-21T11:16:58.807 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 8 - Extras 40 kB/s | 18 kB 00:00 2024-01-21T11:16:59.318 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 8 - Extras common packages 18 kB/s | 6.9 kB 00:00 2024-01-21T11:16:59.654 INFO:teuthology.orchestra.run.smithi029.stdout:Copr repo for python3-asyncssh owned by ceph 14 kB/s | 3.5 kB 00:00 2024-01-21T11:17:01.631 INFO:teuthology.orchestra.run.smithi029.stdout:Extra Packages for Enterprise Linux 8.6 MB/s | 16 MB 00:01 2024-01-21T11:17:05.581 INFO:teuthology.orchestra.run.smithi029.stdout:lab-extras 454 kB/s | 24 kB 00:00 2024-01-21T11:17:08.721 INFO:teuthology.orchestra.run.smithi029.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-01-21T11:17:08.721 INFO:teuthology.orchestra.run.smithi029.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-01-21T11:17:08.859 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-01-21T11:17:08.860 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-21T11:17:08.860 INFO:teuthology.orchestra.run.smithi029.stdout: Package Architecture Version Repository Size 2024-01-21T11:17:08.860 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-21T11:17:08.860 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-01-21T11:17:08.860 INFO:teuthology.orchestra.run.smithi029.stdout: kernel x86_64 4.18.0-535.el8 baseos 10 M 2024-01-21T11:17:08.860 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-01-21T11:17:08.861 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-core x86_64 4.18.0-535.el8 baseos 43 M 2024-01-21T11:17:08.861 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-modules x86_64 4.18.0-535.el8 baseos 36 M 2024-01-21T11:17:08.861 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-21T11:17:08.861 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-01-21T11:17:08.861 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-21T11:17:08.861 INFO:teuthology.orchestra.run.smithi029.stdout:Install 3 Packages 2024-01-21T11:17:08.861 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-21T11:17:08.861 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 89 M 2024-01-21T11:17:08.862 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 96 M 2024-01-21T11:17:08.862 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-01-21T11:18:03.397 INFO:teuthology.orchestra.run.smithi134.stdout:(1/3): kernel-4.18.0-535.el8.x86_64.rpm 148 kB/s | 10 MB 01:10 2024-01-21T11:18:31.381 INFO:teuthology.orchestra.run.smithi029.stdout:(1/3): kernel-4.18.0-535.el8.x86_64.rpm 127 kB/s | 10 MB 01:22 2024-01-21T11:21:21.586 INFO:teuthology.orchestra.run.smithi134.stdout:(2/3): kernel-modules-4.18.0-535.el8.x86_64.rpm 137 kB/s | 36 MB 04:28 2024-01-21T11:22:29.918 INFO:teuthology.orchestra.run.smithi134.stdout:(3/3): kernel-core-4.18.0-535.el8.x86_64.rpm 131 kB/s | 43 MB 05:37 2024-01-21T11:22:29.919 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-21T11:22:29.919 INFO:teuthology.orchestra.run.smithi134.stdout:Total 271 kB/s | 89 MB 05:37 2024-01-21T11:22:30.598 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-01-21T11:22:31.023 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-01-21T11:22:31.023 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-01-21T11:22:32.266 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-01-21T11:22:32.299 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-01-21T11:22:35.079 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-01-21T11:22:35.635 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : kernel-core-4.18.0-535.el8.x86_64 1/3 2024-01-21T11:22:39.226 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-core-4.18.0-535.el8.x86_64 1/3 2024-01-21T11:22:39.601 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : kernel-modules-4.18.0-535.el8.x86_64 2/3 2024-01-21T11:22:43.024 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-modules-4.18.0-535.el8.x86_64 2/3 2024-01-21T11:22:43.315 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : kernel-4.18.0-535.el8.x86_64 3/3 2024-01-21T11:22:52.561 INFO:teuthology.orchestra.run.smithi029.stdout:(2/3): kernel-core-4.18.0-535.el8.x86_64.rpm 128 kB/s | 43 MB 05:43 2024-01-21T11:22:58.253 INFO:teuthology.orchestra.run.smithi029.stdout:(3/3): kernel-modules-4.18.0-535.el8.x86_64.rpm 105 kB/s | 36 MB 05:49 2024-01-21T11:22:58.253 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-01-21T11:22:58.253 INFO:teuthology.orchestra.run.smithi029.stdout:Total 261 kB/s | 89 MB 05:49 2024-01-21T11:22:58.962 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-01-21T11:22:59.444 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-01-21T11:22:59.444 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-01-21T11:23:00.729 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-01-21T11:23:00.769 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-01-21T11:23:03.705 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-01-21T11:23:04.281 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : kernel-core-4.18.0-535.el8.x86_64 1/3 2024-01-21T11:23:05.183 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-core-4.18.0-535.el8.x86_64 3/3 2024-01-21T11:23:05.204 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-modules-4.18.0-535.el8.x86_64 3/3 2024-01-21T11:23:05.555 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-4.18.0-535.el8.x86_64 3/3 2024-01-21T11:23:05.555 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : kernel-4.18.0-535.el8.x86_64 1/3 2024-01-21T11:23:05.555 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : kernel-core-4.18.0-535.el8.x86_64 2/3 2024-01-21T11:23:06.560 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : kernel-modules-4.18.0-535.el8.x86_64 3/3 2024-01-21T11:23:06.560 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:23:06.560 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-01-21T11:23:06.561 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-4.18.0-535.el8.x86_64 kernel-core-4.18.0-535.el8.x86_64 2024-01-21T11:23:06.561 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-modules-4.18.0-535.el8.x86_64 2024-01-21T11:23:06.561 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:23:06.561 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-21T11:23:06.705 DEBUG:teuthology.orchestra.run.smithi134:> rpm -q kernel | sort -rV | head -n 1 2024-01-21T11:23:06.921 INFO:teuthology.orchestra.run.smithi134.stdout:kernel-4.18.0-535.el8.x86_64 2024-01-21T11:23:06.922 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-535.el8.x86_64 2024-01-21T11:23:06.922 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-535.el8.x86_64 2024-01-21T11:23:06.922 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-01-21T11:23:06.922 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-01-21T11:23:06.922 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-01-21T11:23:06.922 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi134.front.sepia.ceph.com, path=None, version=distro) 2024-01-21T11:23:06.922 DEBUG:teuthology.orchestra.run.smithi134:> rpm -q kernel | sort -rV | head -n 1 2024-01-21T11:23:07.131 INFO:teuthology.orchestra.run.smithi134.stdout:kernel-4.18.0-535.el8.x86_64 2024-01-21T11:23:07.132 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-535.el8.x86_64 2024-01-21T11:23:07.132 DEBUG:teuthology.orchestra.run.smithi134:> sudo rpm -qi grub2-tools 2024-01-21T11:23:07.165 INFO:teuthology.orchestra.run.smithi134.stdout:Name : grub2-tools 2024-01-21T11:23:07.165 INFO:teuthology.orchestra.run.smithi134.stdout:Epoch : 1 2024-01-21T11:23:07.165 INFO:teuthology.orchestra.run.smithi134.stdout:Version : 2.02 2024-01-21T11:23:07.165 INFO:teuthology.orchestra.run.smithi134.stdout:Release : 129.el8 2024-01-21T11:23:07.165 INFO:teuthology.orchestra.run.smithi134.stdout:Architecture: x86_64 2024-01-21T11:23:07.166 INFO:teuthology.orchestra.run.smithi134.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-01-21T11:23:07.166 INFO:teuthology.orchestra.run.smithi134.stdout:Group : System Environment/Base 2024-01-21T11:23:07.166 INFO:teuthology.orchestra.run.smithi134.stdout:Size : 9265586 2024-01-21T11:23:07.166 INFO:teuthology.orchestra.run.smithi134.stdout:License : GPLv3+ 2024-01-21T11:23:07.166 INFO:teuthology.orchestra.run.smithi134.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-01-21T11:23:07.166 INFO:teuthology.orchestra.run.smithi134.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-01-21T11:23:07.166 INFO:teuthology.orchestra.run.smithi134.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-01-21T11:23:07.166 INFO:teuthology.orchestra.run.smithi134.stdout:Build Host : kbuilder.bsys.centos.org 2024-01-21T11:23:07.166 INFO:teuthology.orchestra.run.smithi134.stdout:Relocations : (not relocatable) 2024-01-21T11:23:07.166 INFO:teuthology.orchestra.run.smithi134.stdout:Packager : CentOS BuildSystem 2024-01-21T11:23:07.166 INFO:teuthology.orchestra.run.smithi134.stdout:Vendor : CentOS 2024-01-21T11:23:07.166 INFO:teuthology.orchestra.run.smithi134.stdout:URL : http://www.gnu.org/software/grub/ 2024-01-21T11:23:07.166 INFO:teuthology.orchestra.run.smithi134.stdout:Summary : Support tools for GRUB. 2024-01-21T11:23:07.166 INFO:teuthology.orchestra.run.smithi134.stdout:Description : 2024-01-21T11:23:07.166 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:23:07.167 INFO:teuthology.orchestra.run.smithi134.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-01-21T11:23:07.167 INFO:teuthology.orchestra.run.smithi134.stdout:customizable bootloader with modular architecture. It supports a rich 2024-01-21T11:23:07.167 INFO:teuthology.orchestra.run.smithi134.stdout:variety of kernel formats, file systems, computer architectures and 2024-01-21T11:23:07.167 INFO:teuthology.orchestra.run.smithi134.stdout:hardware devices. 2024-01-21T11:23:07.167 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:23:07.167 INFO:teuthology.orchestra.run.smithi134.stdout:This subpackage provides tools for support of all platforms. 2024-01-21T11:23:07.168 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-01-21T11:23:07.168 INFO:teuthology.task.kernel:Updating grub on smithi134 to boot 4.18.0-535.el8.x86_64 2024-01-21T11:23:07.168 DEBUG:teuthology.orchestra.run.smithi134:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-01-21T11:23:08.087 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-core-4.18.0-535.el8.x86_64 1/3 2024-01-21T11:23:08.383 INFO:teuthology.orchestra.run.smithi134.stderr:Generating grub configuration file ... 2024-01-21T11:23:08.639 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : kernel-modules-4.18.0-535.el8.x86_64 2/3 2024-01-21T11:23:10.579 INFO:teuthology.orchestra.run.smithi134.stderr:done 2024-01-21T11:23:10.581 DEBUG:teuthology.orchestra.run.smithi134:> mktemp 2024-01-21T11:23:10.610 INFO:teuthology.orchestra.run.smithi134.stdout:/tmp/tmp.bo8fsxp8yx 2024-01-21T11:23:10.611 DEBUG:teuthology.orchestra.run.smithi134:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.bo8fsxp8yx 2024-01-21T11:23:10.684 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod 0666 /tmp/tmp.bo8fsxp8yx 2024-01-21T11:23:10.828 DEBUG:teuthology.orchestra.remote:smithi134:/tmp/tmp.bo8fsxp8yx is 6KB 2024-01-21T11:23:10.876 DEBUG:teuthology.orchestra.run.smithi134:> rm -fr /tmp/tmp.bo8fsxp8yx 2024-01-21T11:23:10.893 DEBUG:teuthology.orchestra.run.smithi134:> sudo /bin/ls /boot/loader/entries || true 2024-01-21T11:23:10.958 INFO:teuthology.orchestra.run.smithi134.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-01-21T11:23:10.959 INFO:teuthology.orchestra.run.smithi134.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-01-21T11:23:10.959 INFO:teuthology.orchestra.run.smithi134.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-01-21T11:23:10.959 INFO:teuthology.orchestra.run.smithi134.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-535.el8.x86_64.conf 2024-01-21T11:23:10.960 DEBUG:teuthology.orchestra.run.smithi134:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-535.el8.x86_64 2024-01-21T11:23:11.118 DEBUG:teuthology.orchestra.run.smithi134:> sudo shutdown -r now 2024-01-21T11:23:12.314 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-modules-4.18.0-535.el8.x86_64 2/3 2024-01-21T11:23:12.649 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : kernel-4.18.0-535.el8.x86_64 3/3 2024-01-21T11:23:37.045 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-core-4.18.0-535.el8.x86_64 3/3 2024-01-21T11:23:37.069 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-modules-4.18.0-535.el8.x86_64 3/3 2024-01-21T11:23:37.454 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-4.18.0-535.el8.x86_64 3/3 2024-01-21T11:23:37.454 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : kernel-4.18.0-535.el8.x86_64 1/3 2024-01-21T11:23:37.454 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : kernel-core-4.18.0-535.el8.x86_64 2/3 2024-01-21T11:23:37.992 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : kernel-modules-4.18.0-535.el8.x86_64 3/3 2024-01-21T11:23:37.992 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-21T11:23:37.992 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-01-21T11:23:37.992 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-4.18.0-535.el8.x86_64 kernel-core-4.18.0-535.el8.x86_64 2024-01-21T11:23:37.992 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-modules-4.18.0-535.el8.x86_64 2024-01-21T11:23:37.992 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-21T11:23:37.992 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-01-21T11:23:38.147 DEBUG:teuthology.orchestra.run.smithi029:> rpm -q kernel | sort -rV | head -n 1 2024-01-21T11:23:38.373 INFO:teuthology.orchestra.run.smithi029.stdout:kernel-4.18.0-535.el8.x86_64 2024-01-21T11:23:38.373 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-535.el8.x86_64 2024-01-21T11:23:38.373 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-535.el8.x86_64 2024-01-21T11:23:38.373 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-01-21T11:23:38.373 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-01-21T11:23:38.373 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-01-21T11:23:38.374 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi029.front.sepia.ceph.com, path=None, version=distro) 2024-01-21T11:23:38.374 DEBUG:teuthology.orchestra.run.smithi029:> rpm -q kernel | sort -rV | head -n 1 2024-01-21T11:23:38.594 INFO:teuthology.orchestra.run.smithi029.stdout:kernel-4.18.0-535.el8.x86_64 2024-01-21T11:23:38.594 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-535.el8.x86_64 2024-01-21T11:23:38.594 DEBUG:teuthology.orchestra.run.smithi029:> sudo rpm -qi grub2-tools 2024-01-21T11:23:38.629 INFO:teuthology.orchestra.run.smithi029.stdout:Name : grub2-tools 2024-01-21T11:23:38.629 INFO:teuthology.orchestra.run.smithi029.stdout:Epoch : 1 2024-01-21T11:23:38.629 INFO:teuthology.orchestra.run.smithi029.stdout:Version : 2.02 2024-01-21T11:23:38.629 INFO:teuthology.orchestra.run.smithi029.stdout:Release : 129.el8 2024-01-21T11:23:38.629 INFO:teuthology.orchestra.run.smithi029.stdout:Architecture: x86_64 2024-01-21T11:23:38.629 INFO:teuthology.orchestra.run.smithi029.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-01-21T11:23:38.629 INFO:teuthology.orchestra.run.smithi029.stdout:Group : System Environment/Base 2024-01-21T11:23:38.630 INFO:teuthology.orchestra.run.smithi029.stdout:Size : 9265586 2024-01-21T11:23:38.630 INFO:teuthology.orchestra.run.smithi029.stdout:License : GPLv3+ 2024-01-21T11:23:38.630 INFO:teuthology.orchestra.run.smithi029.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-01-21T11:23:38.630 INFO:teuthology.orchestra.run.smithi029.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-01-21T11:23:38.630 INFO:teuthology.orchestra.run.smithi029.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-01-21T11:23:38.630 INFO:teuthology.orchestra.run.smithi029.stdout:Build Host : kbuilder.bsys.centos.org 2024-01-21T11:23:38.630 INFO:teuthology.orchestra.run.smithi029.stdout:Relocations : (not relocatable) 2024-01-21T11:23:38.630 INFO:teuthology.orchestra.run.smithi029.stdout:Packager : CentOS BuildSystem 2024-01-21T11:23:38.630 INFO:teuthology.orchestra.run.smithi029.stdout:Vendor : CentOS 2024-01-21T11:23:38.630 INFO:teuthology.orchestra.run.smithi029.stdout:URL : http://www.gnu.org/software/grub/ 2024-01-21T11:23:38.630 INFO:teuthology.orchestra.run.smithi029.stdout:Summary : Support tools for GRUB. 2024-01-21T11:23:38.630 INFO:teuthology.orchestra.run.smithi029.stdout:Description : 2024-01-21T11:23:38.630 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-21T11:23:38.630 INFO:teuthology.orchestra.run.smithi029.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-01-21T11:23:38.630 INFO:teuthology.orchestra.run.smithi029.stdout:customizable bootloader with modular architecture. It supports a rich 2024-01-21T11:23:38.630 INFO:teuthology.orchestra.run.smithi029.stdout:variety of kernel formats, file systems, computer architectures and 2024-01-21T11:23:38.631 INFO:teuthology.orchestra.run.smithi029.stdout:hardware devices. 2024-01-21T11:23:38.631 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-21T11:23:38.631 INFO:teuthology.orchestra.run.smithi029.stdout:This subpackage provides tools for support of all platforms. 2024-01-21T11:23:38.631 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-01-21T11:23:38.632 INFO:teuthology.task.kernel:Updating grub on smithi029 to boot 4.18.0-535.el8.x86_64 2024-01-21T11:23:38.632 DEBUG:teuthology.orchestra.run.smithi029:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-01-21T11:23:40.531 INFO:teuthology.orchestra.run.smithi029.stderr:Generating grub configuration file ... 2024-01-21T11:23:41.121 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-01-21T11:23:41.121 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-01-21T11:23:41.122 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T11:23:42.810 INFO:teuthology.orchestra.run.smithi029.stderr:done 2024-01-21T11:23:42.811 DEBUG:teuthology.orchestra.run.smithi029:> mktemp 2024-01-21T11:23:42.865 INFO:teuthology.orchestra.run.smithi029.stdout:/tmp/tmp.QPSVW0TYWL 2024-01-21T11:23:42.866 DEBUG:teuthology.orchestra.run.smithi029:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.QPSVW0TYWL 2024-01-21T11:23:42.898 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 0666 /tmp/tmp.QPSVW0TYWL 2024-01-21T11:23:43.051 DEBUG:teuthology.orchestra.remote:smithi029:/tmp/tmp.QPSVW0TYWL is 6KB 2024-01-21T11:23:43.098 DEBUG:teuthology.orchestra.run.smithi029:> rm -fr /tmp/tmp.QPSVW0TYWL 2024-01-21T11:23:43.116 DEBUG:teuthology.orchestra.run.smithi029:> sudo /bin/ls /boot/loader/entries || true 2024-01-21T11:23:43.191 INFO:teuthology.orchestra.run.smithi029.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-01-21T11:23:43.191 INFO:teuthology.orchestra.run.smithi029.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-01-21T11:23:43.191 INFO:teuthology.orchestra.run.smithi029.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-01-21T11:23:43.191 INFO:teuthology.orchestra.run.smithi029.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-535.el8.x86_64.conf 2024-01-21T11:23:43.192 DEBUG:teuthology.orchestra.run.smithi029:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-535.el8.x86_64 2024-01-21T11:23:43.355 DEBUG:teuthology.orchestra.run.smithi029:> sudo shutdown -r now 2024-01-21T11:24:13.379 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-01-21T11:24:13.380 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-01-21T11:24:13.381 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T11:24:41.145 DEBUG:teuthology.orchestra.remote:timed out 2024-01-21T11:24:50.152 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-01-21T11:24:50.153 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T11:24:50.596 DEBUG:teuthology.orchestra.run.smithi134:> true 2024-01-21T11:24:50.917 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-01-21T11:24:50.918 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "4.18.0-535.el8.x86_64"... 2024-01-21T11:24:50.918 DEBUG:teuthology.orchestra.run.smithi134:> uname -r 2024-01-21T11:24:50.935 INFO:teuthology.orchestra.run.smithi134.stdout:4.18.0-535.el8.x86_64 2024-01-21T11:24:50.935 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-535.el8.x86_64 vs 4.18.0-535.el8.x86_64 2024-01-21T11:24:50.935 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-01-21T11:24:50.935 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-01-21T11:24:51.936 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-01-21T11:24:51.937 DEBUG:teuthology.orchestra.run.smithi134:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-01-21T11:24:52.017 INFO:teuthology.orchestra.run.smithi134.stdout:ttyS1 2024-01-21T11:24:52.035 DEBUG:teuthology.parallel:result is None 2024-01-21T11:25:13.383 DEBUG:teuthology.orchestra.remote:timed out 2024-01-21T11:25:22.384 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-01-21T11:25:22.385 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T11:25:22.818 DEBUG:teuthology.orchestra.run.smithi029:> true 2024-01-21T11:25:23.495 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-01-21T11:25:23.496 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "4.18.0-535.el8.x86_64"... 2024-01-21T11:25:23.496 DEBUG:teuthology.orchestra.run.smithi029:> uname -r 2024-01-21T11:25:23.517 INFO:teuthology.orchestra.run.smithi029.stdout:4.18.0-535.el8.x86_64 2024-01-21T11:25:23.517 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-535.el8.x86_64 vs 4.18.0-535.el8.x86_64 2024-01-21T11:25:23.517 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-01-21T11:25:23.517 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-01-21T11:25:24.518 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-01-21T11:25:24.519 DEBUG:teuthology.orchestra.run.smithi029:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-01-21T11:25:24.812 INFO:teuthology.orchestra.run.smithi029.stdout:ttyS1 2024-01-21T11:25:24.826 DEBUG:teuthology.parallel:result is None 2024-01-21T11:25:24.826 INFO:teuthology.run_tasks:Running task internal.base... 2024-01-21T11:25:24.833 INFO:teuthology.task.internal:Creating test directory... 2024-01-21T11:25:24.834 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-21T11:25:24.836 DEBUG:teuthology.orchestra.run.smithi134:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-21T11:25:24.861 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-01-21T11:25:24.867 INFO:teuthology.run_tasks:Running task internal.archive... 2024-01-21T11:25:24.875 INFO:teuthology.task.internal:Creating archive directory... 2024-01-21T11:25:24.875 DEBUG:teuthology.orchestra.run.smithi029:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-21T11:25:24.893 DEBUG:teuthology.orchestra.run.smithi134:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-21T11:25:24.956 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-01-21T11:25:24.962 INFO:teuthology.task.internal:Enabling coredump saving... 2024-01-21T11:25:24.962 DEBUG:teuthology.orchestra.run.smithi029:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-21T11:25:24.964 DEBUG:teuthology.orchestra.run.smithi134:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-21T11:25:25.000 INFO:teuthology.orchestra.run.smithi029.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-21T11:25:25.010 INFO:teuthology.orchestra.run.smithi029.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-21T11:25:25.013 INFO:teuthology.orchestra.run.smithi134.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-21T11:25:25.024 INFO:teuthology.orchestra.run.smithi134.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-21T11:25:25.026 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-01-21T11:25:25.032 INFO:teuthology.task.internal:Configuring sudo... 2024-01-21T11:25:25.032 DEBUG:teuthology.orchestra.run.smithi029:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-21T11:25:25.054 DEBUG:teuthology.orchestra.run.smithi134:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-21T11:25:25.083 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-01-21T11:25:25.091 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-01-21T11:25:25.091 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-21T11:25:25.123 DEBUG:teuthology.orchestra.run.smithi134:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-21T11:25:25.142 DEBUG:teuthology.orchestra.run.smithi029:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-21T11:25:25.222 DEBUG:teuthology.orchestra.run.smithi029:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-21T11:25:25.284 DEBUG:teuthology.orchestra.run.smithi029:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-21T11:25:25.370 DEBUG:teuthology.orchestra.run.smithi029:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-21T11:25:25.414 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-01-21T11:25:25.414 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-21T11:25:25.500 DEBUG:teuthology.orchestra.run.smithi134:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-21T11:25:25.547 DEBUG:teuthology.orchestra.run.smithi134:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-21T11:25:25.609 DEBUG:teuthology.orchestra.run.smithi134:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-21T11:25:25.656 DEBUG:teuthology.orchestra.run.smithi134:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-21T11:25:25.700 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-01-21T11:25:25.700 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-21T11:25:25.768 DEBUG:teuthology.orchestra.run.smithi029:> sudo service rsyslog restart 2024-01-21T11:25:25.770 DEBUG:teuthology.orchestra.run.smithi134:> sudo service rsyslog restart 2024-01-21T11:25:25.860 INFO:teuthology.orchestra.run.smithi029.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-21T11:25:25.871 INFO:teuthology.orchestra.run.smithi134.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-21T11:25:26.067 INFO:teuthology.run_tasks:Running task internal.timer... 2024-01-21T11:25:26.074 INFO:teuthology.task.internal:Starting timer... 2024-01-21T11:25:26.074 INFO:teuthology.run_tasks:Running task pcp... 2024-01-21T11:25:26.087 INFO:teuthology.run_tasks:Running task selinux... 2024-01-21T11:25:26.095 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-01-21T11:25:26.096 DEBUG:teuthology.orchestra.run.smithi029:> sudo service auditd rotate 2024-01-21T11:25:26.219 INFO:teuthology.orchestra.run.smithi029.stdout:Rotating logs: 2024-01-21T11:25:26.221 DEBUG:teuthology.orchestra.run.smithi134:> sudo service auditd rotate 2024-01-21T11:25:26.283 INFO:teuthology.orchestra.run.smithi134.stdout:Rotating logs: 2024-01-21T11:25:26.285 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-01-21T11:25:26.285 DEBUG:teuthology.orchestra.run.smithi029:> /usr/sbin/getenforce 2024-01-21T11:25:26.327 INFO:teuthology.orchestra.run.smithi029.stdout:Permissive 2024-01-21T11:25:26.328 DEBUG:teuthology.orchestra.run.smithi134:> /usr/sbin/getenforce 2024-01-21T11:25:26.378 INFO:teuthology.orchestra.run.smithi134.stdout:Permissive 2024-01-21T11:25:26.378 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi029.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi134.front.sepia.ceph.com': 'permissive'} 2024-01-21T11:25:26.378 DEBUG:teuthology.orchestra.run.smithi029:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-21T11:25:26.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-21T11:25:26.406 DEBUG:teuthology.orchestra.run.smithi134:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-21T11:25:26.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-21T11:25:26.433 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-01-21T11:25:26.433 DEBUG:teuthology.orchestra.run.smithi029:> sudo /usr/sbin/setenforce permissive 2024-01-21T11:25:26.476 DEBUG:teuthology.orchestra.run.smithi134:> sudo /usr/sbin/setenforce permissive 2024-01-21T11:25:26.504 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-01-21T11:25:26.513 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-01-21T11:25:26.528 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-01-21T11:25:26.529 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi029.front.sepia.ceph.com,smithi134.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-01-21T11:34:31.255 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi029.front.sepia.ceph.com'), Remote(name='ubuntu@smithi134.front.sepia.ceph.com')] 2024-01-21T11:34:31.257 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-01-21T11:34:31.258 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T11:34:31.334 DEBUG:teuthology.orchestra.run.smithi029:> true 2024-01-21T11:34:31.402 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-01-21T11:34:31.402 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-01-21T11:34:31.403 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T11:34:31.473 DEBUG:teuthology.orchestra.run.smithi134:> true 2024-01-21T11:34:31.543 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-01-21T11:34:31.543 INFO:teuthology.run_tasks:Running task clock... 2024-01-21T11:34:31.553 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-01-21T11:34:31.554 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-21T11:34:31.554 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-21T11:34:31.556 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-21T11:34:31.556 DEBUG:teuthology.orchestra.run.smithi134:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-21T11:34:31.587 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-01-21T11:34:31.601 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-01-21T11:34:31.618 INFO:teuthology.orchestra.run.smithi134.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-01-21T11:34:31.632 INFO:teuthology.orchestra.run.smithi134.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-01-21T11:34:31.672 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: ntpd: command not found 2024-01-21T11:34:31.683 INFO:teuthology.orchestra.run.smithi029.stdout:506 Cannot talk to daemon 2024-01-21T11:34:31.695 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: ntpd: command not found 2024-01-21T11:34:31.697 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-01-21T11:34:31.706 INFO:teuthology.orchestra.run.smithi134.stdout:506 Cannot talk to daemon 2024-01-21T11:34:31.711 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-01-21T11:34:31.720 INFO:teuthology.orchestra.run.smithi134.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-01-21T11:34:31.733 INFO:teuthology.orchestra.run.smithi134.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-01-21T11:34:31.779 INFO:teuthology.orchestra.run.smithi029.stderr:bash: ntpq: command not found 2024-01-21T11:34:31.783 INFO:teuthology.orchestra.run.smithi029.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-21T11:34:31.783 INFO:teuthology.orchestra.run.smithi029.stdout:=============================================================================== 2024-01-21T11:34:31.783 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T11:34:31.783 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T11:34:31.783 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T11:34:31.784 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T11:34:31.813 INFO:teuthology.orchestra.run.smithi134.stderr:bash: ntpq: command not found 2024-01-21T11:34:31.817 INFO:teuthology.orchestra.run.smithi134.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-21T11:34:31.817 INFO:teuthology.orchestra.run.smithi134.stdout:=============================================================================== 2024-01-21T11:34:31.817 INFO:teuthology.orchestra.run.smithi134.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T11:34:31.817 INFO:teuthology.orchestra.run.smithi134.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T11:34:31.817 INFO:teuthology.orchestra.run.smithi134.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T11:34:31.817 INFO:teuthology.orchestra.run.smithi134.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T11:34:31.818 INFO:teuthology.run_tasks:Running task pexec... 2024-01-21T11:34:31.827 INFO:teuthology.task.pexec:Executing custom commands... 2024-01-21T11:34:31.827 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi029.front.sepia.ceph.com 2024-01-21T11:34:31.828 DEBUG:teuthology.orchestra.run.smithi029:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-01-21T11:34:31.828 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi134.front.sepia.ceph.com 2024-01-21T11:34:31.828 DEBUG:teuthology.orchestra.run.smithi134:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-01-21T11:34:32.616 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:02:27 ago on Sun 21 Jan 2024 11:32:05 AM UTC. 2024-01-21T11:34:32.616 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:39 ago on Sun 21 Jan 2024 11:31:53 AM UTC. 2024-01-21T11:34:34.568 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-21T11:34:34.569 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-21T11:34:34.569 INFO:teuthology.orchestra.run.smithi134.stdout: Package Architecture Version Repository Size 2024-01-21T11:34:34.569 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-21T11:34:34.569 INFO:teuthology.orchestra.run.smithi134.stdout:Resetting modules: 2024-01-21T11:34:34.569 INFO:teuthology.orchestra.run.smithi134.stdout: container-tools 2024-01-21T11:34:34.569 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:34:34.569 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-21T11:34:34.569 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-21T11:34:34.569 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:34:34.699 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-01-21T11:34:34.700 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-21T11:34:34.700 INFO:teuthology.orchestra.run.smithi029.stdout: Package Architecture Version Repository Size 2024-01-21T11:34:34.700 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-21T11:34:34.700 INFO:teuthology.orchestra.run.smithi029.stdout:Resetting modules: 2024-01-21T11:34:34.700 INFO:teuthology.orchestra.run.smithi029.stdout: container-tools 2024-01-21T11:34:34.700 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-21T11:34:34.700 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-01-21T11:34:34.701 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-21T11:34:34.701 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-21T11:34:34.832 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-21T11:34:34.965 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-01-21T11:34:35.656 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:42 ago on Sun 21 Jan 2024 11:31:53 AM UTC. 2024-01-21T11:34:35.816 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:02:30 ago on Sun 21 Jan 2024 11:32:05 AM UTC. 2024-01-21T11:34:37.613 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-21T11:34:37.647 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-21T11:34:37.707 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-01-21T11:34:37.707 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-21T11:34:37.707 INFO:teuthology.orchestra.run.smithi134.stdout:Installing group/module packages: 2024-01-21T11:34:37.708 INFO:teuthology.orchestra.run.smithi134.stdout: aardvark-dns x86_64 2:1.8.0-1.module_el8+712+4cd1bd69 appstream 1.0 M 2024-01-21T11:34:37.708 INFO:teuthology.orchestra.run.smithi134.stdout: buildah x86_64 1:1.32.0-1.module_el8+712+4cd1bd69 appstream 9.6 M 2024-01-21T11:34:37.708 INFO:teuthology.orchestra.run.smithi134.stdout: cockpit-podman noarch 78-1.module_el8+712+4cd1bd69 appstream 750 k 2024-01-21T11:34:37.708 INFO:teuthology.orchestra.run.smithi134.stdout: crun x86_64 1.9.2-1.module_el8+712+4cd1bd69 appstream 241 k 2024-01-21T11:34:37.708 INFO:teuthology.orchestra.run.smithi134.stdout: netavark x86_64 2:1.8.0-2.module_el8+712+4cd1bd69 appstream 3.9 M 2024-01-21T11:34:37.708 INFO:teuthology.orchestra.run.smithi134.stdout: python3-podman noarch 4.7.0-1.module_el8+712+4cd1bd69 appstream 178 k 2024-01-21T11:34:37.708 INFO:teuthology.orchestra.run.smithi134.stdout: skopeo x86_64 2:1.13.3-1.module_el8+712+4cd1bd69 appstream 8.1 M 2024-01-21T11:34:37.708 INFO:teuthology.orchestra.run.smithi134.stdout: toolbox x86_64 0.0.99.4-6.module_el8+712+4cd1bd69 appstream 2.5 M 2024-01-21T11:34:37.708 INFO:teuthology.orchestra.run.smithi134.stdout: udica noarch 0.2.6-20.module_el8+712+4cd1bd69 appstream 53 k 2024-01-21T11:34:37.708 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-01-21T11:34:37.708 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-01-21T11:34:37.708 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-01-21T11:34:37.708 INFO:teuthology.orchestra.run.smithi134.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-01-21T11:34:37.708 INFO:teuthology.orchestra.run.smithi134.stdout:Installing weak dependencies: 2024-01-21T11:34:37.709 INFO:teuthology.orchestra.run.smithi134.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-01-21T11:34:37.709 INFO:teuthology.orchestra.run.smithi134.stdout:Installing module profiles: 2024-01-21T11:34:37.709 INFO:teuthology.orchestra.run.smithi134.stdout: container-tools/common 2024-01-21T11:34:37.709 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:34:37.709 INFO:teuthology.orchestra.run.smithi134.stdout:Enabling module streams: 2024-01-21T11:34:37.709 INFO:teuthology.orchestra.run.smithi134.stdout: container-tools rhel8 2024-01-21T11:34:37.709 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:34:37.709 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-21T11:34:37.709 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-21T11:34:37.709 INFO:teuthology.orchestra.run.smithi134.stdout:Install 13 Packages 2024-01-21T11:34:37.709 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:34:37.710 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 27 M 2024-01-21T11:34:37.710 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 89 M 2024-01-21T11:34:37.710 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-01-21T11:34:37.889 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-01-21T11:34:37.926 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-21T11:34:37.926 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2024-01-21T11:34:37.926 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-21T11:34:37.927 INFO:teuthology.orchestra.run.smithi029.stdout:Installing group/module packages: 2024-01-21T11:34:37.927 INFO:teuthology.orchestra.run.smithi029.stdout: aardvark-dns x86_64 2:1.8.0-1.module_el8+712+4cd1bd69 appstream 1.0 M 2024-01-21T11:34:37.927 INFO:teuthology.orchestra.run.smithi029.stdout: buildah x86_64 1:1.32.0-1.module_el8+712+4cd1bd69 appstream 9.6 M 2024-01-21T11:34:37.927 INFO:teuthology.orchestra.run.smithi029.stdout: cockpit-podman noarch 78-1.module_el8+712+4cd1bd69 appstream 750 k 2024-01-21T11:34:37.927 INFO:teuthology.orchestra.run.smithi029.stdout: crun x86_64 1.9.2-1.module_el8+712+4cd1bd69 appstream 241 k 2024-01-21T11:34:37.927 INFO:teuthology.orchestra.run.smithi029.stdout: netavark x86_64 2:1.8.0-2.module_el8+712+4cd1bd69 appstream 3.9 M 2024-01-21T11:34:37.927 INFO:teuthology.orchestra.run.smithi029.stdout: python3-podman noarch 4.7.0-1.module_el8+712+4cd1bd69 appstream 178 k 2024-01-21T11:34:37.927 INFO:teuthology.orchestra.run.smithi029.stdout: skopeo x86_64 2:1.13.3-1.module_el8+712+4cd1bd69 appstream 8.1 M 2024-01-21T11:34:37.927 INFO:teuthology.orchestra.run.smithi029.stdout: toolbox x86_64 0.0.99.4-6.module_el8+712+4cd1bd69 appstream 2.5 M 2024-01-21T11:34:37.927 INFO:teuthology.orchestra.run.smithi029.stdout: udica noarch 0.2.6-20.module_el8+712+4cd1bd69 appstream 53 k 2024-01-21T11:34:37.927 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-01-21T11:34:37.927 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-01-21T11:34:37.927 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-01-21T11:34:37.927 INFO:teuthology.orchestra.run.smithi029.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-01-21T11:34:37.927 INFO:teuthology.orchestra.run.smithi029.stdout:Installing weak dependencies: 2024-01-21T11:34:37.928 INFO:teuthology.orchestra.run.smithi029.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-01-21T11:34:37.928 INFO:teuthology.orchestra.run.smithi029.stdout:Installing module profiles: 2024-01-21T11:34:37.928 INFO:teuthology.orchestra.run.smithi029.stdout: container-tools/common 2024-01-21T11:34:37.928 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-21T11:34:37.928 INFO:teuthology.orchestra.run.smithi029.stdout:Enabling module streams: 2024-01-21T11:34:37.928 INFO:teuthology.orchestra.run.smithi029.stdout: container-tools rhel8 2024-01-21T11:34:37.928 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-21T11:34:37.928 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-01-21T11:34:37.928 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-21T11:34:37.928 INFO:teuthology.orchestra.run.smithi029.stdout:Install 13 Packages 2024-01-21T11:34:37.928 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-21T11:34:37.929 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 27 M 2024-01-21T11:34:37.929 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 89 M 2024-01-21T11:34:37.929 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-01-21T11:34:40.462 INFO:teuthology.orchestra.run.smithi134.stdout:(1/13): cockpit-podman-78-1.module_el8+712+4cd1 271 kB/s | 750 kB 00:02 2024-01-21T11:34:40.679 INFO:teuthology.orchestra.run.smithi134.stdout:(2/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 175 kB/s | 38 kB 00:00 2024-01-21T11:34:40.684 INFO:teuthology.orchestra.run.smithi029.stdout:(1/13): aardvark-dns-1.8.0-1.module_el8+712+4cd 399 kB/s | 1.0 MB 00:02 2024-01-21T11:34:40.834 INFO:teuthology.orchestra.run.smithi029.stdout:(2/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 252 kB/s | 38 kB 00:00 2024-01-21T11:34:40.940 INFO:teuthology.orchestra.run.smithi134.stdout:(3/13): aardvark-dns-1.8.0-1.module_el8+712+4cd 316 kB/s | 1.0 MB 00:03 2024-01-21T11:34:41.368 INFO:teuthology.orchestra.run.smithi029.stdout:(3/13): crun-1.9.2-1.module_el8+712+4cd1bd69.x8 451 kB/s | 241 kB 00:00 2024-01-21T11:34:41.477 INFO:teuthology.orchestra.run.smithi029.stdout:(4/13): cockpit-podman-78-1.module_el8+712+4cd1 223 kB/s | 750 kB 00:03 2024-01-21T11:34:41.896 INFO:teuthology.orchestra.run.smithi134.stdout:(4/13): crun-1.9.2-1.module_el8+712+4cd1bd69.x8 198 kB/s | 241 kB 00:01 2024-01-21T11:34:42.336 INFO:teuthology.orchestra.run.smithi029.stdout:(5/13): python3-podman-4.7.0-1.module_el8+712+4 207 kB/s | 178 kB 00:00 2024-01-21T11:34:42.495 INFO:teuthology.orchestra.run.smithi029.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 156 kB/s | 25 kB 00:00 2024-01-21T11:34:42.747 INFO:teuthology.orchestra.run.smithi134.stdout:(5/13): python3-podman-4.7.0-1.module_el8+712+4 209 kB/s | 178 kB 00:00 2024-01-21T11:34:42.845 INFO:teuthology.orchestra.run.smithi029.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 268 kB/s | 94 kB 00:00 2024-01-21T11:34:42.897 INFO:teuthology.orchestra.run.smithi134.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 165 kB/s | 25 kB 00:00 2024-01-21T11:34:43.355 INFO:teuthology.orchestra.run.smithi134.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 205 kB/s | 94 kB 00:00 2024-01-21T11:34:53.563 INFO:teuthology.orchestra.run.smithi134.stdout:(8/13): netavark-1.8.0-2.module_el8+712+4cd1bd6 318 kB/s | 3.9 MB 00:12 2024-01-21T11:34:54.490 INFO:teuthology.orchestra.run.smithi029.stdout:(8/13): netavark-1.8.0-2.module_el8+712+4cd1bd6 306 kB/s | 3.9 MB 00:13 2024-01-21T11:35:02.388 INFO:teuthology.orchestra.run.smithi134.stdout:(9/13): toolbox-0.0.99.4-6.module_el8+712+4cd1b 294 kB/s | 2.5 MB 00:08 2024-01-21T11:35:02.615 INFO:teuthology.orchestra.run.smithi134.stdout:(10/13): udica-0.2.6-20.module_el8+712+4cd1bd69 234 kB/s | 53 kB 00:00 2024-01-21T11:35:02.788 INFO:teuthology.orchestra.run.smithi134.stdout:(11/13): yajl-2.1.0-12.el8.x86_64.rpm 234 kB/s | 41 kB 00:00 2024-01-21T11:35:05.890 INFO:teuthology.orchestra.run.smithi029.stdout:(9/13): toolbox-0.0.99.4-6.module_el8+712+4cd1b 227 kB/s | 2.5 MB 00:11 2024-01-21T11:35:06.174 INFO:teuthology.orchestra.run.smithi029.stdout:(10/13): udica-0.2.6-20.module_el8+712+4cd1bd69 186 kB/s | 53 kB 00:00 2024-01-21T11:35:06.400 INFO:teuthology.orchestra.run.smithi029.stdout:(11/13): yajl-2.1.0-12.el8.x86_64.rpm 182 kB/s | 41 kB 00:00 2024-01-21T11:35:08.692 INFO:teuthology.orchestra.run.smithi029.stdout:(12/13): skopeo-1.13.3-1.module_el8+712+4cd1bd6 323 kB/s | 8.1 MB 00:25 2024-01-21T11:35:09.571 INFO:teuthology.orchestra.run.smithi134.stdout:(12/13): buildah-1.32.0-1.module_el8+712+4cd1bd 309 kB/s | 9.6 MB 00:31 2024-01-21T11:35:10.371 INFO:teuthology.orchestra.run.smithi134.stdout:(13/13): skopeo-1.13.3-1.module_el8+712+4cd1bd6 309 kB/s | 8.1 MB 00:27 2024-01-21T11:35:10.372 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-21T11:35:10.372 INFO:teuthology.orchestra.run.smithi134.stdout:Total 832 kB/s | 27 MB 00:32 2024-01-21T11:35:10.532 INFO:teuthology.orchestra.run.smithi029.stdout:(13/13): buildah-1.32.0-1.module_el8+712+4cd1bd 303 kB/s | 9.6 MB 00:32 2024-01-21T11:35:10.534 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-01-21T11:35:10.534 INFO:teuthology.orchestra.run.smithi029.stdout:Total 835 kB/s | 27 MB 00:32 2024-01-21T11:35:10.637 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-01-21T11:35:10.658 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-01-21T11:35:10.658 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-01-21T11:35:10.812 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-01-21T11:35:10.834 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-01-21T11:35:10.834 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-01-21T11:35:10.920 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-01-21T11:35:10.923 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-01-21T11:35:11.111 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-01-21T11:35:11.114 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-01-21T11:35:11.615 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-01-21T11:35:11.794 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-01-21T11:35:11.831 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-01-21T11:35:12.012 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-01-21T11:35:12.035 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2024-01-21T11:35:12.173 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2024-01-21T11:35:12.229 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2024-01-21T11:35:12.387 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2024-01-21T11:35:12.453 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-01-21T11:35:12.622 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-01-21T11:35:13.045 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 5/13 2024-01-21T11:35:13.202 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 5/13 2024-01-21T11:35:14.034 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2024-01-21T11:35:14.239 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2024-01-21T11:35:14.264 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 7/13 2024-01-21T11:35:14.450 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 8/13 2024-01-21T11:35:14.490 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 7/13 2024-01-21T11:35:14.680 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 9/13 2024-01-21T11:35:14.711 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 8/13 2024-01-21T11:35:14.961 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 9/13 2024-01-21T11:35:15.101 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 10/13 2024-01-21T11:35:15.402 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 10/13 2024-01-21T11:35:15.977 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2024-01-21T11:35:16.214 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 12/13 2024-01-21T11:35:16.336 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2024-01-21T11:35:16.392 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2024-01-21T11:35:16.565 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 12/13 2024-01-21T11:35:16.763 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2024-01-21T11:35:16.812 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2024-01-21T11:35:16.812 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 1/13 2024-01-21T11:35:16.812 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2/13 2024-01-21T11:35:16.812 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 3/13 2024-01-21T11:35:16.812 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-01-21T11:35:16.813 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 5/13 2024-01-21T11:35:16.813 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2024-01-21T11:35:16.813 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 7/13 2024-01-21T11:35:16.813 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-01-21T11:35:16.813 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-01-21T11:35:16.813 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 10/13 2024-01-21T11:35:16.813 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2024-01-21T11:35:16.813 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 12/13 2024-01-21T11:35:17.207 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2024-01-21T11:35:17.208 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 1/13 2024-01-21T11:35:17.208 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2/13 2024-01-21T11:35:17.208 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 3/13 2024-01-21T11:35:17.208 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-01-21T11:35:17.208 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 5/13 2024-01-21T11:35:17.208 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2024-01-21T11:35:17.208 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 7/13 2024-01-21T11:35:17.208 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-01-21T11:35:17.208 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-01-21T11:35:17.208 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 10/13 2024-01-21T11:35:17.208 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2024-01-21T11:35:17.208 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 12/13 2024-01-21T11:35:17.363 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-01-21T11:35:17.363 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:35:17.363 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-01-21T11:35:17.363 INFO:teuthology.orchestra.run.smithi134.stdout: aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86_64 2024-01-21T11:35:17.363 INFO:teuthology.orchestra.run.smithi134.stdout: buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2024-01-21T11:35:17.363 INFO:teuthology.orchestra.run.smithi134.stdout: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 2024-01-21T11:35:17.363 INFO:teuthology.orchestra.run.smithi134.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-01-21T11:35:17.363 INFO:teuthology.orchestra.run.smithi134.stdout: crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 2024-01-21T11:35:17.363 INFO:teuthology.orchestra.run.smithi134.stdout: netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 2024-01-21T11:35:17.364 INFO:teuthology.orchestra.run.smithi134.stdout: python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noarch 2024-01-21T11:35:17.364 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-01-21T11:35:17.364 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-01-21T11:35:17.364 INFO:teuthology.orchestra.run.smithi134.stdout: skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 2024-01-21T11:35:17.364 INFO:teuthology.orchestra.run.smithi134.stdout: toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 2024-01-21T11:35:17.364 INFO:teuthology.orchestra.run.smithi134.stdout: udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 2024-01-21T11:35:17.364 INFO:teuthology.orchestra.run.smithi134.stdout: yajl-2.1.0-12.el8.x86_64 2024-01-21T11:35:17.364 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:35:17.364 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-21T11:35:17.694 DEBUG:teuthology.parallel:result is None 2024-01-21T11:35:17.727 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-01-21T11:35:17.727 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-21T11:35:17.727 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-01-21T11:35:17.727 INFO:teuthology.orchestra.run.smithi029.stdout: aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86_64 2024-01-21T11:35:17.727 INFO:teuthology.orchestra.run.smithi029.stdout: buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2024-01-21T11:35:17.727 INFO:teuthology.orchestra.run.smithi029.stdout: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 2024-01-21T11:35:17.728 INFO:teuthology.orchestra.run.smithi029.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-01-21T11:35:17.728 INFO:teuthology.orchestra.run.smithi029.stdout: crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 2024-01-21T11:35:17.728 INFO:teuthology.orchestra.run.smithi029.stdout: netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 2024-01-21T11:35:17.728 INFO:teuthology.orchestra.run.smithi029.stdout: python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noarch 2024-01-21T11:35:17.728 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-01-21T11:35:17.728 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-01-21T11:35:17.728 INFO:teuthology.orchestra.run.smithi029.stdout: skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 2024-01-21T11:35:17.728 INFO:teuthology.orchestra.run.smithi029.stdout: toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 2024-01-21T11:35:17.728 INFO:teuthology.orchestra.run.smithi029.stdout: udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 2024-01-21T11:35:17.728 INFO:teuthology.orchestra.run.smithi029.stdout: yajl-2.1.0-12.el8.x86_64 2024-01-21T11:35:17.728 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-21T11:35:17.728 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-01-21T11:35:17.999 DEBUG:teuthology.parallel:result is None 2024-01-21T11:35:17.999 INFO:teuthology.run_tasks:Running task pexec... 2024-01-21T11:35:18.020 INFO:teuthology.task.pexec:Executing custom commands... 2024-01-21T11:35:18.021 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi029.front.sepia.ceph.com 2024-01-21T11:35:18.021 DEBUG:teuthology.orchestra.run.smithi029:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-01-21T11:35:18.021 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi134.front.sepia.ceph.com 2024-01-21T11:35:18.021 DEBUG:teuthology.orchestra.run.smithi134:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-01-21T11:35:18.202 DEBUG:teuthology.parallel:result is None 2024-01-21T11:35:18.228 DEBUG:teuthology.parallel:result is None 2024-01-21T11:35:18.228 INFO:teuthology.run_tasks:Running task install... 2024-01-21T11:35:18.237 DEBUG:teuthology.task.install:project ceph 2024-01-21T11:35:18.237 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec'}} 2024-01-21T11:35:18.237 DEBUG:teuthology.task.install:config {'exclude_packages': ['ceph-volume'], 'tag': 'v16.2.4', 'flavor': 'default', 'sha1': 'ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec'} 2024-01-21T11:35:18.237 INFO:teuthology.task.install:Using flavor: default 2024-01-21T11:35:18.245 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-01-21T11:35:18.245 INFO:teuthology.task.install:extra packages: [] 2024-01-21T11:35:18.245 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec', 'tag': 'v16.2.4', 'wait_for_package': False} 2024-01-21T11:35:18.246 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-01-21T11:35:18.246 INFO:teuthology.packaging:ref: None 2024-01-21T11:35:18.246 INFO:teuthology.packaging:tag: v16.2.4 2024-01-21T11:35:18.246 INFO:teuthology.packaging:branch: None 2024-01-21T11:35:18.246 INFO:teuthology.packaging:sha1: ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec 2024-01-21T11:35:18.437 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-01-21T11:35:18.437 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-01-21T11:35:18.440 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec', 'tag': 'v16.2.4', 'wait_for_package': False} 2024-01-21T11:35:18.440 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-01-21T11:35:18.440 INFO:teuthology.packaging:ref: None 2024-01-21T11:35:18.441 INFO:teuthology.packaging:tag: v16.2.4 2024-01-21T11:35:18.441 INFO:teuthology.packaging:branch: None 2024-01-21T11:35:18.441 INFO:teuthology.packaging:sha1: ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec 2024-01-21T11:35:18.441 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-01-21T11:35:18.582 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2024-01-21T11:35:18.582 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2024-01-21T11:35:18.620 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-01-21T11:35:18.620 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-01-21T11:35:18.620 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-01-21T11:35:18.650 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-01-21T11:35:18.650 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-01-21T11:35:18.650 INFO:teuthology.packaging:ref: None 2024-01-21T11:35:18.650 INFO:teuthology.packaging:tag: v16.2.4 2024-01-21T11:35:18.650 INFO:teuthology.packaging:branch: None 2024-01-21T11:35:18.650 INFO:teuthology.packaging:sha1: ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec 2024-01-21T11:35:18.650 DEBUG:teuthology.orchestra.run.smithi029:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-01-21T11:35:18.656 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2024-01-21T11:35:18.656 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2024-01-21T11:35:18.681 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-01-21T11:35:18.681 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-01-21T11:35:18.681 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-01-21T11:35:18.710 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-01-21T11:35:18.710 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-01-21T11:35:18.710 INFO:teuthology.packaging:ref: None 2024-01-21T11:35:18.710 INFO:teuthology.packaging:tag: v16.2.4 2024-01-21T11:35:18.710 INFO:teuthology.packaging:branch: None 2024-01-21T11:35:18.710 INFO:teuthology.packaging:sha1: ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec 2024-01-21T11:35:18.710 DEBUG:teuthology.orchestra.run.smithi134:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-01-21T11:35:18.719 DEBUG:teuthology.orchestra.run.smithi029:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-01-21T11:35:18.778 DEBUG:teuthology.orchestra.run.smithi134:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-01-21T11:35:18.799 DEBUG:teuthology.orchestra.run.smithi029:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-01-21T11:35:18.855 DEBUG:teuthology.orchestra.run.smithi134:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-01-21T11:35:18.867 INFO:teuthology.orchestra.run.smithi029.stdout:check_obsoletes = 1 2024-01-21T11:35:18.868 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum clean all 2024-01-21T11:35:18.922 INFO:teuthology.orchestra.run.smithi134.stdout:check_obsoletes = 1 2024-01-21T11:35:18.923 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum clean all 2024-01-21T11:35:19.144 INFO:teuthology.orchestra.run.smithi029.stdout:76 files removed 2024-01-21T11:35:19.173 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-radosgw 2024-01-21T11:35:19.185 INFO:teuthology.orchestra.run.smithi134.stdout:76 files removed 2024-01-21T11:35:19.212 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-radosgw 2024-01-21T11:35:19.682 INFO:teuthology.orchestra.run.smithi029.stdout:ceph packages for x86_64 934 kB/s | 74 kB 00:00 2024-01-21T11:35:19.698 INFO:teuthology.orchestra.run.smithi134.stdout:ceph packages for x86_64 1.0 MB/s | 74 kB 00:00 2024-01-21T11:35:19.800 INFO:teuthology.orchestra.run.smithi029.stdout:ceph noarch packages 196 kB/s | 15 kB 00:00 2024-01-21T11:35:19.815 INFO:teuthology.orchestra.run.smithi134.stdout:ceph noarch packages 192 kB/s | 15 kB 00:00 2024-01-21T11:35:19.910 INFO:teuthology.orchestra.run.smithi029.stdout:ceph source packages 22 kB/s | 1.7 kB 00:00 2024-01-21T11:35:19.941 INFO:teuthology.orchestra.run.smithi134.stdout:ceph source packages 18 kB/s | 1.7 kB 00:00 2024-01-21T11:36:35.919 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 8 - AppStream 355 kB/s | 26 MB 01:15 2024-01-21T11:36:39.699 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 8 - AppStream 339 kB/s | 26 MB 01:19 2024-01-21T11:37:20.836 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 8 - BaseOS 264 kB/s | 10 MB 00:39 2024-01-21T11:37:23.078 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 8 - Extras 39 kB/s | 18 kB 00:00 2024-01-21T11:37:23.429 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 8 - Extras common packages 30 kB/s | 6.9 kB 00:00 2024-01-21T11:37:24.223 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 8 - PowerTools 6.8 MB/s | 4.7 MB 00:00 2024-01-21T11:37:25.226 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 8 - BaseOS 258 kB/s | 10 MB 00:40 2024-01-21T11:37:25.310 INFO:teuthology.orchestra.run.smithi029.stdout:Copr repo for python3-asyncssh owned by ceph 25 kB/s | 3.5 kB 00:00 2024-01-21T11:37:26.823 INFO:teuthology.orchestra.run.smithi029.stdout:Extra Packages for Enterprise Linux 11 MB/s | 16 MB 00:01 2024-01-21T11:37:27.251 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 8 - Extras 40 kB/s | 18 kB 00:00 2024-01-21T11:37:27.610 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 8 - Extras common packages 27 kB/s | 6.9 kB 00:00 2024-01-21T11:37:28.570 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 8 - PowerTools 5.4 MB/s | 4.7 MB 00:00 2024-01-21T11:37:29.580 INFO:teuthology.orchestra.run.smithi134.stdout:Copr repo for python3-asyncssh owned by ceph 29 kB/s | 3.5 kB 00:00 2024-01-21T11:37:30.813 INFO:teuthology.orchestra.run.smithi029.stdout:lab-extras 441 kB/s | 24 kB 00:00 2024-01-21T11:37:33.615 INFO:teuthology.orchestra.run.smithi134.stdout:Extra Packages for Enterprise Linux 4.1 MB/s | 16 MB 00:03 2024-01-21T11:37:34.539 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-01-21T11:37:34.542 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-21T11:37:34.543 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2024-01-21T11:37:34.543 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-21T11:37:34.543 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-01-21T11:37:34.543 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2024-01-21T11:37:34.543 INFO:teuthology.orchestra.run.smithi029.stdout:Upgrading: 2024-01-21T11:37:34.543 INFO:teuthology.orchestra.run.smithi029.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-01-21T11:37:34.543 INFO:teuthology.orchestra.run.smithi029.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2024-01-21T11:37:34.543 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-01-21T11:37:34.543 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2024-01-21T11:37:34.543 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2024-01-21T11:37:34.543 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2024-01-21T11:37:34.543 INFO:teuthology.orchestra.run.smithi029.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2024-01-21T11:37:34.543 INFO:teuthology.orchestra.run.smithi029.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2024-01-21T11:37:34.544 INFO:teuthology.orchestra.run.smithi029.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-01-21T11:37:34.544 INFO:teuthology.orchestra.run.smithi029.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-01-21T11:37:34.544 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2024-01-21T11:37:34.544 INFO:teuthology.orchestra.run.smithi029.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-01-21T11:37:34.544 INFO:teuthology.orchestra.run.smithi029.stdout: libpmemobj x86_64 1.6.1-1.el8 appstream 145 k 2024-01-21T11:37:34.544 INFO:teuthology.orchestra.run.smithi029.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-01-21T11:37:34.544 INFO:teuthology.orchestra.run.smithi029.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2024-01-21T11:37:34.544 INFO:teuthology.orchestra.run.smithi029.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-01-21T11:37:34.544 INFO:teuthology.orchestra.run.smithi029.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2024-01-21T11:37:34.544 INFO:teuthology.orchestra.run.smithi029.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-01-21T11:37:34.544 INFO:teuthology.orchestra.run.smithi029.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-01-21T11:37:34.544 INFO:teuthology.orchestra.run.smithi029.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2024-01-21T11:37:34.544 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2024-01-21T11:37:34.544 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2024-01-21T11:37:34.545 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2024-01-21T11:37:34.545 INFO:teuthology.orchestra.run.smithi029.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-01-21T11:37:34.545 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2024-01-21T11:37:34.545 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2024-01-21T11:37:34.545 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2024-01-21T11:37:34.545 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-21T11:37:34.545 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-01-21T11:37:34.545 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-21T11:37:34.545 INFO:teuthology.orchestra.run.smithi029.stdout:Install 25 Packages 2024-01-21T11:37:34.545 INFO:teuthology.orchestra.run.smithi029.stdout:Upgrade 2 Packages 2024-01-21T11:37:34.545 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-21T11:37:34.546 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 59 M 2024-01-21T11:37:34.546 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-01-21T11:37:35.458 INFO:teuthology.orchestra.run.smithi029.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 28 MB/s | 6.0 MB 00:00 2024-01-21T11:37:35.483 INFO:teuthology.orchestra.run.smithi029.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 990 kB/s | 25 kB 00:00 2024-01-21T11:37:35.543 INFO:teuthology.orchestra.run.smithi029.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 13 MB/s | 801 kB 00:00 2024-01-21T11:37:35.586 INFO:teuthology.orchestra.run.smithi029.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 11 MB/s | 486 kB 00:00 2024-01-21T11:37:35.728 INFO:teuthology.orchestra.run.smithi029.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 29 MB/s | 4.1 MB 00:00 2024-01-21T11:37:35.762 INFO:teuthology.orchestra.run.smithi029.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 1.3 MB/s | 44 kB 00:00 2024-01-21T11:37:35.787 INFO:teuthology.orchestra.run.smithi029.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 3.1 MB/s | 80 kB 00:00 2024-01-21T11:37:35.821 INFO:teuthology.orchestra.run.smithi029.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 6.2 MB/s | 213 kB 00:00 2024-01-21T11:37:35.855 INFO:teuthology.orchestra.run.smithi029.stdout:(9/27): python3-rados-16.2.4-0.el8.x86_64.rpm 11 MB/s | 387 kB 00:00 2024-01-21T11:37:35.889 INFO:teuthology.orchestra.run.smithi029.stdout:(10/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 11 MB/s | 368 kB 00:00 2024-01-21T11:37:35.914 INFO:teuthology.orchestra.run.smithi029.stdout:(11/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 4.5 MB/s | 114 kB 00:00 2024-01-21T11:37:36.132 INFO:teuthology.orchestra.run.smithi029.stdout:(12/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 14 MB/s | 13 MB 00:00 2024-01-21T11:37:36.583 INFO:teuthology.orchestra.run.smithi029.stdout:(13/27): ceph-common-16.2.4-0.el8.x86_64.rpm 18 MB/s | 24 MB 00:01 2024-01-21T11:37:36.891 INFO:teuthology.orchestra.run.smithi029.stdout:(14/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 149 kB/s | 145 kB 00:00 2024-01-21T11:37:37.184 INFO:teuthology.orchestra.run.smithi029.stdout:(15/27): python3-prettytable-0.7.2-14.el8.noarc 150 kB/s | 44 kB 00:00 2024-01-21T11:37:37.342 INFO:teuthology.orchestra.run.smithi029.stdout:(16/27): daxctl-libs-71.1-7.el8.x86_64.rpm 266 kB/s | 42 kB 00:00 2024-01-21T11:37:37.388 INFO:teuthology.orchestra.run.smithi134.stdout:lab-extras 504 kB/s | 24 kB 00:00 2024-01-21T11:37:37.411 INFO:teuthology.orchestra.run.smithi029.stdout:(17/27): librabbitmq-0.9.0-5.el8.x86_64.rpm 686 kB/s | 47 kB 00:00 2024-01-21T11:37:37.487 INFO:teuthology.orchestra.run.smithi029.stdout:(18/27): ndctl-libs-71.1-7.el8.x86_64.rpm 1.0 MB/s | 80 kB 00:00 2024-01-21T11:37:37.712 INFO:teuthology.orchestra.run.smithi029.stdout:(19/27): fmt-6.2.1-1.el8.x86_64.rpm 517 kB/s | 116 kB 00:00 2024-01-21T11:37:37.838 INFO:teuthology.orchestra.run.smithi029.stdout:(20/27): gperftools-libs-2.7-9.el8.x86_64.rpm 2.4 MB/s | 306 kB 00:00 2024-01-21T11:37:37.905 INFO:teuthology.orchestra.run.smithi029.stdout:(21/27): leveldb-1.22-1.el8.x86_64.rpm 2.6 MB/s | 181 kB 00:00 2024-01-21T11:37:37.964 INFO:teuthology.orchestra.run.smithi029.stdout:(22/27): liboath-2.6.2-3.el8.x86_64.rpm 1.0 MB/s | 59 kB 00:00 2024-01-21T11:37:38.031 INFO:teuthology.orchestra.run.smithi029.stdout:(23/27): libunwind-1.3.1-3.el8.x86_64.rpm 1.1 MB/s | 75 kB 00:00 2024-01-21T11:37:38.148 INFO:teuthology.orchestra.run.smithi029.stdout:(24/27): librados2-16.2.4-0.el8.x86_64.rpm 32 MB/s | 3.7 MB 00:00 2024-01-21T11:37:38.182 INFO:teuthology.orchestra.run.smithi029.stdout:(25/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 162 kB/s | 259 kB 00:01 2024-01-21T11:37:38.299 INFO:teuthology.orchestra.run.smithi029.stdout:(26/27): librbd1-16.2.4-0.el8.x86_64.rpm 27 MB/s | 4.0 MB 00:00 2024-01-21T11:37:39.683 INFO:teuthology.orchestra.run.smithi029.stdout:(27/27): librdkafka-1.6.1-1.el8.x86_64.rpm 191 kB/s | 677 kB 00:03 2024-01-21T11:37:39.685 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-01-21T11:37:39.686 INFO:teuthology.orchestra.run.smithi029.stdout:Total 11 MB/s | 59 MB 00:05 2024-01-21T11:37:39.757 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-01-21T11:37:39.810 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-01-21T11:37:39.810 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-01-21T11:37:40.170 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-01-21T11:37:40.173 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-01-21T11:37:40.849 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-21T11:37:40.852 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-21T11:37:40.852 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-01-21T11:37:40.852 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-21T11:37:40.852 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-01-21T11:37:40.852 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2024-01-21T11:37:40.852 INFO:teuthology.orchestra.run.smithi134.stdout:Upgrading: 2024-01-21T11:37:40.852 INFO:teuthology.orchestra.run.smithi134.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-01-21T11:37:40.852 INFO:teuthology.orchestra.run.smithi134.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2024-01-21T11:37:40.852 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-01-21T11:37:40.852 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2024-01-21T11:37:40.852 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2024-01-21T11:37:40.852 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2024-01-21T11:37:40.853 INFO:teuthology.orchestra.run.smithi134.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2024-01-21T11:37:40.853 INFO:teuthology.orchestra.run.smithi134.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2024-01-21T11:37:40.853 INFO:teuthology.orchestra.run.smithi134.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-01-21T11:37:40.853 INFO:teuthology.orchestra.run.smithi134.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-01-21T11:37:40.853 INFO:teuthology.orchestra.run.smithi134.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2024-01-21T11:37:40.853 INFO:teuthology.orchestra.run.smithi134.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-01-21T11:37:40.853 INFO:teuthology.orchestra.run.smithi134.stdout: libpmemobj x86_64 1.6.1-1.el8 appstream 145 k 2024-01-21T11:37:40.853 INFO:teuthology.orchestra.run.smithi134.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-01-21T11:37:40.853 INFO:teuthology.orchestra.run.smithi134.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2024-01-21T11:37:40.853 INFO:teuthology.orchestra.run.smithi134.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-01-21T11:37:40.853 INFO:teuthology.orchestra.run.smithi134.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2024-01-21T11:37:40.853 INFO:teuthology.orchestra.run.smithi134.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-01-21T11:37:40.854 INFO:teuthology.orchestra.run.smithi134.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-01-21T11:37:40.854 INFO:teuthology.orchestra.run.smithi134.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2024-01-21T11:37:40.854 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2024-01-21T11:37:40.854 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2024-01-21T11:37:40.854 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2024-01-21T11:37:40.854 INFO:teuthology.orchestra.run.smithi134.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-01-21T11:37:40.854 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2024-01-21T11:37:40.854 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2024-01-21T11:37:40.854 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2024-01-21T11:37:40.854 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:37:40.854 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-21T11:37:40.854 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-21T11:37:40.854 INFO:teuthology.orchestra.run.smithi134.stdout:Install 25 Packages 2024-01-21T11:37:40.854 INFO:teuthology.orchestra.run.smithi134.stdout:Upgrade 2 Packages 2024-01-21T11:37:40.854 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:37:40.855 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 59 M 2024-01-21T11:37:40.855 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-01-21T11:37:41.256 INFO:teuthology.orchestra.run.smithi134.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 41 MB/s | 6.0 MB 00:00 2024-01-21T11:37:41.276 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-01-21T11:37:41.281 INFO:teuthology.orchestra.run.smithi134.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 995 kB/s | 25 kB 00:00 2024-01-21T11:37:41.315 INFO:teuthology.orchestra.run.smithi134.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 23 MB/s | 801 kB 00:00 2024-01-21T11:37:41.348 INFO:teuthology.orchestra.run.smithi134.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 14 MB/s | 486 kB 00:00 2024-01-21T11:37:41.432 INFO:teuthology.orchestra.run.smithi134.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 49 MB/s | 4.1 MB 00:00 2024-01-21T11:37:41.457 INFO:teuthology.orchestra.run.smithi134.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 1.7 MB/s | 44 kB 00:00 2024-01-21T11:37:41.474 INFO:teuthology.orchestra.run.smithi134.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 4.7 MB/s | 80 kB 00:00 2024-01-21T11:37:41.480 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2024-01-21T11:37:41.508 INFO:teuthology.orchestra.run.smithi134.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 6.2 MB/s | 213 kB 00:00 2024-01-21T11:37:41.533 INFO:teuthology.orchestra.run.smithi134.stdout:(9/27): python3-rados-16.2.4-0.el8.x86_64.rpm 15 MB/s | 387 kB 00:00 2024-01-21T11:37:41.667 INFO:teuthology.orchestra.run.smithi134.stdout:(10/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 23 MB/s | 13 MB 00:00 2024-01-21T11:37:41.692 INFO:teuthology.orchestra.run.smithi134.stdout:(11/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 2.3 MB/s | 368 kB 00:00 2024-01-21T11:37:41.705 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2024-01-21T11:37:41.717 INFO:teuthology.orchestra.run.smithi134.stdout:(12/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 2.2 MB/s | 114 kB 00:00 2024-01-21T11:37:41.879 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-01-21T11:37:42.001 INFO:teuthology.orchestra.run.smithi134.stdout:(13/27): ceph-common-16.2.4-0.el8.x86_64.rpm 27 MB/s | 24 MB 00:00 2024-01-21T11:37:42.309 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-01-21T11:37:42.529 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2024-01-21T11:37:42.632 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2024-01-21T11:37:42.860 INFO:teuthology.orchestra.run.smithi134.stdout:(14/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 124 kB/s | 145 kB 00:01 2024-01-21T11:37:42.900 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2024-01-21T11:37:43.055 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-01-21T11:37:43.121 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-01-21T11:37:43.143 INFO:teuthology.orchestra.run.smithi134.stdout:(15/27): python3-prettytable-0.7.2-14.el8.noarc 155 kB/s | 44 kB 00:00 2024-01-21T11:37:43.244 INFO:teuthology.orchestra.run.smithi134.stdout:(16/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 209 kB/s | 259 kB 00:01 2024-01-21T11:37:43.285 INFO:teuthology.orchestra.run.smithi134.stdout:(17/27): daxctl-libs-71.1-7.el8.x86_64.rpm 298 kB/s | 42 kB 00:00 2024-01-21T11:37:43.343 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 7/29 2024-01-21T11:37:43.369 INFO:teuthology.orchestra.run.smithi134.stdout:(18/27): ndctl-libs-71.1-7.el8.x86_64.rpm 963 kB/s | 80 kB 00:00 2024-01-21T11:37:43.419 INFO:teuthology.orchestra.run.smithi134.stdout:(19/27): librabbitmq-0.9.0-5.el8.x86_64.rpm 268 kB/s | 47 kB 00:00 2024-01-21T11:37:43.711 INFO:teuthology.orchestra.run.smithi134.stdout:(20/27): fmt-6.2.1-1.el8.x86_64.rpm 340 kB/s | 116 kB 00:00 2024-01-21T11:37:43.828 INFO:teuthology.orchestra.run.smithi134.stdout:(21/27): gperftools-libs-2.7-9.el8.x86_64.rpm 758 kB/s | 306 kB 00:00 2024-01-21T11:37:43.853 INFO:teuthology.orchestra.run.smithi134.stdout:(22/27): leveldb-1.22-1.el8.x86_64.rpm 1.2 MB/s | 181 kB 00:00 2024-01-21T11:37:43.895 INFO:teuthology.orchestra.run.smithi134.stdout:(23/27): liboath-2.6.2-3.el8.x86_64.rpm 879 kB/s | 59 kB 00:00 2024-01-21T11:37:43.918 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 8/29 2024-01-21T11:37:43.937 INFO:teuthology.orchestra.run.smithi134.stdout:(24/27): libunwind-1.3.1-3.el8.x86_64.rpm 899 kB/s | 75 kB 00:00 2024-01-21T11:37:43.996 INFO:teuthology.orchestra.run.smithi134.stdout:(25/27): librados2-16.2.4-0.el8.x86_64.rpm 37 MB/s | 3.7 MB 00:00 2024-01-21T11:37:44.063 INFO:teuthology.orchestra.run.smithi134.stdout:(26/27): librbd1-16.2.4-0.el8.x86_64.rpm 32 MB/s | 4.0 MB 00:00 2024-01-21T11:37:44.079 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-01-21T11:37:44.155 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-01-21T11:37:44.351 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2024-01-21T11:37:44.529 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 11/29 2024-01-21T11:37:44.598 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 11/29 2024-01-21T11:37:44.764 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2024-01-21T11:37:44.972 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2024-01-21T11:37:45.063 INFO:teuthology.orchestra.run.smithi134.stdout:(27/27): librdkafka-1.6.1-1.el8.x86_64.rpm 202 kB/s | 677 kB 00:03 2024-01-21T11:37:45.067 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-21T11:37:45.067 INFO:teuthology.orchestra.run.smithi134.stdout:Total 14 MB/s | 59 MB 00:04 2024-01-21T11:37:45.135 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-01-21T11:37:45.171 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 14/29 2024-01-21T11:37:45.183 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-01-21T11:37:45.183 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-01-21T11:37:45.253 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 14/29 2024-01-21T11:37:45.438 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-01-21T11:37:45.582 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-01-21T11:37:45.585 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-01-21T11:37:45.906 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-01-21T11:37:46.079 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-01-21T11:37:46.181 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-01-21T11:37:46.383 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2024-01-21T11:37:46.633 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2024-01-21T11:37:46.709 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-01-21T11:37:46.770 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-01-21T11:37:46.851 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-01-21T11:37:46.906 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2024-01-21T11:37:47.057 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2024-01-21T11:37:47.135 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2024-01-21T11:37:47.247 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2024-01-21T11:37:47.302 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-01-21T11:37:47.443 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2024-01-21T11:37:47.663 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2024-01-21T11:37:47.706 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-01-21T11:37:47.918 INFO:teuthology.orchestra.run.smithi134.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2024-01-21T11:37:48.030 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2024-01-21T11:37:48.282 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2024-01-21T11:37:48.485 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-01-21T11:37:48.551 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-01-21T11:37:48.778 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 7/29 2024-01-21T11:37:49.327 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 8/29 2024-01-21T11:37:49.493 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-01-21T11:37:49.573 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-01-21T11:37:49.790 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2024-01-21T11:37:49.968 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 11/29 2024-01-21T11:37:50.022 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 11/29 2024-01-21T11:37:50.206 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2024-01-21T11:37:50.402 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2024-01-21T11:37:50.422 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-01-21T11:37:50.602 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 14/29 2024-01-21T11:37:50.662 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-01-21T11:37:50.667 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 14/29 2024-01-21T11:37:50.843 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-01-21T11:37:51.276 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-01-21T11:37:51.424 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-01-21T11:37:51.443 INFO:teuthology.orchestra.run.smithi134.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-01-21T11:37:51.524 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-01-21T11:37:51.712 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-01-21T11:37:51.713 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2024-01-21T11:37:51.863 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-01-21T11:37:51.970 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2024-01-21T11:37:51.994 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-01-21T11:37:52.126 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-01-21T11:37:52.192 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-01-21T11:37:52.394 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2024-01-21T11:37:52.601 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2024-01-21T11:37:52.784 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2024-01-21T11:37:53.010 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2024-01-21T11:37:55.601 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-01-21T11:37:55.835 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-01-21T11:37:56.554 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-01-21T11:37:56.875 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-01-21T11:37:56.977 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-01-21T11:37:57.117 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-01-21T11:38:13.508 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-01-21T11:38:13.508 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /sys 2024-01-21T11:38:13.508 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /proc 2024-01-21T11:38:13.508 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /dev 2024-01-21T11:38:13.508 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /run 2024-01-21T11:38:13.508 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /mnt 2024-01-21T11:38:13.508 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /var/tmp 2024-01-21T11:38:13.508 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /home 2024-01-21T11:38:13.509 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /tmp 2024-01-21T11:38:13.509 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /dev 2024-01-21T11:38:13.509 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-21T11:38:14.909 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-01-21T11:38:14.986 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-01-21T11:38:14.987 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-01-21T11:38:15.211 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-01-21T11:38:15.211 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2024-01-21T11:38:15.916 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2024-01-21T11:38:15.916 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2024-01-21T11:38:15.916 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2024-01-21T11:38:15.916 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2024-01-21T11:38:15.916 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2024-01-21T11:38:15.916 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2024-01-21T11:38:15.916 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2024-01-21T11:38:15.916 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2024-01-21T11:38:15.916 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2024-01-21T11:38:15.916 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2024-01-21T11:38:15.916 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2024-01-21T11:38:15.917 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2024-01-21T11:38:15.917 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2024-01-21T11:38:15.917 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2024-01-21T11:38:15.917 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2024-01-21T11:38:15.917 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 15/29 2024-01-21T11:38:15.917 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2024-01-21T11:38:15.917 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2024-01-21T11:38:15.917 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 18/29 2024-01-21T11:38:15.917 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 19/29 2024-01-21T11:38:15.917 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 20/29 2024-01-21T11:38:15.917 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2024-01-21T11:38:15.917 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2024-01-21T11:38:15.917 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2024-01-21T11:38:15.917 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2024-01-21T11:38:15.918 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2024-01-21T11:38:15.918 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2024-01-21T11:38:15.918 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2024-01-21T11:38:15.918 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2024-01-21T11:38:16.317 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-01-21T11:38:16.317 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /sys 2024-01-21T11:38:16.318 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /proc 2024-01-21T11:38:16.318 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /dev 2024-01-21T11:38:16.318 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /run 2024-01-21T11:38:16.318 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /mnt 2024-01-21T11:38:16.318 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /var/tmp 2024-01-21T11:38:16.318 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /home 2024-01-21T11:38:16.318 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /tmp 2024-01-21T11:38:16.318 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /dev 2024-01-21T11:38:16.318 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:38:16.849 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2024-01-21T11:38:16.849 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-21T11:38:16.849 INFO:teuthology.orchestra.run.smithi029.stdout:Upgraded: 2024-01-21T11:38:16.850 INFO:teuthology.orchestra.run.smithi029.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2024-01-21T11:38:16.850 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-01-21T11:38:16.850 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2024-01-21T11:38:16.850 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2024-01-21T11:38:16.850 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2024-01-21T11:38:16.850 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2024-01-21T11:38:16.850 INFO:teuthology.orchestra.run.smithi029.stdout: daxctl-libs-71.1-7.el8.x86_64 2024-01-21T11:38:16.850 INFO:teuthology.orchestra.run.smithi029.stdout: fmt-6.2.1-1.el8.x86_64 2024-01-21T11:38:16.850 INFO:teuthology.orchestra.run.smithi029.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-01-21T11:38:16.850 INFO:teuthology.orchestra.run.smithi029.stdout: leveldb-1.22-1.el8.x86_64 2024-01-21T11:38:16.850 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2024-01-21T11:38:16.850 INFO:teuthology.orchestra.run.smithi029.stdout: liboath-2.6.2-3.el8.x86_64 2024-01-21T11:38:16.850 INFO:teuthology.orchestra.run.smithi029.stdout: libpmemobj-1.6.1-1.el8.x86_64 2024-01-21T11:38:16.850 INFO:teuthology.orchestra.run.smithi029.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-01-21T11:38:16.851 INFO:teuthology.orchestra.run.smithi029.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2024-01-21T11:38:16.851 INFO:teuthology.orchestra.run.smithi029.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-01-21T11:38:16.851 INFO:teuthology.orchestra.run.smithi029.stdout: librgw2-2:16.2.4-0.el8.x86_64 2024-01-21T11:38:16.851 INFO:teuthology.orchestra.run.smithi029.stdout: libunwind-1.3.1-3.el8.x86_64 2024-01-21T11:38:16.851 INFO:teuthology.orchestra.run.smithi029.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-01-21T11:38:16.851 INFO:teuthology.orchestra.run.smithi029.stdout: ndctl-libs-71.1-7.el8.x86_64 2024-01-21T11:38:16.851 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2024-01-21T11:38:16.851 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2024-01-21T11:38:16.851 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2024-01-21T11:38:16.851 INFO:teuthology.orchestra.run.smithi029.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-01-21T11:38:16.851 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2024-01-21T11:38:16.851 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2024-01-21T11:38:16.851 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2024-01-21T11:38:16.851 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-21T11:38:16.851 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-01-21T11:38:17.127 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-test 2024-01-21T11:38:17.646 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-01-21T11:38:17.719 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-01-21T11:38:17.719 INFO:teuthology.orchestra.run.smithi134.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-01-21T11:38:17.886 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:00:47 ago on Sun 21 Jan 2024 11:37:30 AM UTC. 2024-01-21T11:38:17.924 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-01-21T11:38:17.924 INFO:teuthology.orchestra.run.smithi134.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2024-01-21T11:38:18.671 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2024-01-21T11:38:18.672 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2024-01-21T11:38:18.672 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2024-01-21T11:38:18.672 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2024-01-21T11:38:18.672 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2024-01-21T11:38:18.672 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2024-01-21T11:38:18.672 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2024-01-21T11:38:18.672 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2024-01-21T11:38:18.672 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2024-01-21T11:38:18.672 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2024-01-21T11:38:18.672 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2024-01-21T11:38:18.672 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2024-01-21T11:38:18.672 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2024-01-21T11:38:18.672 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2024-01-21T11:38:18.672 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2024-01-21T11:38:18.673 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 15/29 2024-01-21T11:38:18.673 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2024-01-21T11:38:18.673 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2024-01-21T11:38:18.673 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 18/29 2024-01-21T11:38:18.673 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 19/29 2024-01-21T11:38:18.673 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 20/29 2024-01-21T11:38:18.673 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2024-01-21T11:38:18.673 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2024-01-21T11:38:18.673 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2024-01-21T11:38:18.673 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2024-01-21T11:38:18.673 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2024-01-21T11:38:18.673 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2024-01-21T11:38:18.673 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2024-01-21T11:38:18.673 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2024-01-21T11:38:19.514 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-01-21T11:38:19.514 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-21T11:38:19.515 INFO:teuthology.orchestra.run.smithi029.stdout: Package Architecture Version Repository Size 2024-01-21T11:38:19.515 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-21T11:38:19.515 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-01-21T11:38:19.515 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2024-01-21T11:38:19.515 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-01-21T11:38:19.515 INFO:teuthology.orchestra.run.smithi029.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-01-21T11:38:19.515 INFO:teuthology.orchestra.run.smithi029.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2024-01-21T11:38:19.515 INFO:teuthology.orchestra.run.smithi029.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-01-21T11:38:19.515 INFO:teuthology.orchestra.run.smithi029.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-01-21T11:38:19.515 INFO:teuthology.orchestra.run.smithi029.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-01-21T11:38:19.515 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-21T11:38:19.515 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-01-21T11:38:19.515 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-21T11:38:19.516 INFO:teuthology.orchestra.run.smithi029.stdout:Install 6 Packages 2024-01-21T11:38:19.516 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-21T11:38:19.516 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 53 M 2024-01-21T11:38:19.516 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 229 M 2024-01-21T11:38:19.516 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-01-21T11:38:19.567 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2024-01-21T11:38:19.567 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:38:19.567 INFO:teuthology.orchestra.run.smithi134.stdout:Upgraded: 2024-01-21T11:38:19.567 INFO:teuthology.orchestra.run.smithi134.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2024-01-21T11:38:19.568 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-01-21T11:38:19.568 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2024-01-21T11:38:19.568 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2024-01-21T11:38:19.568 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2024-01-21T11:38:19.568 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2024-01-21T11:38:19.568 INFO:teuthology.orchestra.run.smithi134.stdout: daxctl-libs-71.1-7.el8.x86_64 2024-01-21T11:38:19.568 INFO:teuthology.orchestra.run.smithi134.stdout: fmt-6.2.1-1.el8.x86_64 2024-01-21T11:38:19.568 INFO:teuthology.orchestra.run.smithi134.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-01-21T11:38:19.568 INFO:teuthology.orchestra.run.smithi134.stdout: leveldb-1.22-1.el8.x86_64 2024-01-21T11:38:19.568 INFO:teuthology.orchestra.run.smithi134.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2024-01-21T11:38:19.568 INFO:teuthology.orchestra.run.smithi134.stdout: liboath-2.6.2-3.el8.x86_64 2024-01-21T11:38:19.568 INFO:teuthology.orchestra.run.smithi134.stdout: libpmemobj-1.6.1-1.el8.x86_64 2024-01-21T11:38:19.568 INFO:teuthology.orchestra.run.smithi134.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-01-21T11:38:19.569 INFO:teuthology.orchestra.run.smithi134.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2024-01-21T11:38:19.569 INFO:teuthology.orchestra.run.smithi134.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-01-21T11:38:19.569 INFO:teuthology.orchestra.run.smithi134.stdout: librgw2-2:16.2.4-0.el8.x86_64 2024-01-21T11:38:19.569 INFO:teuthology.orchestra.run.smithi134.stdout: libunwind-1.3.1-3.el8.x86_64 2024-01-21T11:38:19.569 INFO:teuthology.orchestra.run.smithi134.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-01-21T11:38:19.569 INFO:teuthology.orchestra.run.smithi134.stdout: ndctl-libs-71.1-7.el8.x86_64 2024-01-21T11:38:19.569 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2024-01-21T11:38:19.569 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2024-01-21T11:38:19.569 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2024-01-21T11:38:19.569 INFO:teuthology.orchestra.run.smithi134.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-01-21T11:38:19.569 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2024-01-21T11:38:19.569 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2024-01-21T11:38:19.569 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2024-01-21T11:38:19.569 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:38:19.569 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-21T11:38:19.794 INFO:teuthology.orchestra.run.smithi029.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 1.8 MB/s | 169 kB 00:00 2024-01-21T11:38:19.847 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-test 2024-01-21T11:38:20.584 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:00:43 ago on Sun 21 Jan 2024 11:37:37 AM UTC. 2024-01-21T11:38:20.778 INFO:teuthology.orchestra.run.smithi029.stdout:(2/6): ceph-test-16.2.4-0.el8.x86_64.rpm 48 MB/s | 52 MB 00:01 2024-01-21T11:38:20.987 INFO:teuthology.orchestra.run.smithi029.stdout:(3/6): oniguruma-6.8.2-3.el8.x86_64.rpm 158 kB/s | 188 kB 00:01 2024-01-21T11:38:21.087 INFO:teuthology.orchestra.run.smithi029.stdout:(4/6): jq-1.6-8.el8.x86_64.rpm 146 kB/s | 203 kB 00:01 2024-01-21T11:38:21.396 INFO:teuthology.orchestra.run.smithi029.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 173 kB/s | 71 kB 00:00 2024-01-21T11:38:22.077 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-21T11:38:22.078 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-21T11:38:22.078 INFO:teuthology.orchestra.run.smithi134.stdout: Package Architecture Version Repository Size 2024-01-21T11:38:22.078 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-21T11:38:22.078 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-01-21T11:38:22.078 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2024-01-21T11:38:22.078 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-01-21T11:38:22.078 INFO:teuthology.orchestra.run.smithi134.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-01-21T11:38:22.079 INFO:teuthology.orchestra.run.smithi134.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2024-01-21T11:38:22.079 INFO:teuthology.orchestra.run.smithi134.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-01-21T11:38:22.079 INFO:teuthology.orchestra.run.smithi134.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-01-21T11:38:22.079 INFO:teuthology.orchestra.run.smithi134.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-01-21T11:38:22.079 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:38:22.079 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-21T11:38:22.079 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-21T11:38:22.079 INFO:teuthology.orchestra.run.smithi134.stdout:Install 6 Packages 2024-01-21T11:38:22.079 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:38:22.080 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 53 M 2024-01-21T11:38:22.080 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 229 M 2024-01-21T11:38:22.080 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-01-21T11:38:22.169 INFO:teuthology.orchestra.run.smithi134.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 3.6 MB/s | 169 kB 00:00 2024-01-21T11:38:22.688 INFO:teuthology.orchestra.run.smithi029.stdout:(6/6): socat-1.7.4.1-1.el8.x86_64.rpm 169 kB/s | 323 kB 00:01 2024-01-21T11:38:22.689 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-01-21T11:38:22.689 INFO:teuthology.orchestra.run.smithi029.stdout:Total 17 MB/s | 53 MB 00:03 2024-01-21T11:38:22.733 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-01-21T11:38:22.756 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-01-21T11:38:22.756 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-01-21T11:38:22.802 INFO:teuthology.orchestra.run.smithi134.stdout:(2/6): ceph-test-16.2.4-0.el8.x86_64.rpm 76 MB/s | 52 MB 00:00 2024-01-21T11:38:23.086 INFO:teuthology.orchestra.run.smithi134.stdout:(3/6): oniguruma-6.8.2-3.el8.x86_64.rpm 205 kB/s | 188 kB 00:00 2024-01-21T11:38:23.088 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-01-21T11:38:23.092 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-01-21T11:38:23.219 INFO:teuthology.orchestra.run.smithi134.stdout:(4/6): jq-1.6-8.el8.x86_64.rpm 185 kB/s | 203 kB 00:01 2024-01-21T11:38:23.361 INFO:teuthology.orchestra.run.smithi134.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 257 kB/s | 71 kB 00:00 2024-01-21T11:38:23.595 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-01-21T11:38:23.842 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-01-21T11:38:24.058 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-01-21T11:38:24.170 INFO:teuthology.orchestra.run.smithi134.stdout:(6/6): socat-1.7.4.1-1.el8.x86_64.rpm 236 kB/s | 323 kB 00:01 2024-01-21T11:38:24.171 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-21T11:38:24.171 INFO:teuthology.orchestra.run.smithi134.stdout:Total 25 MB/s | 53 MB 00:02 2024-01-21T11:38:24.213 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-01-21T11:38:24.233 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-01-21T11:38:24.234 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-01-21T11:38:24.234 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-01-21T11:38:24.315 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-01-21T11:38:24.508 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-01-21T11:38:24.554 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-01-21T11:38:24.557 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-01-21T11:38:24.666 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-01-21T11:38:25.034 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-01-21T11:38:25.307 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-01-21T11:38:25.506 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-01-21T11:38:25.654 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-01-21T11:38:25.716 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-01-21T11:38:25.936 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-01-21T11:38:26.079 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-01-21T11:38:29.865 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-01-21T11:38:30.089 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-01-21T11:38:30.434 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-01-21T11:38:30.434 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2024-01-21T11:38:30.434 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2024-01-21T11:38:30.434 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-01-21T11:38:30.434 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-01-21T11:38:30.434 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-01-21T11:38:30.879 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-01-21T11:38:30.879 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-21T11:38:30.879 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-01-21T11:38:30.879 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-8.el8.x86_64 2024-01-21T11:38:30.879 INFO:teuthology.orchestra.run.smithi029.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-3.el8.x86_64 2024-01-21T11:38:30.879 INFO:teuthology.orchestra.run.smithi029.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2024-01-21T11:38:30.880 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-21T11:38:30.880 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-01-21T11:38:30.927 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-01-21T11:38:31.091 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph 2024-01-21T11:38:31.169 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-01-21T11:38:31.460 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-01-21T11:38:31.460 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2024-01-21T11:38:31.460 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2024-01-21T11:38:31.460 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-01-21T11:38:31.460 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-01-21T11:38:31.460 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-01-21T11:38:31.847 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:01 ago on Sun 21 Jan 2024 11:37:30 AM UTC. 2024-01-21T11:38:31.888 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-01-21T11:38:31.889 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:38:31.889 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-01-21T11:38:31.889 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-8.el8.x86_64 2024-01-21T11:38:31.889 INFO:teuthology.orchestra.run.smithi134.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-3.el8.x86_64 2024-01-21T11:38:31.889 INFO:teuthology.orchestra.run.smithi134.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2024-01-21T11:38:31.889 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:38:31.889 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-21T11:38:32.254 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph 2024-01-21T11:38:32.986 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:00:55 ago on Sun 21 Jan 2024 11:37:37 AM UTC. 2024-01-21T11:38:33.439 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-01-21T11:38:33.443 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-21T11:38:33.443 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2024-01-21T11:38:33.443 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-21T11:38:33.443 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-01-21T11:38:33.443 INFO:teuthology.orchestra.run.smithi029.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2024-01-21T11:38:33.443 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-01-21T11:38:33.443 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2024-01-21T11:38:33.443 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2024-01-21T11:38:33.444 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2024-01-21T11:38:33.444 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2024-01-21T11:38:33.444 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2024-01-21T11:38:33.444 INFO:teuthology.orchestra.run.smithi029.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-01-21T11:38:33.444 INFO:teuthology.orchestra.run.smithi029.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-01-21T11:38:33.444 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-01-21T11:38:33.444 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-01-21T11:38:33.444 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-01-21T11:38:33.444 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-01-21T11:38:33.445 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-01-21T11:38:33.445 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-01-21T11:38:33.445 INFO:teuthology.orchestra.run.smithi029.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-01-21T11:38:33.445 INFO:teuthology.orchestra.run.smithi029.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-01-21T11:38:33.445 INFO:teuthology.orchestra.run.smithi029.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-01-21T11:38:33.445 INFO:teuthology.orchestra.run.smithi029.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-01-21T11:38:33.445 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-01-21T11:38:33.445 INFO:teuthology.orchestra.run.smithi029.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-01-21T11:38:33.445 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-01-21T11:38:33.445 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-01-21T11:38:33.445 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-01-21T11:38:33.446 INFO:teuthology.orchestra.run.smithi029.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-01-21T11:38:33.446 INFO:teuthology.orchestra.run.smithi029.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-01-21T11:38:33.446 INFO:teuthology.orchestra.run.smithi029.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-01-21T11:38:33.446 INFO:teuthology.orchestra.run.smithi029.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-01-21T11:38:33.446 INFO:teuthology.orchestra.run.smithi029.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-01-21T11:38:33.446 INFO:teuthology.orchestra.run.smithi029.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-01-21T11:38:33.446 INFO:teuthology.orchestra.run.smithi029.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-01-21T11:38:33.446 INFO:teuthology.orchestra.run.smithi029.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-01-21T11:38:33.446 INFO:teuthology.orchestra.run.smithi029.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-01-21T11:38:33.446 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-21T11:38:33.446 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-01-21T11:38:33.447 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-21T11:38:33.447 INFO:teuthology.orchestra.run.smithi029.stdout:Install 32 Packages 2024-01-21T11:38:33.447 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-21T11:38:33.447 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 31 M 2024-01-21T11:38:33.447 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 111 M 2024-01-21T11:38:33.448 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-01-21T11:38:33.857 INFO:teuthology.orchestra.run.smithi029.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 128 kB/s | 6.3 kB 00:00 2024-01-21T11:38:33.916 INFO:teuthology.orchestra.run.smithi029.stdout:(2/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-01-21T11:38:33.991 INFO:teuthology.orchestra.run.smithi029.stdout:(3/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 13 MB/s | 2.4 MB 00:00 2024-01-21T11:38:34.017 INFO:teuthology.orchestra.run.smithi029.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 8.5 MB/s | 220 kB 00:00 2024-01-21T11:38:34.143 INFO:teuthology.orchestra.run.smithi029.stdout:(5/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 17 MB/s | 5.0 MB 00:00 2024-01-21T11:38:34.459 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-21T11:38:34.462 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-21T11:38:34.462 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-01-21T11:38:34.463 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-21T11:38:34.463 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-01-21T11:38:34.463 INFO:teuthology.orchestra.run.smithi134.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2024-01-21T11:38:34.463 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-01-21T11:38:34.463 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2024-01-21T11:38:34.463 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2024-01-21T11:38:34.463 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2024-01-21T11:38:34.463 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2024-01-21T11:38:34.463 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2024-01-21T11:38:34.463 INFO:teuthology.orchestra.run.smithi134.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-01-21T11:38:34.463 INFO:teuthology.orchestra.run.smithi134.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-01-21T11:38:34.463 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-01-21T11:38:34.463 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-01-21T11:38:34.464 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-01-21T11:38:34.464 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-01-21T11:38:34.464 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-01-21T11:38:34.464 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-01-21T11:38:34.464 INFO:teuthology.orchestra.run.smithi134.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-01-21T11:38:34.464 INFO:teuthology.orchestra.run.smithi134.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-01-21T11:38:34.464 INFO:teuthology.orchestra.run.smithi134.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-01-21T11:38:34.464 INFO:teuthology.orchestra.run.smithi134.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-01-21T11:38:34.464 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-01-21T11:38:34.464 INFO:teuthology.orchestra.run.smithi134.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-01-21T11:38:34.464 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-01-21T11:38:34.464 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-01-21T11:38:34.464 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-01-21T11:38:34.465 INFO:teuthology.orchestra.run.smithi134.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-01-21T11:38:34.465 INFO:teuthology.orchestra.run.smithi134.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-01-21T11:38:34.465 INFO:teuthology.orchestra.run.smithi134.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-01-21T11:38:34.465 INFO:teuthology.orchestra.run.smithi134.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-01-21T11:38:34.465 INFO:teuthology.orchestra.run.smithi134.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-01-21T11:38:34.465 INFO:teuthology.orchestra.run.smithi134.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-01-21T11:38:34.465 INFO:teuthology.orchestra.run.smithi134.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-01-21T11:38:34.465 INFO:teuthology.orchestra.run.smithi134.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-01-21T11:38:34.465 INFO:teuthology.orchestra.run.smithi134.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-01-21T11:38:34.465 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:38:34.465 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-21T11:38:34.465 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-21T11:38:34.465 INFO:teuthology.orchestra.run.smithi134.stdout:Install 32 Packages 2024-01-21T11:38:34.465 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:38:34.466 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 31 M 2024-01-21T11:38:34.466 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 111 M 2024-01-21T11:38:34.466 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-01-21T11:38:34.535 INFO:teuthology.orchestra.run.smithi029.stdout:(6/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 29 MB/s | 18 MB 00:00 2024-01-21T11:38:34.752 INFO:teuthology.orchestra.run.smithi029.stdout:(7/32): python3-markupsafe-0.23-19.el8.x86_64.r 64 kB/s | 39 kB 00:00 2024-01-21T11:38:34.836 INFO:teuthology.orchestra.run.smithi029.stdout:(8/32): python3-mako-1.0.6-14.el8.noarch.rpm 192 kB/s | 157 kB 00:00 2024-01-21T11:38:34.954 INFO:teuthology.orchestra.run.smithi134.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 114 kB/s | 6.3 kB 00:00 2024-01-21T11:38:35.004 INFO:teuthology.orchestra.run.smithi134.stdout:(2/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-01-21T11:38:35.028 INFO:teuthology.orchestra.run.smithi029.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 207 kB/s | 57 kB 00:00 2024-01-21T11:38:35.054 INFO:teuthology.orchestra.run.smithi134.stdout:(3/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 15 MB/s | 2.4 MB 00:00 2024-01-21T11:38:35.080 INFO:teuthology.orchestra.run.smithi134.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 8.6 MB/s | 220 kB 00:00 2024-01-21T11:38:35.080 INFO:teuthology.orchestra.run.smithi029.stdout:(10/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 190 kB/s | 103 kB 00:00 2024-01-21T11:38:35.172 INFO:teuthology.orchestra.run.smithi134.stdout:(5/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 23 MB/s | 5.0 MB 00:00 2024-01-21T11:38:35.264 INFO:teuthology.orchestra.run.smithi029.stdout:(11/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 1.0 MB/s | 249 kB 00:00 2024-01-21T11:38:35.323 INFO:teuthology.orchestra.run.smithi029.stdout:(12/32): python3-pycparser-2.14-14.el8.noarch.r 1.8 MB/s | 109 kB 00:00 2024-01-21T11:38:35.398 INFO:teuthology.orchestra.run.smithi029.stdout:(13/32): python3-cryptography-3.2.1-7.el8.x86_6 2.0 MB/s | 641 kB 00:00 2024-01-21T11:38:35.405 INFO:teuthology.orchestra.run.smithi134.stdout:(6/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 45 MB/s | 18 MB 00:00 2024-01-21T11:38:35.616 INFO:teuthology.orchestra.run.smithi029.stdout:(14/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 149 kB/s | 44 kB 00:00 2024-01-21T11:38:35.722 INFO:teuthology.orchestra.run.smithi134.stdout:(7/32): python3-markupsafe-0.23-19.el8.x86_64.r 70 kB/s | 39 kB 00:00 2024-01-21T11:38:35.758 INFO:teuthology.orchestra.run.smithi029.stdout:(15/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 523 kB/s | 185 kB 00:00 2024-01-21T11:38:35.792 INFO:teuthology.orchestra.run.smithi029.stdout:(16/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 982 kB/s | 173 kB 00:00 2024-01-21T11:38:35.867 INFO:teuthology.orchestra.run.smithi029.stdout:(17/32): python3-jaraco-6.2-6.el8.noarch.rpm 147 kB/s | 11 kB 00:00 2024-01-21T11:38:35.918 INFO:teuthology.orchestra.run.smithi029.stdout:(18/32): python3-cherrypy-18.4.0-1.el8.noarch.r 2.4 MB/s | 384 kB 00:00 2024-01-21T11:38:35.952 INFO:teuthology.orchestra.run.smithi029.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 209 kB/s | 18 kB 00:00 2024-01-21T11:38:36.002 INFO:teuthology.orchestra.run.smithi029.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 581 kB/s | 49 kB 00:00 2024-01-21T11:38:36.014 INFO:teuthology.orchestra.run.smithi134.stdout:(8/32): python3-pytz-2017.2-11.el8.noarch.rpm 195 kB/s | 57 kB 00:00 2024-01-21T11:38:36.036 INFO:teuthology.orchestra.run.smithi029.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 697 kB/s | 59 kB 00:00 2024-01-21T11:38:36.104 INFO:teuthology.orchestra.run.smithi029.stdout:(22/32): python3-pecan-1.3.2-9.el8.noarch.rpm 2.7 MB/s | 283 kB 00:00 2024-01-21T11:38:36.129 INFO:teuthology.orchestra.run.smithi029.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 168 kB/s | 16 kB 00:00 2024-01-21T11:38:36.131 INFO:teuthology.orchestra.run.smithi134.stdout:(9/32): python3-mako-1.0.6-14.el8.noarch.rpm 149 kB/s | 157 kB 00:01 2024-01-21T11:38:36.179 INFO:teuthology.orchestra.run.smithi029.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 247 kB/s | 19 kB 00:00 2024-01-21T11:38:36.190 INFO:teuthology.orchestra.run.smithi134.stdout:(10/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 132 kB/s | 103 kB 00:00 2024-01-21T11:38:36.206 INFO:teuthology.orchestra.run.smithi029.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 309 kB/s | 24 kB 00:00 2024-01-21T11:38:36.257 INFO:teuthology.orchestra.run.smithi029.stdout:(26/32): python3-tempora-1.14.1-5.el8.noarch.rp 379 kB/s | 29 kB 00:00 2024-01-21T11:38:36.282 INFO:teuthology.orchestra.run.smithi029.stdout:(27/32): python3-trustme-0.6.0-4.el8.noarch.rpm 363 kB/s | 27 kB 00:00 2024-01-21T11:38:36.384 INFO:teuthology.orchestra.run.smithi029.stdout:(28/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 2.4 MB/s | 251 kB 00:00 2024-01-21T11:38:36.399 INFO:teuthology.orchestra.run.smithi134.stdout:(11/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 931 kB/s | 249 kB 00:00 2024-01-21T11:38:36.409 INFO:teuthology.orchestra.run.smithi029.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 1.6 MB/s | 255 kB 00:00 2024-01-21T11:38:36.457 INFO:teuthology.orchestra.run.smithi134.stdout:(12/32): python3-pycparser-2.14-14.el8.noarch.r 1.8 MB/s | 109 kB 00:00 2024-01-21T11:38:36.467 INFO:teuthology.orchestra.run.smithi029.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 1.0 MB/s | 85 kB 00:00 2024-01-21T11:38:36.493 INFO:teuthology.orchestra.run.smithi029.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 273 kB/s | 23 kB 00:00 2024-01-21T11:38:36.524 INFO:teuthology.orchestra.run.smithi134.stdout:(13/32): python3-cryptography-3.2.1-7.el8.x86_6 1.9 MB/s | 641 kB 00:00 2024-01-21T11:38:36.527 INFO:teuthology.orchestra.run.smithi029.stdout:(32/32): python3-werkzeug-0.12.2-4.el8.noarch.r 271 kB/s | 457 kB 00:01 2024-01-21T11:38:36.530 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-01-21T11:38:36.530 INFO:teuthology.orchestra.run.smithi029.stdout:Total 10 MB/s | 31 MB 00:03 2024-01-21T11:38:36.614 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-01-21T11:38:36.656 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-01-21T11:38:36.656 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-01-21T11:38:36.724 INFO:teuthology.orchestra.run.smithi134.stdout:(14/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 163 kB/s | 44 kB 00:00 2024-01-21T11:38:36.883 INFO:teuthology.orchestra.run.smithi134.stdout:(15/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 523 kB/s | 185 kB 00:00 2024-01-21T11:38:36.908 INFO:teuthology.orchestra.run.smithi134.stdout:(16/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 940 kB/s | 173 kB 00:00 2024-01-21T11:38:37.020 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-01-21T11:38:37.024 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-01-21T11:38:37.041 INFO:teuthology.orchestra.run.smithi134.stdout:(17/32): python3-jaraco-6.2-6.el8.noarch.rpm 83 kB/s | 11 kB 00:00 2024-01-21T11:38:37.083 INFO:teuthology.orchestra.run.smithi134.stdout:(18/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.9 MB/s | 384 kB 00:00 2024-01-21T11:38:37.117 INFO:teuthology.orchestra.run.smithi134.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 233 kB/s | 18 kB 00:00 2024-01-21T11:38:37.159 INFO:teuthology.orchestra.run.smithi134.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 647 kB/s | 49 kB 00:00 2024-01-21T11:38:37.201 INFO:teuthology.orchestra.run.smithi134.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 702 kB/s | 59 kB 00:00 2024-01-21T11:38:37.260 INFO:teuthology.orchestra.run.smithi134.stdout:(22/32): python3-pecan-1.3.2-9.el8.noarch.rpm 2.8 MB/s | 283 kB 00:00 2024-01-21T11:38:37.285 INFO:teuthology.orchestra.run.smithi134.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 186 kB/s | 16 kB 00:00 2024-01-21T11:38:37.343 INFO:teuthology.orchestra.run.smithi134.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 224 kB/s | 19 kB 00:00 2024-01-21T11:38:37.369 INFO:teuthology.orchestra.run.smithi134.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 286 kB/s | 24 kB 00:00 2024-01-21T11:38:37.419 INFO:teuthology.orchestra.run.smithi134.stdout:(26/32): python3-tempora-1.14.1-5.el8.noarch.rp 388 kB/s | 29 kB 00:00 2024-01-21T11:38:37.461 INFO:teuthology.orchestra.run.smithi134.stdout:(27/32): python3-trustme-0.6.0-4.el8.noarch.rpm 300 kB/s | 27 kB 00:00 2024-01-21T11:38:37.553 INFO:teuthology.orchestra.run.smithi134.stdout:(28/32): python3-waitress-1.4.3-1.el8.noarch.rp 1.9 MB/s | 255 kB 00:00 2024-01-21T11:38:37.578 INFO:teuthology.orchestra.run.smithi134.stdout:(29/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 2.1 MB/s | 251 kB 00:00 2024-01-21T11:38:37.637 INFO:teuthology.orchestra.run.smithi134.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 1.0 MB/s | 85 kB 00:00 2024-01-21T11:38:37.662 INFO:teuthology.orchestra.run.smithi134.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 275 kB/s | 23 kB 00:00 2024-01-21T11:38:38.118 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-01-21T11:38:38.262 INFO:teuthology.orchestra.run.smithi134.stdout:(32/32): python3-werkzeug-0.12.2-4.el8.noarch.r 203 kB/s | 457 kB 00:02 2024-01-21T11:38:38.265 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-21T11:38:38.266 INFO:teuthology.orchestra.run.smithi134.stdout:Total 8.2 MB/s | 31 MB 00:03 2024-01-21T11:38:38.340 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-01-21T11:38:38.344 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-01-21T11:38:38.383 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-01-21T11:38:38.383 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-01-21T11:38:38.556 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-01-21T11:38:38.687 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-01-21T11:38:38.691 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-01-21T11:38:38.818 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-01-21T11:38:39.021 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-01-21T11:38:39.219 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-01-21T11:38:39.417 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-01-21T11:38:39.601 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-01-21T11:38:39.779 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-01-21T11:38:39.803 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-01-21T11:38:39.975 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-01-21T11:38:40.048 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-01-21T11:38:40.203 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-01-21T11:38:40.226 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-01-21T11:38:40.432 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-01-21T11:38:40.468 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-01-21T11:38:40.716 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-01-21T11:38:40.724 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-01-21T11:38:40.914 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-01-21T11:38:41.118 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-01-21T11:38:41.119 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-01-21T11:38:41.322 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-01-21T11:38:41.368 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-01-21T11:38:41.508 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-01-21T11:38:41.556 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-01-21T11:38:41.716 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-01-21T11:38:41.845 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-01-21T11:38:41.928 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-01-21T11:38:42.041 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-01-21T11:38:42.183 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-01-21T11:38:42.361 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-01-21T11:38:42.423 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-01-21T11:38:42.580 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-01-21T11:38:42.785 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-01-21T11:38:42.787 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-01-21T11:38:43.001 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-01-21T11:38:43.029 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-01-21T11:38:43.217 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-01-21T11:38:43.267 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-01-21T11:38:43.508 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-01-21T11:38:43.563 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-01-21T11:38:43.677 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-01-21T11:38:43.827 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-01-21T11:38:43.992 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-01-21T11:38:44.188 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-01-21T11:38:44.240 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-01-21T11:38:44.420 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-01-21T11:38:44.510 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-01-21T11:38:44.649 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-01-21T11:38:44.886 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2024-01-21T11:38:44.924 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-01-21T11:38:45.097 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-01-21T11:38:45.157 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-01-21T11:38:45.369 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-01-21T11:38:45.749 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-01-21T11:38:46.056 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-01-21T11:38:46.488 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2024-01-21T11:38:46.725 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-01-21T11:38:46.729 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-01-21T11:38:46.922 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-01-21T11:38:47.381 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-01-21T11:38:47.605 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-01-21T11:38:47.840 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-01-21T11:38:48.055 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-01-21T11:38:48.075 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-01-21T11:38:48.259 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-01-21T11:38:48.483 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-01-21T11:38:48.915 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-01-21T11:38:49.006 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2024-01-21T11:38:49.166 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-01-21T11:38:49.388 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-01-21T11:38:49.448 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2024-01-21T11:38:49.448 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2024-01-21T11:38:49.449 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2024-01-21T11:38:49.449 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2024-01-21T11:38:49.449 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2024-01-21T11:38:49.449 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2024-01-21T11:38:49.449 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2024-01-21T11:38:49.449 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-01-21T11:38:49.449 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-01-21T11:38:49.450 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-01-21T11:38:49.450 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-01-21T11:38:49.450 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-01-21T11:38:49.450 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-01-21T11:38:49.450 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-01-21T11:38:49.451 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-01-21T11:38:49.451 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-01-21T11:38:49.451 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-01-21T11:38:49.451 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-01-21T11:38:49.451 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-01-21T11:38:49.451 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-01-21T11:38:49.451 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-01-21T11:38:49.452 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-01-21T11:38:49.452 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-01-21T11:38:49.452 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-01-21T11:38:49.452 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-01-21T11:38:49.452 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-01-21T11:38:49.453 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-01-21T11:38:49.453 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-01-21T11:38:49.453 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-01-21T11:38:49.453 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-01-21T11:38:49.453 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-01-21T11:38:49.453 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-01-21T11:38:49.584 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-01-21T11:38:49.602 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-01-21T11:38:49.767 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2024-01-21T11:38:50.165 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2024-01-21T11:38:50.165 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2024-01-21T11:38:50.165 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2024-01-21T11:38:50.165 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2024-01-21T11:38:50.165 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2024-01-21T11:38:50.165 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2024-01-21T11:38:50.165 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2024-01-21T11:38:50.165 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-01-21T11:38:50.165 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-01-21T11:38:50.165 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-01-21T11:38:50.165 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-01-21T11:38:50.165 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-01-21T11:38:50.165 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-01-21T11:38:50.166 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-01-21T11:38:50.166 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-01-21T11:38:50.166 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-01-21T11:38:50.166 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-01-21T11:38:50.166 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-01-21T11:38:50.166 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-01-21T11:38:50.166 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-01-21T11:38:50.166 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-01-21T11:38:50.166 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-01-21T11:38:50.166 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-01-21T11:38:50.166 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-01-21T11:38:50.166 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-01-21T11:38:50.166 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-01-21T11:38:50.166 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-01-21T11:38:50.166 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-01-21T11:38:50.166 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-01-21T11:38:50.167 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-01-21T11:38:50.167 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-01-21T11:38:50.167 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-01-21T11:38:50.349 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-01-21T11:38:50.349 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-21T11:38:50.349 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-01-21T11:38:50.349 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-2:16.2.4-0.el8.x86_64 2024-01-21T11:38:50.349 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2024-01-21T11:38:50.349 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2024-01-21T11:38:50.349 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2024-01-21T11:38:50.350 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2024-01-21T11:38:50.350 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2024-01-21T11:38:50.350 INFO:teuthology.orchestra.run.smithi029.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-01-21T11:38:50.350 INFO:teuthology.orchestra.run.smithi029.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-01-21T11:38:50.350 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-01-21T11:38:50.350 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-01-21T11:38:50.350 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-01-21T11:38:50.350 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-01-21T11:38:50.350 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-6.2-6.el8.noarch 2024-01-21T11:38:50.350 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-01-21T11:38:50.350 INFO:teuthology.orchestra.run.smithi029.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-01-21T11:38:50.350 INFO:teuthology.orchestra.run.smithi029.stdout: python3-mako-1.0.6-14.el8.noarch 2024-01-21T11:38:50.350 INFO:teuthology.orchestra.run.smithi029.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-01-21T11:38:50.350 INFO:teuthology.orchestra.run.smithi029.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-01-21T11:38:50.350 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-01-21T11:38:50.351 INFO:teuthology.orchestra.run.smithi029.stdout: python3-portend-2.6-1.el8.noarch 2024-01-21T11:38:50.351 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-01-21T11:38:50.351 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pycparser-2.14-14.el8.noarch 2024-01-21T11:38:50.351 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pytz-2017.2-11.el8.noarch 2024-01-21T11:38:50.351 INFO:teuthology.orchestra.run.smithi029.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-01-21T11:38:50.351 INFO:teuthology.orchestra.run.smithi029.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-01-21T11:38:50.351 INFO:teuthology.orchestra.run.smithi029.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-01-21T11:38:50.351 INFO:teuthology.orchestra.run.smithi029.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-01-21T11:38:50.351 INFO:teuthology.orchestra.run.smithi029.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-01-21T11:38:50.351 INFO:teuthology.orchestra.run.smithi029.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-01-21T11:38:50.351 INFO:teuthology.orchestra.run.smithi029.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-01-21T11:38:50.351 INFO:teuthology.orchestra.run.smithi029.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-01-21T11:38:50.351 INFO:teuthology.orchestra.run.smithi029.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-01-21T11:38:50.351 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-21T11:38:50.351 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-01-21T11:38:50.589 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-base 2024-01-21T11:38:51.101 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-01-21T11:38:51.101 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:38:51.101 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-01-21T11:38:51.102 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-2:16.2.4-0.el8.x86_64 2024-01-21T11:38:51.102 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2024-01-21T11:38:51.102 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2024-01-21T11:38:51.102 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2024-01-21T11:38:51.102 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2024-01-21T11:38:51.102 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2024-01-21T11:38:51.102 INFO:teuthology.orchestra.run.smithi134.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-01-21T11:38:51.102 INFO:teuthology.orchestra.run.smithi134.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-01-21T11:38:51.102 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-01-21T11:38:51.102 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-01-21T11:38:51.102 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-01-21T11:38:51.102 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-01-21T11:38:51.102 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-6.2-6.el8.noarch 2024-01-21T11:38:51.102 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-01-21T11:38:51.102 INFO:teuthology.orchestra.run.smithi134.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-01-21T11:38:51.103 INFO:teuthology.orchestra.run.smithi134.stdout: python3-mako-1.0.6-14.el8.noarch 2024-01-21T11:38:51.103 INFO:teuthology.orchestra.run.smithi134.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-01-21T11:38:51.103 INFO:teuthology.orchestra.run.smithi134.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-01-21T11:38:51.103 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-01-21T11:38:51.103 INFO:teuthology.orchestra.run.smithi134.stdout: python3-portend-2.6-1.el8.noarch 2024-01-21T11:38:51.103 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-01-21T11:38:51.103 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pycparser-2.14-14.el8.noarch 2024-01-21T11:38:51.103 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pytz-2017.2-11.el8.noarch 2024-01-21T11:38:51.103 INFO:teuthology.orchestra.run.smithi134.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-01-21T11:38:51.103 INFO:teuthology.orchestra.run.smithi134.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-01-21T11:38:51.103 INFO:teuthology.orchestra.run.smithi134.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-01-21T11:38:51.103 INFO:teuthology.orchestra.run.smithi134.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-01-21T11:38:51.103 INFO:teuthology.orchestra.run.smithi134.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-01-21T11:38:51.103 INFO:teuthology.orchestra.run.smithi134.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-01-21T11:38:51.103 INFO:teuthology.orchestra.run.smithi134.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-01-21T11:38:51.104 INFO:teuthology.orchestra.run.smithi134.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-01-21T11:38:51.104 INFO:teuthology.orchestra.run.smithi134.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-01-21T11:38:51.104 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:38:51.104 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-21T11:38:51.350 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:21 ago on Sun 21 Jan 2024 11:37:30 AM UTC. 2024-01-21T11:38:52.300 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-base 2024-01-21T11:38:52.882 INFO:teuthology.orchestra.run.smithi029.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T11:38:52.931 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-01-21T11:38:52.933 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2024-01-21T11:38:52.933 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-01-21T11:38:53.037 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:16 ago on Sun 21 Jan 2024 11:37:37 AM UTC. 2024-01-21T11:38:53.047 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install cephadm 2024-01-21T11:38:53.808 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:23 ago on Sun 21 Jan 2024 11:37:30 AM UTC. 2024-01-21T11:38:54.477 INFO:teuthology.orchestra.run.smithi134.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T11:38:54.523 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-21T11:38:54.524 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-01-21T11:38:54.524 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-21T11:38:54.639 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install cephadm 2024-01-21T11:38:55.360 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:18 ago on Sun 21 Jan 2024 11:37:37 AM UTC. 2024-01-21T11:38:55.405 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-01-21T11:38:55.406 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-21T11:38:55.406 INFO:teuthology.orchestra.run.smithi029.stdout: Package Architecture Version Repository Size 2024-01-21T11:38:55.406 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-21T11:38:55.407 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-01-21T11:38:55.407 INFO:teuthology.orchestra.run.smithi029.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2024-01-21T11:38:55.407 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-21T11:38:55.407 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-01-21T11:38:55.407 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-21T11:38:55.407 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2024-01-21T11:38:55.407 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-21T11:38:55.407 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 71 k 2024-01-21T11:38:55.407 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 290 k 2024-01-21T11:38:55.407 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-01-21T11:38:55.456 INFO:teuthology.orchestra.run.smithi029.stdout:cephadm-16.2.4-0.el8.noarch.rpm 1.4 MB/s | 71 kB 00:00 2024-01-21T11:38:55.456 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-01-21T11:38:55.456 INFO:teuthology.orchestra.run.smithi029.stdout:Total 1.4 MB/s | 71 kB 00:00 2024-01-21T11:38:55.456 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-01-21T11:38:55.464 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-01-21T11:38:55.464 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-01-21T11:38:55.466 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-01-21T11:38:55.469 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-01-21T11:38:55.510 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-01-21T11:38:56.034 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-21T11:38:56.192 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-21T11:38:56.432 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-21T11:38:56.814 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-21T11:38:56.814 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-21T11:38:56.814 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-01-21T11:38:56.814 INFO:teuthology.orchestra.run.smithi029.stdout: cephadm-2:16.2.4-0.el8.noarch 2024-01-21T11:38:56.814 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-21T11:38:56.814 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-01-21T11:38:56.855 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-21T11:38:56.856 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-21T11:38:56.856 INFO:teuthology.orchestra.run.smithi134.stdout: Package Architecture Version Repository Size 2024-01-21T11:38:56.856 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-21T11:38:56.856 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-01-21T11:38:56.856 INFO:teuthology.orchestra.run.smithi134.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2024-01-21T11:38:56.856 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:38:56.856 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-21T11:38:56.856 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-21T11:38:56.856 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-01-21T11:38:56.856 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:38:56.856 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 71 k 2024-01-21T11:38:56.857 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 290 k 2024-01-21T11:38:56.857 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-01-21T11:38:56.908 INFO:teuthology.orchestra.run.smithi134.stdout:cephadm-16.2.4-0.el8.noarch.rpm 1.4 MB/s | 71 kB 00:00 2024-01-21T11:38:56.909 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-21T11:38:56.909 INFO:teuthology.orchestra.run.smithi134.stdout:Total 1.3 MB/s | 71 kB 00:00 2024-01-21T11:38:56.909 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-01-21T11:38:56.916 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-01-21T11:38:56.916 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-01-21T11:38:56.917 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-01-21T11:38:56.920 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-01-21T11:38:56.966 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-01-21T11:38:57.005 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-immutable-object-cache 2024-01-21T11:38:57.767 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:27 ago on Sun 21 Jan 2024 11:37:30 AM UTC. 2024-01-21T11:38:58.376 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-21T11:38:58.553 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-21T11:38:58.793 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-21T11:38:59.165 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-21T11:38:59.165 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:38:59.165 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-01-21T11:38:59.165 INFO:teuthology.orchestra.run.smithi134.stdout: cephadm-2:16.2.4-0.el8.noarch 2024-01-21T11:38:59.166 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:38:59.166 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-21T11:38:59.354 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-01-21T11:38:59.355 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-21T11:38:59.355 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repo Size 2024-01-21T11:38:59.355 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-21T11:38:59.355 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-01-21T11:38:59.355 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2024-01-21T11:38:59.355 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-21T11:38:59.355 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-01-21T11:38:59.356 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-21T11:38:59.356 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2024-01-21T11:38:59.356 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-21T11:38:59.356 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 163 k 2024-01-21T11:38:59.356 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 587 k 2024-01-21T11:38:59.356 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-01-21T11:38:59.373 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-immutable-object-cache 2024-01-21T11:38:59.835 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 341 kB/s | 163 kB 00:00 2024-01-21T11:38:59.835 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-01-21T11:38:59.835 INFO:teuthology.orchestra.run.smithi029.stdout:Total 340 kB/s | 163 kB 00:00 2024-01-21T11:38:59.836 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-01-21T11:38:59.849 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-01-21T11:38:59.849 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-01-21T11:38:59.905 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-01-21T11:38:59.909 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-01-21T11:39:00.111 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:23 ago on Sun 21 Jan 2024 11:37:37 AM UTC. 2024-01-21T11:39:00.721 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-01-21T11:39:01.597 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-21T11:39:01.598 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-21T11:39:01.598 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-01-21T11:39:01.598 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-21T11:39:01.598 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-01-21T11:39:01.598 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2024-01-21T11:39:01.599 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:39:01.599 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-21T11:39:01.599 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-21T11:39:01.599 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-01-21T11:39:01.599 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:39:01.599 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 163 k 2024-01-21T11:39:01.599 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 587 k 2024-01-21T11:39:01.599 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-01-21T11:39:01.646 INFO:teuthology.orchestra.run.smithi134.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 3.4 MB/s | 163 kB 00:00 2024-01-21T11:39:01.646 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-21T11:39:01.647 INFO:teuthology.orchestra.run.smithi134.stdout:Total 3.3 MB/s | 163 kB 00:00 2024-01-21T11:39:01.647 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-01-21T11:39:01.657 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-01-21T11:39:01.657 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-01-21T11:39:01.706 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-01-21T11:39:01.709 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-01-21T11:39:01.823 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-01-21T11:39:02.015 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T11:39:02.424 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T11:39:02.782 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T11:39:02.782 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:39:02.782 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-01-21T11:39:02.782 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2024-01-21T11:39:02.782 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:39:02.782 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-21T11:39:03.021 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-mgr 2024-01-21T11:39:03.479 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T11:39:03.758 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:26 ago on Sun 21 Jan 2024 11:37:37 AM UTC. 2024-01-21T11:39:03.938 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T11:39:04.339 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T11:39:04.339 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-21T11:39:04.339 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-01-21T11:39:04.339 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2024-01-21T11:39:04.339 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-21T11:39:04.339 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-01-21T11:39:04.556 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-mgr 2024-01-21T11:39:05.197 INFO:teuthology.orchestra.run.smithi134.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T11:39:05.243 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-21T11:39:05.244 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-01-21T11:39:05.244 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-21T11:39:05.312 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:35 ago on Sun 21 Jan 2024 11:37:30 AM UTC. 2024-01-21T11:39:05.347 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-mgr-dashboard 2024-01-21T11:39:06.068 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:29 ago on Sun 21 Jan 2024 11:37:37 AM UTC. 2024-01-21T11:39:06.850 INFO:teuthology.orchestra.run.smithi029.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T11:39:06.900 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-01-21T11:39:06.901 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2024-01-21T11:39:06.901 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-01-21T11:39:07.018 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-mgr-dashboard 2024-01-21T11:39:07.544 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-21T11:39:07.545 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-21T11:39:07.546 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-01-21T11:39:07.546 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-21T11:39:07.546 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-01-21T11:39:07.546 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2024-01-21T11:39:07.546 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-01-21T11:39:07.546 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2024-01-21T11:39:07.546 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2024-01-21T11:39:07.546 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-01-21T11:39:07.546 INFO:teuthology.orchestra.run.smithi134.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-01-21T11:39:07.546 INFO:teuthology.orchestra.run.smithi134.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-01-21T11:39:07.546 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:39:07.546 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-21T11:39:07.546 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-21T11:39:07.546 INFO:teuthology.orchestra.run.smithi134.stdout:Install 6 Packages 2024-01-21T11:39:07.547 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:39:07.547 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 3.0 M 2024-01-21T11:39:07.547 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 15 M 2024-01-21T11:39:07.547 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-01-21T11:39:07.770 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:37 ago on Sun 21 Jan 2024 11:37:30 AM UTC. 2024-01-21T11:39:08.549 INFO:teuthology.orchestra.run.smithi134.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 578 kB/s | 24 kB 00:00 2024-01-21T11:39:08.574 INFO:teuthology.orchestra.run.smithi134.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 134 kB/s | 8.9 kB 00:00 2024-01-21T11:39:08.641 INFO:teuthology.orchestra.run.smithi134.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 20 MB/s | 2.7 MB 00:00 2024-01-21T11:39:08.758 INFO:teuthology.orchestra.run.smithi134.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 204 kB/s | 43 kB 00:00 2024-01-21T11:39:08.866 INFO:teuthology.orchestra.run.smithi134.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 115 kB/s | 34 kB 00:00 2024-01-21T11:39:09.017 INFO:teuthology.orchestra.run.smithi134.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 530 kB/s | 196 kB 00:00 2024-01-21T11:39:09.020 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-21T11:39:09.020 INFO:teuthology.orchestra.run.smithi134.stdout:Total 2.0 MB/s | 3.0 MB 00:01 2024-01-21T11:39:09.032 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-01-21T11:39:09.042 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-01-21T11:39:09.043 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-01-21T11:39:09.224 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-01-21T11:39:09.227 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-01-21T11:39:09.360 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-01-21T11:39:09.360 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-21T11:39:09.361 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2024-01-21T11:39:09.361 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-21T11:39:09.361 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-01-21T11:39:09.361 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2024-01-21T11:39:09.361 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-01-21T11:39:09.361 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2024-01-21T11:39:09.361 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2024-01-21T11:39:09.361 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-01-21T11:39:09.361 INFO:teuthology.orchestra.run.smithi029.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-01-21T11:39:09.361 INFO:teuthology.orchestra.run.smithi029.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-01-21T11:39:09.361 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-21T11:39:09.361 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-01-21T11:39:09.361 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-21T11:39:09.361 INFO:teuthology.orchestra.run.smithi029.stdout:Install 6 Packages 2024-01-21T11:39:09.362 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-21T11:39:09.362 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 3.0 M 2024-01-21T11:39:09.362 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 15 M 2024-01-21T11:39:09.362 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-01-21T11:39:09.492 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-01-21T11:39:09.736 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-01-21T11:39:09.945 INFO:teuthology.orchestra.run.smithi029.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 411 kB/s | 24 kB 00:00 2024-01-21T11:39:09.970 INFO:teuthology.orchestra.run.smithi029.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 107 kB/s | 8.9 kB 00:00 2024-01-21T11:39:10.006 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-01-21T11:39:10.054 INFO:teuthology.orchestra.run.smithi029.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 16 MB/s | 2.7 MB 00:00 2024-01-21T11:39:10.180 INFO:teuthology.orchestra.run.smithi029.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 181 kB/s | 43 kB 00:00 2024-01-21T11:39:10.199 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-01-21T11:39:10.347 INFO:teuthology.orchestra.run.smithi029.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 89 kB/s | 34 kB 00:00 2024-01-21T11:39:10.353 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2024-01-21T11:39:10.606 INFO:teuthology.orchestra.run.smithi029.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 360 kB/s | 196 kB 00:00 2024-01-21T11:39:10.609 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-01-21T11:39:10.609 INFO:teuthology.orchestra.run.smithi029.stdout:Total 2.4 MB/s | 3.0 MB 00:01 2024-01-21T11:39:10.621 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-01-21T11:39:10.632 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-01-21T11:39:10.632 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-01-21T11:39:10.825 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-01-21T11:39:10.829 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-01-21T11:39:11.104 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-01-21T11:39:11.319 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-01-21T11:39:11.322 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2024-01-21T11:39:11.580 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-01-21T11:39:11.596 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-01-21T11:39:11.787 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-01-21T11:39:11.909 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2024-01-21T11:39:11.916 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-01-21T11:39:11.916 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2024-01-21T11:39:11.917 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2024-01-21T11:39:11.917 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2024-01-21T11:39:11.917 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-01-21T11:39:11.917 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-01-21T11:39:12.315 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-01-21T11:39:12.315 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:39:12.315 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-01-21T11:39:12.315 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2024-01-21T11:39:12.315 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2024-01-21T11:39:12.315 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2024-01-21T11:39:12.315 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-01-21T11:39:12.315 INFO:teuthology.orchestra.run.smithi134.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-01-21T11:39:12.315 INFO:teuthology.orchestra.run.smithi134.stdout: python3-routes-2.4.1-12.el8.noarch 2024-01-21T11:39:12.316 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:39:12.316 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-21T11:39:12.469 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-mgr-diskprediction-local 2024-01-21T11:39:12.949 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2024-01-21T11:39:13.195 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:36 ago on Sun 21 Jan 2024 11:37:37 AM UTC. 2024-01-21T11:39:13.202 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-01-21T11:39:13.506 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-01-21T11:39:13.507 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2024-01-21T11:39:13.507 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2024-01-21T11:39:13.507 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2024-01-21T11:39:13.507 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-01-21T11:39:13.507 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-01-21T11:39:14.239 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-01-21T11:39:14.240 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-21T11:39:14.240 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-01-21T11:39:14.240 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2024-01-21T11:39:14.240 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2024-01-21T11:39:14.240 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2024-01-21T11:39:14.240 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-01-21T11:39:14.240 INFO:teuthology.orchestra.run.smithi029.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-01-21T11:39:14.240 INFO:teuthology.orchestra.run.smithi029.stdout: python3-routes-2.4.1-12.el8.noarch 2024-01-21T11:39:14.240 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-21T11:39:14.240 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-01-21T11:39:14.367 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-mgr-diskprediction-local 2024-01-21T11:39:14.672 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-21T11:39:14.672 INFO:teuthology.orchestra.run.smithi134.stdout:============================================================================================ 2024-01-21T11:39:14.673 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-01-21T11:39:14.673 INFO:teuthology.orchestra.run.smithi134.stdout:============================================================================================ 2024-01-21T11:39:14.673 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-01-21T11:39:14.673 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2024-01-21T11:39:14.673 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-01-21T11:39:14.673 INFO:teuthology.orchestra.run.smithi134.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-01-21T11:39:14.674 INFO:teuthology.orchestra.run.smithi134.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-01-21T11:39:14.674 INFO:teuthology.orchestra.run.smithi134.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-01-21T11:39:14.674 INFO:teuthology.orchestra.run.smithi134.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-01-21T11:39:14.674 INFO:teuthology.orchestra.run.smithi134.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-01-21T11:39:14.674 INFO:teuthology.orchestra.run.smithi134.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-01-21T11:39:14.674 INFO:teuthology.orchestra.run.smithi134.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-01-21T11:39:14.674 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:39:14.675 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-21T11:39:14.675 INFO:teuthology.orchestra.run.smithi134.stdout:============================================================================================ 2024-01-21T11:39:14.675 INFO:teuthology.orchestra.run.smithi134.stdout:Install 8 Packages 2024-01-21T11:39:14.675 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:39:14.676 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 37 M 2024-01-21T11:39:14.676 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 210 M 2024-01-21T11:39:14.677 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-01-21T11:39:15.128 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:45 ago on Sun 21 Jan 2024 11:37:30 AM UTC. 2024-01-21T11:39:15.222 INFO:teuthology.orchestra.run.smithi134.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 48 MB/s | 8.6 MB 00:00 2024-01-21T11:39:16.715 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-01-21T11:39:16.717 INFO:teuthology.orchestra.run.smithi029.stdout:============================================================================================ 2024-01-21T11:39:16.717 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2024-01-21T11:39:16.717 INFO:teuthology.orchestra.run.smithi029.stdout:============================================================================================ 2024-01-21T11:39:16.717 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-01-21T11:39:16.717 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2024-01-21T11:39:16.717 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-01-21T11:39:16.717 INFO:teuthology.orchestra.run.smithi029.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-01-21T11:39:16.717 INFO:teuthology.orchestra.run.smithi029.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-01-21T11:39:16.717 INFO:teuthology.orchestra.run.smithi029.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-01-21T11:39:16.717 INFO:teuthology.orchestra.run.smithi029.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-01-21T11:39:16.717 INFO:teuthology.orchestra.run.smithi029.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-01-21T11:39:16.717 INFO:teuthology.orchestra.run.smithi029.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-01-21T11:39:16.717 INFO:teuthology.orchestra.run.smithi029.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-01-21T11:39:16.718 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-21T11:39:16.718 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-01-21T11:39:16.718 INFO:teuthology.orchestra.run.smithi029.stdout:============================================================================================ 2024-01-21T11:39:16.718 INFO:teuthology.orchestra.run.smithi029.stdout:Install 8 Packages 2024-01-21T11:39:16.718 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-21T11:39:16.718 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 37 M 2024-01-21T11:39:16.718 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 210 M 2024-01-21T11:39:16.719 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-01-21T11:39:16.822 INFO:teuthology.orchestra.run.smithi134.stdout:(2/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 2.3 MB/s | 3.7 MB 00:01 2024-01-21T11:39:16.981 INFO:teuthology.orchestra.run.smithi134.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 1.4 MB/s | 225 kB 00:00 2024-01-21T11:39:17.021 INFO:teuthology.orchestra.run.smithi029.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 39 MB/s | 8.6 MB 00:00 2024-01-21T11:39:17.056 INFO:teuthology.orchestra.run.smithi134.stdout:(4/8): openblas-0.3.15-4.el8.x86_64.rpm 2.3 MB/s | 4.7 MB 00:02 2024-01-21T11:39:17.123 INFO:teuthology.orchestra.run.smithi134.stdout:(5/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 2.3 MB/s | 4.9 MB 00:02 2024-01-21T11:39:17.315 INFO:teuthology.orchestra.run.smithi134.stdout:(6/8): libquadmath-8.5.0-21.el8.x86_64.rpm 901 kB/s | 172 kB 00:00 2024-01-21T11:39:17.348 INFO:teuthology.orchestra.run.smithi134.stdout:(7/8): libgfortran-8.5.0-21.el8.x86_64.rpm 2.2 MB/s | 645 kB 00:00 2024-01-21T11:39:18.567 INFO:teuthology.orchestra.run.smithi029.stdout:(2/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 2.4 MB/s | 3.7 MB 00:01 2024-01-21T11:39:18.691 INFO:teuthology.orchestra.run.smithi029.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 1.8 MB/s | 225 kB 00:00 2024-01-21T11:39:18.875 INFO:teuthology.orchestra.run.smithi029.stdout:(4/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 2.3 MB/s | 4.9 MB 00:02 2024-01-21T11:39:19.167 INFO:teuthology.orchestra.run.smithi029.stdout:(5/8): libgfortran-8.5.0-21.el8.x86_64.rpm 2.2 MB/s | 645 kB 00:00 2024-01-21T11:39:19.226 INFO:teuthology.orchestra.run.smithi029.stdout:(6/8): libquadmath-8.5.0-21.el8.x86_64.rpm 2.9 MB/s | 172 kB 00:00 2024-01-21T11:39:20.121 INFO:teuthology.orchestra.run.smithi029.stdout:(7/8): openblas-0.3.15-4.el8.x86_64.rpm 1.4 MB/s | 4.7 MB 00:03 2024-01-21T11:39:20.642 INFO:teuthology.orchestra.run.smithi134.stdout:(8/8): python3-scipy-1.0.0-21.module_el8+762+77 3.9 MB/s | 14 MB 00:03 2024-01-21T11:39:20.642 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-21T11:39:20.643 INFO:teuthology.orchestra.run.smithi134.stdout:Total 6.2 MB/s | 37 MB 00:05 2024-01-21T11:39:20.877 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-01-21T11:39:20.896 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-01-21T11:39:20.896 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-01-21T11:39:21.231 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-01-21T11:39:21.233 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-01-21T11:39:21.836 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-01-21T11:39:21.995 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-01-21T11:39:22.118 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-01-21T11:39:22.295 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-01-21T11:39:22.811 INFO:teuthology.orchestra.run.smithi029.stdout:(8/8): python3-scipy-1.0.0-21.module_el8+762+77 3.4 MB/s | 14 MB 00:04 2024-01-21T11:39:22.813 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-01-21T11:39:22.813 INFO:teuthology.orchestra.run.smithi029.stdout:Total 6.1 MB/s | 37 MB 00:06 2024-01-21T11:39:22.880 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-01-21T11:39:23.054 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-01-21T11:39:23.073 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-01-21T11:39:23.073 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-01-21T11:39:23.103 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-01-21T11:39:23.421 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-01-21T11:39:23.425 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-01-21T11:39:23.699 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-01-21T11:39:23.887 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-01-21T11:39:24.057 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-01-21T11:39:24.201 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-01-21T11:39:24.338 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-01-21T11:39:24.475 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-01-21T11:39:24.556 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-01-21T11:39:24.898 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-01-21T11:39:25.104 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-01-21T11:39:25.170 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-01-21T11:39:25.301 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-01-21T11:39:25.950 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-01-21T11:39:26.125 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-01-21T11:39:26.882 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-01-21T11:39:26.947 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-01-21T11:39:27.242 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-01-21T11:39:27.492 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-01-21T11:39:28.283 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-01-21T11:39:28.494 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-01-21T11:39:28.823 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-01-21T11:39:28.823 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2024-01-21T11:39:28.823 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-01-21T11:39:28.823 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-01-21T11:39:28.823 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-01-21T11:39:28.823 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-01-21T11:39:28.823 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-01-21T11:39:28.823 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-01-21T11:39:29.236 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-01-21T11:39:29.236 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:39:29.236 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-01-21T11:39:29.236 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2024-01-21T11:39:29.236 INFO:teuthology.orchestra.run.smithi134.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-01-21T11:39:29.236 INFO:teuthology.orchestra.run.smithi134.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-01-21T11:39:29.236 INFO:teuthology.orchestra.run.smithi134.stdout: openblas-0.3.15-4.el8.x86_64 2024-01-21T11:39:29.236 INFO:teuthology.orchestra.run.smithi134.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-01-21T11:39:29.236 INFO:teuthology.orchestra.run.smithi134.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-01-21T11:39:29.236 INFO:teuthology.orchestra.run.smithi134.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-01-21T11:39:29.236 INFO:teuthology.orchestra.run.smithi134.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-01-21T11:39:29.236 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:39:29.237 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-21T11:39:29.398 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-mgr-rook 2024-01-21T11:39:29.409 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-01-21T11:39:30.128 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:53 ago on Sun 21 Jan 2024 11:37:37 AM UTC. 2024-01-21T11:39:30.790 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-01-21T11:39:31.016 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-01-21T11:39:31.383 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-01-21T11:39:31.383 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2024-01-21T11:39:31.383 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-01-21T11:39:31.383 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-01-21T11:39:31.383 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-01-21T11:39:31.383 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-01-21T11:39:31.383 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-01-21T11:39:31.383 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-01-21T11:39:31.599 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-21T11:39:31.600 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-21T11:39:31.601 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-01-21T11:39:31.601 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-21T11:39:31.601 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-01-21T11:39:31.601 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2024-01-21T11:39:31.601 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-01-21T11:39:31.601 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-01-21T11:39:31.601 INFO:teuthology.orchestra.run.smithi134.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-01-21T11:39:31.601 INFO:teuthology.orchestra.run.smithi134.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-01-21T11:39:31.601 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-01-21T11:39:31.601 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-01-21T11:39:31.601 INFO:teuthology.orchestra.run.smithi134.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-01-21T11:39:31.601 INFO:teuthology.orchestra.run.smithi134.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-01-21T11:39:31.601 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-01-21T11:39:31.601 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-01-21T11:39:31.602 INFO:teuthology.orchestra.run.smithi134.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-01-21T11:39:31.602 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-01-21T11:39:31.602 INFO:teuthology.orchestra.run.smithi134.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-01-21T11:39:31.602 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:39:31.602 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-21T11:39:31.602 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-21T11:39:31.602 INFO:teuthology.orchestra.run.smithi134.stdout:Install 13 Packages 2024-01-21T11:39:31.602 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:39:31.603 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 1.9 M 2024-01-21T11:39:31.603 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 24 M 2024-01-21T11:39:31.603 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-01-21T11:39:31.813 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-01-21T11:39:31.814 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-21T11:39:31.814 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-01-21T11:39:31.814 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2024-01-21T11:39:31.814 INFO:teuthology.orchestra.run.smithi029.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-01-21T11:39:31.814 INFO:teuthology.orchestra.run.smithi029.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-01-21T11:39:31.814 INFO:teuthology.orchestra.run.smithi029.stdout: openblas-0.3.15-4.el8.x86_64 2024-01-21T11:39:31.814 INFO:teuthology.orchestra.run.smithi029.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-01-21T11:39:31.814 INFO:teuthology.orchestra.run.smithi029.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-01-21T11:39:31.814 INFO:teuthology.orchestra.run.smithi029.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-01-21T11:39:31.814 INFO:teuthology.orchestra.run.smithi029.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-01-21T11:39:31.814 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-21T11:39:31.814 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-01-21T11:39:31.905 INFO:teuthology.orchestra.run.smithi134.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 745 kB/s | 27 kB 00:00 2024-01-21T11:39:31.921 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-mgr-rook 2024-01-21T11:39:32.055 INFO:teuthology.orchestra.run.smithi134.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 107 kB/s | 20 kB 00:00 2024-01-21T11:39:32.080 INFO:teuthology.orchestra.run.smithi134.stdout:(3/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 125 kB/s | 27 kB 00:00 2024-01-21T11:39:32.239 INFO:teuthology.orchestra.run.smithi134.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 599 kB/s | 110 kB 00:00 2024-01-21T11:39:32.272 INFO:teuthology.orchestra.run.smithi134.stdout:(5/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 344 kB/s | 126 kB 00:00 2024-01-21T11:39:32.297 INFO:teuthology.orchestra.run.smithi134.stdout:(6/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 713 kB/s | 155 kB 00:00 2024-01-21T11:39:32.398 INFO:teuthology.orchestra.run.smithi134.stdout:(7/13): python3-requests-oauthlib-1.0.0-1.el8.n 269 kB/s | 43 kB 00:00 2024-01-21T11:39:32.584 INFO:teuthology.orchestra.run.smithi134.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 57 kB/s | 16 kB 00:00 2024-01-21T11:39:32.609 INFO:teuthology.orchestra.run.smithi134.stdout:(9/13): python3-cachetools-3.1.1-4.el8.noarch.r 98 kB/s | 33 kB 00:00 2024-01-21T11:39:32.680 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:02:02 ago on Sun 21 Jan 2024 11:37:30 AM UTC. 2024-01-21T11:39:32.776 INFO:teuthology.orchestra.run.smithi134.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 215 kB/s | 81 kB 00:00 2024-01-21T11:39:32.809 INFO:teuthology.orchestra.run.smithi134.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 321 kB/s | 64 kB 00:00 2024-01-21T11:39:32.926 INFO:teuthology.orchestra.run.smithi134.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 407 kB/s | 61 kB 00:00 2024-01-21T11:39:33.102 INFO:teuthology.orchestra.run.smithi134.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 2.3 MB/s | 1.2 MB 00:00 2024-01-21T11:39:33.102 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-21T11:39:33.102 INFO:teuthology.orchestra.run.smithi134.stdout:Total 1.3 MB/s | 1.9 MB 00:01 2024-01-21T11:39:33.159 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-01-21T11:39:33.175 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-01-21T11:39:33.175 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-01-21T11:39:33.263 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-01-21T11:39:33.266 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-01-21T11:39:33.781 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-01-21T11:39:34.029 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-01-21T11:39:34.224 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-01-21T11:39:34.260 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-01-21T11:39:34.262 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-21T11:39:34.262 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2024-01-21T11:39:34.262 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-21T11:39:34.262 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-01-21T11:39:34.262 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2024-01-21T11:39:34.263 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-01-21T11:39:34.263 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-01-21T11:39:34.263 INFO:teuthology.orchestra.run.smithi029.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-01-21T11:39:34.263 INFO:teuthology.orchestra.run.smithi029.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-01-21T11:39:34.263 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-01-21T11:39:34.263 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-01-21T11:39:34.263 INFO:teuthology.orchestra.run.smithi029.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-01-21T11:39:34.263 INFO:teuthology.orchestra.run.smithi029.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-01-21T11:39:34.263 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-01-21T11:39:34.263 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-01-21T11:39:34.263 INFO:teuthology.orchestra.run.smithi029.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-01-21T11:39:34.263 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-01-21T11:39:34.264 INFO:teuthology.orchestra.run.smithi029.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-01-21T11:39:34.264 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-21T11:39:34.264 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-01-21T11:39:34.264 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-21T11:39:34.264 INFO:teuthology.orchestra.run.smithi029.stdout:Install 13 Packages 2024-01-21T11:39:34.264 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-21T11:39:34.264 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 1.9 M 2024-01-21T11:39:34.264 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 24 M 2024-01-21T11:39:34.265 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-01-21T11:39:34.478 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-01-21T11:39:34.589 INFO:teuthology.orchestra.run.smithi029.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 511 kB/s | 27 kB 00:00 2024-01-21T11:39:34.691 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-01-21T11:39:34.714 INFO:teuthology.orchestra.run.smithi029.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 112 kB/s | 20 kB 00:00 2024-01-21T11:39:34.740 INFO:teuthology.orchestra.run.smithi029.stdout:(3/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 130 kB/s | 27 kB 00:00 2024-01-21T11:39:34.886 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-01-21T11:39:34.890 INFO:teuthology.orchestra.run.smithi029.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 625 kB/s | 110 kB 00:00 2024-01-21T11:39:34.916 INFO:teuthology.orchestra.run.smithi029.stdout:(5/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 386 kB/s | 126 kB 00:00 2024-01-21T11:39:34.942 INFO:teuthology.orchestra.run.smithi029.stdout:(6/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 769 kB/s | 155 kB 00:00 2024-01-21T11:39:35.076 INFO:teuthology.orchestra.run.smithi029.stdout:(7/13): python3-requests-oauthlib-1.0.0-1.el8.n 231 kB/s | 43 kB 00:00 2024-01-21T11:39:35.129 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-01-21T11:39:35.251 INFO:teuthology.orchestra.run.smithi029.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 53 kB/s | 16 kB 00:00 2024-01-21T11:39:35.285 INFO:teuthology.orchestra.run.smithi029.stdout:(9/13): python3-cachetools-3.1.1-4.el8.noarch.r 89 kB/s | 33 kB 00:00 2024-01-21T11:39:35.413 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-01-21T11:39:35.444 INFO:teuthology.orchestra.run.smithi029.stdout:(10/13): python3-rsa-4.9-2.el8.noarch.rpm 405 kB/s | 64 kB 00:00 2024-01-21T11:39:35.478 INFO:teuthology.orchestra.run.smithi029.stdout:(11/13): python3-google-auth-1.1.1-10.el8.noarc 202 kB/s | 81 kB 00:00 2024-01-21T11:39:35.553 INFO:teuthology.orchestra.run.smithi029.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 562 kB/s | 61 kB 00:00 2024-01-21T11:39:35.689 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-01-21T11:39:35.728 INFO:teuthology.orchestra.run.smithi029.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 2.4 MB/s | 1.2 MB 00:00 2024-01-21T11:39:35.732 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-01-21T11:39:35.732 INFO:teuthology.orchestra.run.smithi029.stdout:Total 1.3 MB/s | 1.9 MB 00:01 2024-01-21T11:39:35.791 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-01-21T11:39:35.808 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-01-21T11:39:35.808 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-01-21T11:39:35.904 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-01-21T11:39:35.907 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-01-21T11:39:36.733 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-01-21T11:39:36.842 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-01-21T11:39:37.033 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-01-21T11:39:37.088 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-01-21T11:39:37.267 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-01-21T11:39:37.309 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-01-21T11:39:37.451 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-01-21T11:39:37.528 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-01-21T11:39:37.618 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-01-21T11:39:37.740 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-01-21T11:39:37.920 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-01-21T11:39:37.920 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2024-01-21T11:39:37.920 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-01-21T11:39:37.920 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-01-21T11:39:37.920 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-01-21T11:39:37.920 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-01-21T11:39:37.920 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-01-21T11:39:37.920 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-01-21T11:39:37.920 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-01-21T11:39:37.920 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-01-21T11:39:37.920 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-01-21T11:39:37.920 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-01-21T11:39:37.921 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-01-21T11:39:37.971 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-01-21T11:39:38.216 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-01-21T11:39:38.368 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-01-21T11:39:38.368 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:39:38.368 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-01-21T11:39:38.368 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2024-01-21T11:39:38.368 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-01-21T11:39:38.368 INFO:teuthology.orchestra.run.smithi134.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-01-21T11:39:38.369 INFO:teuthology.orchestra.run.smithi134.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-01-21T11:39:38.369 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-01-21T11:39:38.369 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-01-21T11:39:38.369 INFO:teuthology.orchestra.run.smithi134.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-01-21T11:39:38.369 INFO:teuthology.orchestra.run.smithi134.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-01-21T11:39:38.369 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-01-21T11:39:38.369 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-01-21T11:39:38.369 INFO:teuthology.orchestra.run.smithi134.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-01-21T11:39:38.369 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rsa-4.9-2.el8.noarch 2024-01-21T11:39:38.369 INFO:teuthology.orchestra.run.smithi134.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-01-21T11:39:38.369 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:39:38.369 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-21T11:39:38.491 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-mgr-cephadm 2024-01-21T11:39:38.513 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-01-21T11:39:38.757 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-01-21T11:39:39.219 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:02 ago on Sun 21 Jan 2024 11:37:37 AM UTC. 2024-01-21T11:39:39.853 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-01-21T11:39:40.131 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-01-21T11:39:40.340 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-01-21T11:39:40.524 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-01-21T11:39:40.682 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-01-21T11:39:40.692 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-21T11:39:40.693 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-21T11:39:40.693 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-01-21T11:39:40.693 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-21T11:39:40.693 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-01-21T11:39:40.693 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2024-01-21T11:39:40.693 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-01-21T11:39:40.694 INFO:teuthology.orchestra.run.smithi134.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2024-01-21T11:39:40.694 INFO:teuthology.orchestra.run.smithi134.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-01-21T11:39:40.694 INFO:teuthology.orchestra.run.smithi134.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2024-01-21T11:39:40.694 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2024-01-21T11:39:40.694 INFO:teuthology.orchestra.run.smithi134.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2024-01-21T11:39:40.694 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:39:40.694 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-21T11:39:40.694 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-21T11:39:40.694 INFO:teuthology.orchestra.run.smithi134.stdout:Install 6 Packages 2024-01-21T11:39:40.694 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:39:40.695 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 5.8 M 2024-01-21T11:39:40.695 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 24 M 2024-01-21T11:39:40.695 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-01-21T11:39:40.926 INFO:teuthology.orchestra.run.smithi134.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 2.1 MB/s | 89 kB 00:00 2024-01-21T11:39:41.013 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-01-21T11:39:41.013 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2024-01-21T11:39:41.013 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-01-21T11:39:41.013 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-01-21T11:39:41.013 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-01-21T11:39:41.013 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-01-21T11:39:41.013 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-01-21T11:39:41.013 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-01-21T11:39:41.014 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-01-21T11:39:41.014 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-01-21T11:39:41.014 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-01-21T11:39:41.014 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-01-21T11:39:41.014 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-01-21T11:39:41.068 INFO:teuthology.orchestra.run.smithi134.stdout:(2/6): python3-apipkg-1.5-6.el8.noarch.rpm 150 kB/s | 21 kB 00:00 2024-01-21T11:39:41.300 INFO:teuthology.orchestra.run.smithi134.stdout:(3/6): python3-execnet-1.7.1-1.el8.noarch.rpm 1.5 MB/s | 355 kB 00:00 2024-01-21T11:39:41.368 INFO:teuthology.orchestra.run.smithi134.stdout:(4/6): python3-remoto-1.2.1-1.el8.noarch.rpm 715 kB/s | 49 kB 00:00 2024-01-21T11:39:41.452 INFO:teuthology.orchestra.run.smithi134.stdout:(5/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 947 kB/s | 538 kB 00:00 2024-01-21T11:39:41.479 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-01-21T11:39:41.479 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-21T11:39:41.479 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-01-21T11:39:41.479 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2024-01-21T11:39:41.480 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-01-21T11:39:41.480 INFO:teuthology.orchestra.run.smithi029.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-01-21T11:39:41.480 INFO:teuthology.orchestra.run.smithi029.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-01-21T11:39:41.480 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-01-21T11:39:41.480 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-01-21T11:39:41.480 INFO:teuthology.orchestra.run.smithi029.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-01-21T11:39:41.480 INFO:teuthology.orchestra.run.smithi029.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-01-21T11:39:41.480 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-01-21T11:39:41.480 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-01-21T11:39:41.480 INFO:teuthology.orchestra.run.smithi029.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-01-21T11:39:41.480 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rsa-4.9-2.el8.noarch 2024-01-21T11:39:41.480 INFO:teuthology.orchestra.run.smithi029.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-01-21T11:39:41.480 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-21T11:39:41.480 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-01-21T11:39:41.604 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-mgr-cephadm 2024-01-21T11:39:42.366 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:02:12 ago on Sun 21 Jan 2024 11:37:30 AM UTC. 2024-01-21T11:39:42.754 INFO:teuthology.orchestra.run.smithi134.stdout:(6/6): python3-babel-2.5.1-7.el8.noarch.rpm 2.6 MB/s | 4.8 MB 00:01 2024-01-21T11:39:42.755 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-21T11:39:42.755 INFO:teuthology.orchestra.run.smithi134.stdout:Total 2.8 MB/s | 5.8 MB 00:02 2024-01-21T11:39:42.808 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-01-21T11:39:42.818 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-01-21T11:39:42.818 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-01-21T11:39:42.928 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-01-21T11:39:42.931 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-01-21T11:39:43.213 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-01-21T11:39:43.508 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2024-01-21T11:39:43.761 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2024-01-21T11:39:43.951 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-01-21T11:39:43.952 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-21T11:39:43.952 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2024-01-21T11:39:43.952 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-21T11:39:43.952 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-01-21T11:39:43.952 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2024-01-21T11:39:43.952 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-01-21T11:39:43.953 INFO:teuthology.orchestra.run.smithi029.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2024-01-21T11:39:43.953 INFO:teuthology.orchestra.run.smithi029.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-01-21T11:39:43.953 INFO:teuthology.orchestra.run.smithi029.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2024-01-21T11:39:43.953 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2024-01-21T11:39:43.953 INFO:teuthology.orchestra.run.smithi029.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2024-01-21T11:39:43.953 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-21T11:39:43.953 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-01-21T11:39:43.953 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-21T11:39:43.953 INFO:teuthology.orchestra.run.smithi029.stdout:Install 6 Packages 2024-01-21T11:39:43.953 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-21T11:39:43.954 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 5.8 M 2024-01-21T11:39:43.954 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 24 M 2024-01-21T11:39:43.954 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-01-21T11:39:44.223 INFO:teuthology.orchestra.run.smithi029.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 1.6 MB/s | 89 kB 00:00 2024-01-21T11:39:44.432 INFO:teuthology.orchestra.run.smithi029.stdout:(2/6): python3-apipkg-1.5-6.el8.noarch.rpm 102 kB/s | 21 kB 00:00 2024-01-21T11:39:44.582 INFO:teuthology.orchestra.run.smithi029.stdout:(3/6): python3-execnet-1.7.1-1.el8.noarch.rpm 2.3 MB/s | 355 kB 00:00 2024-01-21T11:39:44.641 INFO:teuthology.orchestra.run.smithi029.stdout:(4/6): python3-remoto-1.2.1-1.el8.noarch.rpm 833 kB/s | 49 kB 00:00 2024-01-21T11:39:44.675 INFO:teuthology.orchestra.run.smithi029.stdout:(5/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 1.0 MB/s | 538 kB 00:00 2024-01-21T11:39:44.692 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2024-01-21T11:39:45.054 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2024-01-21T11:39:45.280 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2024-01-21T11:39:45.535 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-01-21T11:39:45.830 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-01-21T11:39:45.830 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2024-01-21T11:39:45.830 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2024-01-21T11:39:45.830 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2024-01-21T11:39:45.830 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2024-01-21T11:39:45.830 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2024-01-21T11:39:45.984 INFO:teuthology.orchestra.run.smithi029.stdout:(6/6): python3-babel-2.5.1-7.el8.noarch.rpm 2.6 MB/s | 4.8 MB 00:01 2024-01-21T11:39:45.986 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-01-21T11:39:45.986 INFO:teuthology.orchestra.run.smithi029.stdout:Total 2.9 MB/s | 5.8 MB 00:02 2024-01-21T11:39:46.041 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-01-21T11:39:46.052 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-01-21T11:39:46.052 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-01-21T11:39:46.160 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-01-21T11:39:46.164 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-01-21T11:39:46.207 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2024-01-21T11:39:46.207 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:39:46.207 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-01-21T11:39:46.208 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2024-01-21T11:39:46.208 INFO:teuthology.orchestra.run.smithi134.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2024-01-21T11:39:46.208 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jinja2-2.10.1-3.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2024-01-21T11:39:46.208 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:39:46.208 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-21T11:39:46.357 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-fuse 2024-01-21T11:39:46.437 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-01-21T11:39:46.737 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2024-01-21T11:39:47.085 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:10 ago on Sun 21 Jan 2024 11:37:37 AM UTC. 2024-01-21T11:39:47.315 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2024-01-21T11:39:48.318 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2024-01-21T11:39:48.572 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-21T11:39:48.572 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-21T11:39:48.573 INFO:teuthology.orchestra.run.smithi134.stdout: Package Architecture Version Repository Size 2024-01-21T11:39:48.573 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-21T11:39:48.573 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-01-21T11:39:48.573 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2024-01-21T11:39:48.573 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-01-21T11:39:48.573 INFO:teuthology.orchestra.run.smithi134.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2024-01-21T11:39:48.573 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:39:48.573 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-21T11:39:48.573 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-21T11:39:48.573 INFO:teuthology.orchestra.run.smithi134.stdout:Install 2 Packages 2024-01-21T11:39:48.573 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:39:48.574 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 968 k 2024-01-21T11:39:48.574 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 3.2 M 2024-01-21T11:39:48.574 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-01-21T11:39:48.662 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2024-01-21T11:39:48.821 INFO:teuthology.orchestra.run.smithi134.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 16 MB/s | 885 kB 00:00 2024-01-21T11:39:48.888 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2024-01-21T11:39:48.938 INFO:teuthology.orchestra.run.smithi134.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 488 kB/s | 83 kB 00:00 2024-01-21T11:39:48.939 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-21T11:39:48.939 INFO:teuthology.orchestra.run.smithi134.stdout:Total 2.6 MB/s | 968 kB 00:00 2024-01-21T11:39:48.948 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-01-21T11:39:48.969 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-01-21T11:39:48.969 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-01-21T11:39:49.056 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-01-21T11:39:49.060 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-01-21T11:39:49.080 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-01-21T11:39:49.230 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-01-21T11:39:49.400 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-01-21T11:39:49.400 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2024-01-21T11:39:49.400 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2024-01-21T11:39:49.400 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2024-01-21T11:39:49.400 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2024-01-21T11:39:49.400 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2024-01-21T11:39:49.508 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-01-21T11:39:49.707 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-01-21T11:39:49.803 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2024-01-21T11:39:49.803 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-21T11:39:49.803 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-01-21T11:39:49.803 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2024-01-21T11:39:49.804 INFO:teuthology.orchestra.run.smithi029.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2024-01-21T11:39:49.804 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jinja2-2.10.1-3.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2024-01-21T11:39:49.804 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-21T11:39:49.804 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-01-21T11:39:49.936 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-fuse 2024-01-21T11:39:50.112 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-01-21T11:39:50.112 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2024-01-21T11:39:50.551 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-01-21T11:39:50.551 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:39:50.551 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-01-21T11:39:50.551 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-01-21T11:39:50.551 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:39:50.551 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-21T11:39:50.696 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:02:20 ago on Sun 21 Jan 2024 11:37:30 AM UTC. 2024-01-21T11:39:50.773 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install librados-devel 2024-01-21T11:39:51.509 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:14 ago on Sun 21 Jan 2024 11:37:37 AM UTC. 2024-01-21T11:39:52.304 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-01-21T11:39:52.307 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-21T11:39:52.307 INFO:teuthology.orchestra.run.smithi029.stdout: Package Architecture Version Repository Size 2024-01-21T11:39:52.308 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-21T11:39:52.308 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-01-21T11:39:52.308 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2024-01-21T11:39:52.308 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-01-21T11:39:52.308 INFO:teuthology.orchestra.run.smithi029.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2024-01-21T11:39:52.308 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-21T11:39:52.308 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-01-21T11:39:52.308 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-21T11:39:52.308 INFO:teuthology.orchestra.run.smithi029.stdout:Install 2 Packages 2024-01-21T11:39:52.308 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-21T11:39:52.309 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 968 k 2024-01-21T11:39:52.309 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 3.2 M 2024-01-21T11:39:52.309 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-01-21T11:39:52.568 INFO:teuthology.orchestra.run.smithi029.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 14 MB/s | 885 kB 00:00 2024-01-21T11:39:52.710 INFO:teuthology.orchestra.run.smithi029.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 403 kB/s | 83 kB 00:00 2024-01-21T11:39:52.710 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-01-21T11:39:52.710 INFO:teuthology.orchestra.run.smithi029.stdout:Total 2.3 MB/s | 968 kB 00:00 2024-01-21T11:39:52.722 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-01-21T11:39:52.745 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-01-21T11:39:52.745 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-01-21T11:39:52.842 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-01-21T11:39:52.846 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-01-21T11:39:53.000 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-21T11:39:53.001 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-21T11:39:53.001 INFO:teuthology.orchestra.run.smithi134.stdout: Package Architecture Version Repository Size 2024-01-21T11:39:53.001 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-21T11:39:53.001 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-01-21T11:39:53.001 INFO:teuthology.orchestra.run.smithi134.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2024-01-21T11:39:53.001 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:39:53.001 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-21T11:39:53.001 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-21T11:39:53.001 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-01-21T11:39:53.002 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:39:53.002 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 131 k 2024-01-21T11:39:53.002 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 439 k 2024-01-21T11:39:53.002 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-01-21T11:39:53.032 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-01-21T11:39:53.058 INFO:teuthology.orchestra.run.smithi134.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 2.3 MB/s | 131 kB 00:00 2024-01-21T11:39:53.059 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-21T11:39:53.059 INFO:teuthology.orchestra.run.smithi134.stdout:Total 2.3 MB/s | 131 kB 00:00 2024-01-21T11:39:53.059 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-01-21T11:39:53.064 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-01-21T11:39:53.064 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-01-21T11:39:53.113 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-01-21T11:39:53.116 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-01-21T11:39:53.235 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-01-21T11:39:53.335 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-01-21T11:39:53.473 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T11:39:53.540 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-01-21T11:39:53.769 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T11:39:53.948 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-01-21T11:39:53.948 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2024-01-21T11:39:54.156 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T11:39:54.157 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:39:54.157 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-01-21T11:39:54.157 INFO:teuthology.orchestra.run.smithi134.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2024-01-21T11:39:54.157 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:39:54.157 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-21T11:39:54.384 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-01-21T11:39:54.384 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-21T11:39:54.384 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-01-21T11:39:54.384 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-01-21T11:39:54.384 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-21T11:39:54.384 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-01-21T11:39:54.385 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install libcephfs2 2024-01-21T11:39:54.603 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install librados-devel 2024-01-21T11:39:55.123 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:18 ago on Sun 21 Jan 2024 11:37:37 AM UTC. 2024-01-21T11:39:55.367 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:02:25 ago on Sun 21 Jan 2024 11:37:30 AM UTC. 2024-01-21T11:39:56.569 INFO:teuthology.orchestra.run.smithi134.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T11:39:56.615 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-21T11:39:56.616 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-01-21T11:39:56.616 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-21T11:39:56.721 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install libcephfs-devel 2024-01-21T11:39:56.963 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-01-21T11:39:56.964 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-21T11:39:56.964 INFO:teuthology.orchestra.run.smithi029.stdout: Package Architecture Version Repository Size 2024-01-21T11:39:56.964 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-21T11:39:56.964 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-01-21T11:39:56.964 INFO:teuthology.orchestra.run.smithi029.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2024-01-21T11:39:56.964 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-21T11:39:56.964 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-01-21T11:39:56.964 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-21T11:39:56.965 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2024-01-21T11:39:56.965 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-21T11:39:56.965 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 131 k 2024-01-21T11:39:56.965 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 439 k 2024-01-21T11:39:56.965 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-01-21T11:39:57.012 INFO:teuthology.orchestra.run.smithi029.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 2.7 MB/s | 131 kB 00:00 2024-01-21T11:39:57.012 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-01-21T11:39:57.012 INFO:teuthology.orchestra.run.smithi029.stdout:Total 2.7 MB/s | 131 kB 00:00 2024-01-21T11:39:57.013 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-01-21T11:39:57.019 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-01-21T11:39:57.019 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-01-21T11:39:57.071 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-01-21T11:39:57.074 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-01-21T11:39:57.186 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-01-21T11:39:57.413 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T11:39:57.448 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:20 ago on Sun 21 Jan 2024 11:37:37 AM UTC. 2024-01-21T11:39:57.726 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T11:39:58.613 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T11:39:58.613 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-21T11:39:58.614 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-01-21T11:39:58.614 INFO:teuthology.orchestra.run.smithi029.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2024-01-21T11:39:58.614 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-21T11:39:58.614 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-01-21T11:39:58.934 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-21T11:39:58.935 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-21T11:39:58.935 INFO:teuthology.orchestra.run.smithi134.stdout: Package Architecture Version Repository Size 2024-01-21T11:39:58.935 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-21T11:39:58.936 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-01-21T11:39:58.936 INFO:teuthology.orchestra.run.smithi134.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2024-01-21T11:39:58.936 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:39:58.936 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-21T11:39:58.936 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-21T11:39:58.936 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-01-21T11:39:58.936 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:39:58.936 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 24 k 2024-01-21T11:39:58.936 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 94 k 2024-01-21T11:39:58.936 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-01-21T11:39:59.322 INFO:teuthology.orchestra.run.smithi134.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 63 kB/s | 24 kB 00:00 2024-01-21T11:39:59.323 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-21T11:39:59.323 INFO:teuthology.orchestra.run.smithi134.stdout:Total 63 kB/s | 24 kB 00:00 2024-01-21T11:39:59.323 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-01-21T11:39:59.327 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-01-21T11:39:59.327 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-01-21T11:39:59.329 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-01-21T11:39:59.333 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-01-21T11:39:59.384 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-01-21T11:39:59.580 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T11:39:59.835 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T11:40:00.187 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T11:40:00.187 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:40:00.187 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-01-21T11:40:00.187 INFO:teuthology.orchestra.run.smithi134.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2024-01-21T11:40:00.187 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:40:00.188 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-21T11:40:00.308 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install librados2 2024-01-21T11:40:01.035 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:24 ago on Sun 21 Jan 2024 11:37:37 AM UTC. 2024-01-21T11:40:01.337 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install libcephfs2 2024-01-21T11:40:02.106 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:02:32 ago on Sun 21 Jan 2024 11:37:30 AM UTC. 2024-01-21T11:40:02.478 INFO:teuthology.orchestra.run.smithi134.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T11:40:02.523 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-21T11:40:02.524 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-01-21T11:40:02.525 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-21T11:40:02.638 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install librbd1 2024-01-21T11:40:03.362 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:26 ago on Sun 21 Jan 2024 11:37:37 AM UTC. 2024-01-21T11:40:03.671 INFO:teuthology.orchestra.run.smithi029.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T11:40:03.721 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-01-21T11:40:03.722 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2024-01-21T11:40:03.723 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-01-21T11:40:03.847 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install libcephfs-devel 2024-01-21T11:40:04.610 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:02:34 ago on Sun 21 Jan 2024 11:37:30 AM UTC. 2024-01-21T11:40:04.807 INFO:teuthology.orchestra.run.smithi134.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T11:40:04.853 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-21T11:40:04.854 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-01-21T11:40:04.854 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-21T11:40:04.960 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install python3-rados 2024-01-21T11:40:05.690 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:28 ago on Sun 21 Jan 2024 11:37:37 AM UTC. 2024-01-21T11:40:06.207 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-01-21T11:40:06.207 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-21T11:40:06.207 INFO:teuthology.orchestra.run.smithi029.stdout: Package Architecture Version Repository Size 2024-01-21T11:40:06.207 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-21T11:40:06.207 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-01-21T11:40:06.207 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2024-01-21T11:40:06.208 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-21T11:40:06.208 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-01-21T11:40:06.208 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-21T11:40:06.208 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2024-01-21T11:40:06.208 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-21T11:40:06.208 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 24 k 2024-01-21T11:40:06.209 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 94 k 2024-01-21T11:40:06.209 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-01-21T11:40:06.259 INFO:teuthology.orchestra.run.smithi029.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 486 kB/s | 24 kB 00:00 2024-01-21T11:40:06.259 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-01-21T11:40:06.259 INFO:teuthology.orchestra.run.smithi029.stdout:Total 474 kB/s | 24 kB 00:00 2024-01-21T11:40:06.260 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-01-21T11:40:06.263 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-01-21T11:40:06.263 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-01-21T11:40:06.266 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-01-21T11:40:06.269 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-01-21T11:40:06.316 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-01-21T11:40:06.528 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T11:40:06.813 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T11:40:07.113 INFO:teuthology.orchestra.run.smithi134.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T11:40:07.158 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-21T11:40:07.159 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-01-21T11:40:07.159 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-21T11:40:07.269 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install python3-rgw 2024-01-21T11:40:07.457 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T11:40:07.458 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-21T11:40:07.458 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-01-21T11:40:07.458 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2024-01-21T11:40:07.458 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-21T11:40:07.458 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-01-21T11:40:07.583 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install librados2 2024-01-21T11:40:07.998 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:30 ago on Sun 21 Jan 2024 11:37:37 AM UTC. 2024-01-21T11:40:08.343 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:02:38 ago on Sun 21 Jan 2024 11:37:30 AM UTC. 2024-01-21T11:40:09.436 INFO:teuthology.orchestra.run.smithi134.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T11:40:09.482 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-21T11:40:09.483 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-01-21T11:40:09.483 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-21T11:40:09.625 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install python3-cephfs 2024-01-21T11:40:09.900 INFO:teuthology.orchestra.run.smithi029.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T11:40:09.949 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-01-21T11:40:09.950 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2024-01-21T11:40:09.950 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-01-21T11:40:10.065 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install librbd1 2024-01-21T11:40:10.351 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:33 ago on Sun 21 Jan 2024 11:37:37 AM UTC. 2024-01-21T11:40:10.824 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:02:40 ago on Sun 21 Jan 2024 11:37:30 AM UTC. 2024-01-21T11:40:11.781 INFO:teuthology.orchestra.run.smithi134.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T11:40:11.826 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-21T11:40:11.827 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-01-21T11:40:11.827 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-21T11:40:11.936 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install python3-rbd 2024-01-21T11:40:12.370 INFO:teuthology.orchestra.run.smithi029.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T11:40:12.418 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-01-21T11:40:12.420 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2024-01-21T11:40:12.420 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-01-21T11:40:12.538 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install python3-rados 2024-01-21T11:40:12.667 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:35 ago on Sun 21 Jan 2024 11:37:37 AM UTC. 2024-01-21T11:40:13.305 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:02:43 ago on Sun 21 Jan 2024 11:37:30 AM UTC. 2024-01-21T11:40:14.092 INFO:teuthology.orchestra.run.smithi134.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T11:40:14.137 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-21T11:40:14.138 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-01-21T11:40:14.138 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-21T11:40:14.250 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install rbd-fuse 2024-01-21T11:40:14.838 INFO:teuthology.orchestra.run.smithi029.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T11:40:14.888 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-01-21T11:40:14.889 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2024-01-21T11:40:14.889 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-01-21T11:40:14.979 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:37 ago on Sun 21 Jan 2024 11:37:37 AM UTC. 2024-01-21T11:40:15.012 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install python3-rgw 2024-01-21T11:40:15.776 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:02:45 ago on Sun 21 Jan 2024 11:37:30 AM UTC. 2024-01-21T11:40:16.462 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-21T11:40:16.462 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-21T11:40:16.462 INFO:teuthology.orchestra.run.smithi134.stdout: Package Architecture Version Repository Size 2024-01-21T11:40:16.462 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-21T11:40:16.462 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-01-21T11:40:16.463 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2024-01-21T11:40:16.463 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:40:16.463 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-21T11:40:16.463 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-21T11:40:16.463 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-01-21T11:40:16.463 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:40:16.463 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 89 k 2024-01-21T11:40:16.463 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 298 k 2024-01-21T11:40:16.464 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-01-21T11:40:16.519 INFO:teuthology.orchestra.run.smithi134.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 1.6 MB/s | 89 kB 00:00 2024-01-21T11:40:16.520 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-21T11:40:16.520 INFO:teuthology.orchestra.run.smithi134.stdout:Total 1.5 MB/s | 89 kB 00:00 2024-01-21T11:40:16.520 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-01-21T11:40:16.530 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-01-21T11:40:16.530 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-01-21T11:40:16.578 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-01-21T11:40:16.581 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-01-21T11:40:16.683 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-01-21T11:40:16.935 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T11:40:17.235 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T11:40:17.312 INFO:teuthology.orchestra.run.smithi029.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T11:40:17.362 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-01-21T11:40:17.363 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2024-01-21T11:40:17.363 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-01-21T11:40:17.488 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install python3-cephfs 2024-01-21T11:40:17.662 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T11:40:17.662 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:40:17.662 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-01-21T11:40:17.662 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2024-01-21T11:40:17.662 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:40:17.662 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-21T11:40:17.871 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install rbd-mirror 2024-01-21T11:40:18.249 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:02:48 ago on Sun 21 Jan 2024 11:37:30 AM UTC. 2024-01-21T11:40:18.604 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:41 ago on Sun 21 Jan 2024 11:37:37 AM UTC. 2024-01-21T11:40:19.793 INFO:teuthology.orchestra.run.smithi029.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T11:40:19.842 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-01-21T11:40:19.844 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2024-01-21T11:40:19.844 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-01-21T11:40:19.971 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install python3-rbd 2024-01-21T11:40:20.052 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-21T11:40:20.052 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-21T11:40:20.053 INFO:teuthology.orchestra.run.smithi134.stdout: Package Architecture Version Repository Size 2024-01-21T11:40:20.053 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-21T11:40:20.053 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-01-21T11:40:20.053 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-01-21T11:40:20.053 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:40:20.053 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-21T11:40:20.053 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-21T11:40:20.053 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-01-21T11:40:20.053 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:40:20.054 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 3.7 M 2024-01-21T11:40:20.054 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 18 M 2024-01-21T11:40:20.054 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-01-21T11:40:20.148 INFO:teuthology.orchestra.run.smithi134.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 40 MB/s | 3.7 MB 00:00 2024-01-21T11:40:20.148 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-21T11:40:20.148 INFO:teuthology.orchestra.run.smithi134.stdout:Total 39 MB/s | 3.7 MB 00:00 2024-01-21T11:40:20.149 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-01-21T11:40:20.160 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-01-21T11:40:20.160 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-01-21T11:40:20.222 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-01-21T11:40:20.225 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-01-21T11:40:20.693 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-01-21T11:40:20.732 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:02:50 ago on Sun 21 Jan 2024 11:37:30 AM UTC. 2024-01-21T11:40:20.944 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T11:40:21.367 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T11:40:21.753 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T11:40:21.753 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:40:21.753 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-01-21T11:40:21.753 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2024-01-21T11:40:21.753 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:40:21.753 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-21T11:40:22.003 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install rbd-nbd 2024-01-21T11:40:22.284 INFO:teuthology.orchestra.run.smithi029.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T11:40:22.333 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-01-21T11:40:22.335 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2024-01-21T11:40:22.335 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-01-21T11:40:22.455 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install rbd-fuse 2024-01-21T11:40:22.746 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:45 ago on Sun 21 Jan 2024 11:37:37 AM UTC. 2024-01-21T11:40:23.214 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:02:53 ago on Sun 21 Jan 2024 11:37:30 AM UTC. 2024-01-21T11:40:24.243 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-21T11:40:24.244 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-21T11:40:24.244 INFO:teuthology.orchestra.run.smithi134.stdout: Package Architecture Version Repository Size 2024-01-21T11:40:24.244 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-21T11:40:24.244 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-01-21T11:40:24.244 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2024-01-21T11:40:24.244 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:40:24.244 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-21T11:40:24.244 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-21T11:40:24.245 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-01-21T11:40:24.245 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:40:24.245 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 173 k 2024-01-21T11:40:24.245 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 649 k 2024-01-21T11:40:24.245 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-01-21T11:40:24.316 INFO:teuthology.orchestra.run.smithi134.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 2.4 MB/s | 173 kB 00:00 2024-01-21T11:40:24.317 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-21T11:40:24.317 INFO:teuthology.orchestra.run.smithi134.stdout:Total 2.3 MB/s | 173 kB 00:00 2024-01-21T11:40:24.317 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-01-21T11:40:24.328 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-01-21T11:40:24.328 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-01-21T11:40:24.378 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-01-21T11:40:24.381 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-01-21T11:40:24.501 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-01-21T11:40:24.717 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T11:40:24.833 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-01-21T11:40:24.833 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-21T11:40:24.833 INFO:teuthology.orchestra.run.smithi029.stdout: Package Architecture Version Repository Size 2024-01-21T11:40:24.833 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-21T11:40:24.834 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-01-21T11:40:24.834 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2024-01-21T11:40:24.834 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-21T11:40:24.834 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-01-21T11:40:24.834 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-21T11:40:24.834 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2024-01-21T11:40:24.834 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-21T11:40:24.835 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 89 k 2024-01-21T11:40:24.835 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 298 k 2024-01-21T11:40:24.835 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-01-21T11:40:24.877 INFO:teuthology.orchestra.run.smithi029.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 2.1 MB/s | 89 kB 00:00 2024-01-21T11:40:24.878 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-01-21T11:40:24.878 INFO:teuthology.orchestra.run.smithi029.stdout:Total 2.0 MB/s | 89 kB 00:00 2024-01-21T11:40:24.878 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-01-21T11:40:24.888 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-01-21T11:40:24.888 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-01-21T11:40:24.943 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-01-21T11:40:24.946 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-01-21T11:40:25.007 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T11:40:25.055 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-01-21T11:40:25.268 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T11:40:25.375 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T11:40:25.375 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:40:25.376 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-01-21T11:40:25.376 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2024-01-21T11:40:25.376 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:40:25.376 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-21T11:40:25.560 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T11:40:25.636 DEBUG:teuthology.parallel:result is None 2024-01-21T11:40:25.945 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T11:40:25.945 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-21T11:40:25.945 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-01-21T11:40:25.945 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2024-01-21T11:40:25.945 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-21T11:40:25.945 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-01-21T11:40:26.149 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install rbd-mirror 2024-01-21T11:40:26.913 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:02:56 ago on Sun 21 Jan 2024 11:37:30 AM UTC. 2024-01-21T11:40:28.501 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-01-21T11:40:28.501 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-21T11:40:28.502 INFO:teuthology.orchestra.run.smithi029.stdout: Package Architecture Version Repository Size 2024-01-21T11:40:28.502 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-21T11:40:28.502 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-01-21T11:40:28.502 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-01-21T11:40:28.502 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-21T11:40:28.502 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-01-21T11:40:28.502 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-21T11:40:28.502 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2024-01-21T11:40:28.502 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-21T11:40:28.503 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 3.7 M 2024-01-21T11:40:28.503 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 18 M 2024-01-21T11:40:28.503 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-01-21T11:40:28.608 INFO:teuthology.orchestra.run.smithi029.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 35 MB/s | 3.7 MB 00:00 2024-01-21T11:40:28.608 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-01-21T11:40:28.609 INFO:teuthology.orchestra.run.smithi029.stdout:Total 35 MB/s | 3.7 MB 00:00 2024-01-21T11:40:28.609 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-01-21T11:40:28.622 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-01-21T11:40:28.622 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-01-21T11:40:28.689 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-01-21T11:40:28.692 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-01-21T11:40:29.179 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-01-21T11:40:29.401 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T11:40:29.836 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T11:40:30.224 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T11:40:30.224 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-21T11:40:30.224 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-01-21T11:40:30.224 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2024-01-21T11:40:30.224 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-21T11:40:30.224 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-01-21T11:40:30.431 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install rbd-nbd 2024-01-21T11:40:31.198 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:03:01 ago on Sun 21 Jan 2024 11:37:30 AM UTC. 2024-01-21T11:40:32.787 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-01-21T11:40:32.787 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-21T11:40:32.787 INFO:teuthology.orchestra.run.smithi029.stdout: Package Architecture Version Repository Size 2024-01-21T11:40:32.787 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-21T11:40:32.787 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-01-21T11:40:32.787 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2024-01-21T11:40:32.788 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-21T11:40:32.788 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-01-21T11:40:32.788 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-21T11:40:32.788 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2024-01-21T11:40:32.788 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-21T11:40:32.788 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 173 k 2024-01-21T11:40:32.788 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 649 k 2024-01-21T11:40:32.789 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-01-21T11:40:32.835 INFO:teuthology.orchestra.run.smithi029.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 3.7 MB/s | 173 kB 00:00 2024-01-21T11:40:32.835 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-01-21T11:40:32.835 INFO:teuthology.orchestra.run.smithi029.stdout:Total 3.6 MB/s | 173 kB 00:00 2024-01-21T11:40:32.836 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-01-21T11:40:32.847 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-01-21T11:40:32.847 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-01-21T11:40:32.900 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-01-21T11:40:32.903 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-01-21T11:40:33.021 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-01-21T11:40:33.217 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T11:40:33.519 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T11:40:33.904 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T11:40:33.904 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-21T11:40:33.904 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-01-21T11:40:33.904 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2024-01-21T11:40:33.904 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-21T11:40:33.904 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-01-21T11:40:34.111 DEBUG:teuthology.parallel:result is None 2024-01-21T11:40:34.111 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-01-21T11:40:34.111 INFO:teuthology.packaging:ref: None 2024-01-21T11:40:34.111 INFO:teuthology.packaging:tag: v16.2.4 2024-01-21T11:40:34.111 INFO:teuthology.packaging:branch: None 2024-01-21T11:40:34.111 INFO:teuthology.packaging:sha1: ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec 2024-01-21T11:40:34.112 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-01-21T11:40:34.259 DEBUG:teuthology.orchestra.run.smithi029:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-01-21T11:40:34.282 INFO:teuthology.orchestra.run.smithi029.stdout:16.2.4-0.el8 2024-01-21T11:40:34.283 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2024-01-21T11:40:34.283 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2024-01-21T11:40:34.285 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-01-21T11:40:34.285 INFO:teuthology.packaging:ref: None 2024-01-21T11:40:34.285 INFO:teuthology.packaging:tag: v16.2.4 2024-01-21T11:40:34.285 INFO:teuthology.packaging:branch: None 2024-01-21T11:40:34.285 INFO:teuthology.packaging:sha1: ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec 2024-01-21T11:40:34.285 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-01-21T11:40:34.425 DEBUG:teuthology.orchestra.run.smithi134:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-01-21T11:40:34.452 INFO:teuthology.orchestra.run.smithi134.stdout:16.2.4-0.el8 2024-01-21T11:40:34.452 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2024-01-21T11:40:34.452 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2024-01-21T11:40:34.454 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-01-21T11:40:34.454 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-01-21T11:40:34.454 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-01-21T11:40:34.483 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-01-21T11:40:34.483 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-01-21T11:40:34.526 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-01-21T11:40:34.527 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-01-21T11:40:34.527 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/usr/bin/daemon-helper 2024-01-21T11:40:34.554 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-01-21T11:40:34.622 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-01-21T11:40:34.622 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/usr/bin/daemon-helper 2024-01-21T11:40:34.654 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-01-21T11:40:34.724 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-01-21T11:40:34.724 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-01-21T11:40:34.724 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/usr/bin/adjust-ulimits 2024-01-21T11:40:34.753 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-01-21T11:40:34.820 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-01-21T11:40:34.820 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/usr/bin/adjust-ulimits 2024-01-21T11:40:34.850 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-01-21T11:40:34.920 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-01-21T11:40:34.920 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-01-21T11:40:34.920 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/usr/bin/stdin-killer 2024-01-21T11:40:34.947 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-01-21T11:40:35.015 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-01-21T11:40:35.015 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/usr/bin/stdin-killer 2024-01-21T11:40:35.044 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-01-21T11:40:35.112 INFO:teuthology.run_tasks:Running task print... 2024-01-21T11:40:35.123 INFO:teuthology.task.print:**** done install task... 2024-01-21T11:40:35.123 INFO:teuthology.run_tasks:Running task cephadm... 2024-01-21T11:40:35.266 INFO:tasks.cephadm:Config: {'avoid_pacific_features': True, 'cephadm_branch': 'v16.2.4', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'docker.io/ceph/ceph:v16.2.4', 'roleless': True, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': 'ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec'} 2024-01-21T11:40:35.266 INFO:tasks.cephadm:Cluster image is docker.io/ceph/ceph:v16.2.4 2024-01-21T11:40:35.267 INFO:tasks.cephadm:Cluster fsid is e4084074-b851-11ee-95b0-87774f69a715 2024-01-21T11:40:35.267 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-01-21T11:40:35.267 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-01-21T11:40:35.267 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi029': '172.21.15.29', 'mon.smithi134': '172.21.15.134'} 2024-01-21T11:40:35.267 INFO:tasks.cephadm:Normalizing hostnames... 2024-01-21T11:40:35.267 DEBUG:teuthology.orchestra.run.smithi029:> sudo hostname $(hostname -s) 2024-01-21T11:40:35.296 DEBUG:teuthology.orchestra.run.smithi134:> sudo hostname $(hostname -s) 2024-01-21T11:40:35.325 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v16.2.4)... 2024-01-21T11:40:35.325 DEBUG:teuthology.orchestra.run.smithi029:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-01-21T11:40:35.557 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Jan 21 11:40 /home/ubuntu/cephtest/cephadm 2024-01-21T11:40:35.557 DEBUG:teuthology.orchestra.run.smithi134:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-01-21T11:40:35.780 INFO:teuthology.orchestra.run.smithi134.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Jan 21 11:40 /home/ubuntu/cephtest/cephadm 2024-01-21T11:40:35.781 DEBUG:teuthology.orchestra.run.smithi029:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-01-21T11:40:35.804 DEBUG:teuthology.orchestra.run.smithi134:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-01-21T11:40:35.835 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /etc/ceph 2024-01-21T11:40:35.876 DEBUG:teuthology.orchestra.run.smithi134:> sudo mkdir -p /etc/ceph 2024-01-21T11:40:35.906 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 777 /etc/ceph 2024-01-21T11:40:35.947 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod 777 /etc/ceph 2024-01-21T11:40:35.976 INFO:tasks.cephadm:Writing seed config... 2024-01-21T11:40:35.977 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-01-21T11:40:35.977 INFO:tasks.cephadm: override: [client] debug client = 20 2024-01-21T11:40:35.977 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-01-21T11:40:35.977 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-01-21T11:40:35.977 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-01-21T11:40:35.978 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2024-01-21T11:40:35.978 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-01-21T11:40:35.978 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-01-21T11:40:35.978 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-01-21T11:40:35.978 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-01-21T11:40:35.978 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-01-21T11:40:35.978 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-01-21T11:40:35.978 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-01-21T11:40:35.978 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-01-21T11:40:35.978 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-01-21T11:40:35.978 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-01-21T11:40:35.978 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-01-21T11:40:35.978 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-01-21T11:40:35.978 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-01-21T11:40:35.978 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-01-21T11:40:35.978 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-01-21T11:40:35.979 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-01-21T11:40:35.979 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-01-21T11:40:35.979 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-01-21T11:40:35.979 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-01-21T11:40:35.979 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-01-21T11:40:35.979 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-01-21T11:40:35.979 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-01-21T11:40:35.979 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-01-21T11:40:35.979 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-01-21T11:40:35.979 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-01-21T11:40:35.979 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-01-21T11:40:35.979 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-01-21T11:40:35.979 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-01-21T11:40:35.979 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-01-21T11:40:35.979 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-01-21T11:40:35.979 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-01-21T11:40:35.980 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-01-21T11:40:35.980 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-01-21T11:40:35.980 DEBUG:teuthology.orchestra.run.smithi029:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-01-21T11:40:36.006 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = e4084074-b851-11ee-95b0-87774f69a715 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-01-21T11:40:36.007 DEBUG:teuthology.orchestra.run.smithi029:mon.smithi029> sudo journalctl -f -n 0 -u ceph-e4084074-b851-11ee-95b0-87774f69a715@mon.smithi029.service 2024-01-21T11:40:36.049 INFO:tasks.cephadm:Bootstrapping... 2024-01-21T11:40:36.049 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 -v bootstrap --fsid e4084074-b851-11ee-95b0-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.29 && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-01-21T11:40:36.079 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:-- Logs begin at Sun 2024-01-21 11:25:03 UTC. -- 2024-01-21T11:40:36.356 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2024-01-21T11:40:37.549 INFO:teuthology.orchestra.run.smithi029.stderr:/bin/podman: 4.6.1 2024-01-21T11:40:37.557 INFO:teuthology.orchestra.run.smithi029.stderr:Verifying podman|docker is present... 2024-01-21T11:40:37.557 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2024-01-21T11:40:37.583 INFO:teuthology.orchestra.run.smithi029.stderr:/bin/podman: 4.6.1 2024-01-21T11:40:37.593 INFO:teuthology.orchestra.run.smithi029.stderr:Verifying lvm2 is present... 2024-01-21T11:40:37.593 INFO:teuthology.orchestra.run.smithi029.stderr:Verifying time synchronization is in place... 2024-01-21T11:40:37.593 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl is-enabled chrony.service 2024-01-21T11:40:37.601 INFO:teuthology.orchestra.run.smithi029.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-01-21T11:40:37.602 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl is-active chrony.service 2024-01-21T11:40:37.608 INFO:teuthology.orchestra.run.smithi029.stderr:systemctl: inactive 2024-01-21T11:40:37.609 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl is-enabled chronyd.service 2024-01-21T11:40:37.616 INFO:teuthology.orchestra.run.smithi029.stderr:systemctl: enabled 2024-01-21T11:40:37.616 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl is-active chronyd.service 2024-01-21T11:40:37.622 INFO:teuthology.orchestra.run.smithi029.stderr:systemctl: active 2024-01-21T11:40:37.623 INFO:teuthology.orchestra.run.smithi029.stderr:Unit chronyd.service is enabled and running 2024-01-21T11:40:37.623 INFO:teuthology.orchestra.run.smithi029.stderr:Repeating the final host check... 2024-01-21T11:40:37.623 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2024-01-21T11:40:37.645 INFO:teuthology.orchestra.run.smithi029.stderr:/bin/podman: 4.6.1 2024-01-21T11:40:37.655 INFO:teuthology.orchestra.run.smithi029.stderr:podman|docker (/bin/podman) is present 2024-01-21T11:40:37.655 INFO:teuthology.orchestra.run.smithi029.stderr:systemctl is present 2024-01-21T11:40:37.656 INFO:teuthology.orchestra.run.smithi029.stderr:lvcreate is present 2024-01-21T11:40:37.656 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl is-enabled chrony.service 2024-01-21T11:40:37.662 INFO:teuthology.orchestra.run.smithi029.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-01-21T11:40:37.662 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl is-active chrony.service 2024-01-21T11:40:37.668 INFO:teuthology.orchestra.run.smithi029.stderr:systemctl: inactive 2024-01-21T11:40:37.668 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl is-enabled chronyd.service 2024-01-21T11:40:37.674 INFO:teuthology.orchestra.run.smithi029.stderr:systemctl: enabled 2024-01-21T11:40:37.675 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl is-active chronyd.service 2024-01-21T11:40:37.680 INFO:teuthology.orchestra.run.smithi029.stderr:systemctl: active 2024-01-21T11:40:37.681 INFO:teuthology.orchestra.run.smithi029.stderr:Unit chronyd.service is enabled and running 2024-01-21T11:40:37.681 INFO:teuthology.orchestra.run.smithi029.stderr:Host looks OK 2024-01-21T11:40:37.681 INFO:teuthology.orchestra.run.smithi029.stderr:Cluster fsid: e4084074-b851-11ee-95b0-87774f69a715 2024-01-21T11:40:37.682 INFO:teuthology.orchestra.run.smithi029.stderr:Acquiring lock 139660192618984 on /run/cephadm/e4084074-b851-11ee-95b0-87774f69a715.lock 2024-01-21T11:40:37.682 INFO:teuthology.orchestra.run.smithi029.stderr:Lock 139660192618984 acquired on /run/cephadm/e4084074-b851-11ee-95b0-87774f69a715.lock 2024-01-21T11:40:37.682 INFO:teuthology.orchestra.run.smithi029.stderr:Verifying IP 172.21.15.29 port 3300 ... 2024-01-21T11:40:37.682 INFO:teuthology.orchestra.run.smithi029.stderr:Verifying IP 172.21.15.29 port 6789 ... 2024-01-21T11:40:37.683 INFO:teuthology.orchestra.run.smithi029.stderr:Base mon IP is 172.21.15.29, final addrv is [v2:172.21.15.29:3300,v1:172.21.15.29:6789] 2024-01-21T11:40:37.683 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /sbin/ip route ls 2024-01-21T11:40:37.685 INFO:teuthology.orchestra.run.smithi029.stderr:/sbin/ip: default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.29 metric 100 2024-01-21T11:40:37.685 INFO:teuthology.orchestra.run.smithi029.stderr:/sbin/ip: 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.29 metric 100 2024-01-21T11:40:37.686 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /sbin/ip -6 route ls 2024-01-21T11:40:37.688 INFO:teuthology.orchestra.run.smithi029.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-01-21T11:40:37.688 INFO:teuthology.orchestra.run.smithi029.stderr:/sbin/ip: fe80::/64 dev ens1f0 proto kernel metric 256 pref medium 2024-01-21T11:40:37.689 INFO:teuthology.orchestra.run.smithi029.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 1024 expires 1794sec hoplimit 64 pref medium 2024-01-21T11:40:37.689 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /sbin/ip -6 addr ls 2024-01-21T11:40:37.692 INFO:teuthology.orchestra.run.smithi029.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-01-21T11:40:37.692 INFO:teuthology.orchestra.run.smithi029.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-01-21T11:40:37.692 INFO:teuthology.orchestra.run.smithi029.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-01-21T11:40:37.692 INFO:teuthology.orchestra.run.smithi029.stderr:/sbin/ip: 4: ens1f0: mtu 1500 state UP qlen 1000 2024-01-21T11:40:37.692 INFO:teuthology.orchestra.run.smithi029.stderr:/sbin/ip: inet6 fe80::ec4:7aff:febd:125a/64 scope link 2024-01-21T11:40:37.692 INFO:teuthology.orchestra.run.smithi029.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-01-21T11:40:37.693 INFO:teuthology.orchestra.run.smithi029.stderr:Mon IP 172.21.15.29 is in CIDR network 172.21.0.0/20 2024-01-21T11:40:37.693 INFO:teuthology.orchestra.run.smithi029.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-01-21T11:40:37.694 INFO:teuthology.orchestra.run.smithi029.stderr:Pulling container image docker.io/ceph/ceph:v16.2.4... 2024-01-21T11:40:37.695 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman pull docker.io/ceph/ceph:v16.2.4 2024-01-21T11:40:37.794 INFO:teuthology.orchestra.run.smithi029.stderr:/bin/podman: Trying to pull docker.io/ceph/ceph:v16.2.4... 2024-01-21T11:40:44.356 INFO:teuthology.orchestra.run.smithi029.stderr:/bin/podman: Getting image source signatures 2024-01-21T11:40:44.365 INFO:teuthology.orchestra.run.smithi029.stderr:/bin/podman: Copying blob sha256:551101690791055467b1840be133b2e6b029d263771bc6c46a96bb7acdacac39 2024-01-21T11:40:44.365 INFO:teuthology.orchestra.run.smithi029.stderr:/bin/podman: Copying blob sha256:7a0437f04f83f084b7ed68ad9c4a4947e12fc4e1b006b38129bac89114ec3621 2024-01-21T11:40:56.310 INFO:teuthology.orchestra.run.smithi029.stderr:/bin/podman: Copying config sha256:8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2024-01-21T11:40:56.368 INFO:teuthology.orchestra.run.smithi029.stderr:/bin/podman: Writing manifest to image destination 2024-01-21T11:40:56.698 INFO:teuthology.orchestra.run.smithi029.stderr:/bin/podman: 8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2024-01-21T11:40:56.707 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --version 2024-01-21T11:40:57.328 INFO:teuthology.orchestra.run.smithi029.stderr:ceph: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2024-01-21T11:40:57.684 INFO:teuthology.orchestra.run.smithi029.stderr:Ceph version: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2024-01-21T11:40:57.684 INFO:teuthology.orchestra.run.smithi029.stderr:Extracting ceph user uid/gid from container image... 2024-01-21T11:40:57.684 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 -c %u %g /var/lib/ceph 2024-01-21T11:40:57.985 INFO:teuthology.orchestra.run.smithi029.stderr:stat: 167 167 2024-01-21T11:40:58.297 INFO:teuthology.orchestra.run.smithi029.stderr:Creating initial keys... 2024-01-21T11:40:58.297 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2024-01-21T11:40:58.554 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-authtool: AQDKAq1ldBWYIBAAoMBDNZMEjgPgl4U6w3iC6g== 2024-01-21T11:40:58.887 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2024-01-21T11:40:59.154 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-authtool: AQDLAq1llydiCBAAGH6Kapywr53f12wQGUkzwg== 2024-01-21T11:40:59.490 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2024-01-21T11:40:59.745 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-authtool: AQDLAq1lh6n0KxAA3jd5HQf1Y0t6Js/Y3FBM3Q== 2024-01-21T11:41:00.091 INFO:teuthology.orchestra.run.smithi029.stderr:Creating initial monmap... 2024-01-21T11:41:00.092 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/monmaptool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /tmp/ceph-tmpobwgvlvu:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --create --clobber --fsid e4084074-b851-11ee-95b0-87774f69a715 --addv smithi029 [v2:172.21.15.29:3300,v1:172.21.15.29:6789] /tmp/monmap 2024-01-21T11:41:00.362 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-01-21T11:41:00.362 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to e4084074-b851-11ee-95b0-87774f69a715 2024-01-21T11:41:00.363 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-01-21T11:41:00.712 INFO:teuthology.orchestra.run.smithi029.stderr:monmaptool for smithi029 [v2:172.21.15.29:3300,v1:172.21.15.29:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-01-21T11:41:00.712 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/monmaptool: set fsid to e4084074-b851-11ee-95b0-87774f69a715 2024-01-21T11:41:00.712 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-01-21T11:41:00.712 INFO:teuthology.orchestra.run.smithi029.stderr: 2024-01-21T11:41:00.712 INFO:teuthology.orchestra.run.smithi029.stderr:Creating mon... 2024-01-21T11:41:00.712 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-mon --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e4084074-b851-11ee-95b0-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/e4084074-b851-11ee-95b0-87774f69a715/mon.smithi029:/var/lib/ceph/mon/ceph-smithi029:z -v /tmp/ceph-tmp_3896ms8:/tmp/keyring:z -v /tmp/ceph-tmpobwgvlvu:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --mkfs -i smithi029 --fsid e4084074-b851-11ee-95b0-87774f69a715 -c /dev/null --monmap /tmp/monmap --keyring /tmp/keyring --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true 2024-01-21T11:41:01.008 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.973+0000 7f05aaa28700 0 set uid:gid to 167:167 (ceph:ceph) 2024-01-21T11:41:01.008 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.974+0000 7f05aaa28700 1 imported monmap: 2024-01-21T11:41:01.008 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: epoch 0 2024-01-21T11:41:01.008 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: fsid e4084074-b851-11ee-95b0-87774f69a715 2024-01-21T11:41:01.008 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: last_changed 2024-01-21T11:41:00.354236+0000 2024-01-21T11:41:01.008 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: created 2024-01-21T11:41:00.354236+0000 2024-01-21T11:41:01.008 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: min_mon_release 0 (unknown) 2024-01-21T11:41:01.008 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: election_strategy: 1 2024-01-21T11:41:01.008 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: 0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.smithi029 2024-01-21T11:41:01.008 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: 2024-01-21T11:41:01.009 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.974+0000 7f05aaa28700 0 /usr/bin/ceph-mon: set fsid to e4084074-b851-11ee-95b0-87774f69a715 2024-01-21T11:41:01.009 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: RocksDB version: 6.8.1 2024-01-21T11:41:01.009 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: 2024-01-21T11:41:01.009 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-01-21T11:41:01.009 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Compile date May 13 2021 2024-01-21T11:41:01.009 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: DB SUMMARY 2024-01-21T11:41:01.009 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: 2024-01-21T11:41:01.009 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi029/store.db dir, Total Num: 0, files: 2024-01-21T11:41:01.009 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: 2024-01-21T11:41:01.009 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi029/store.db: 2024-01-21T11:41:01.009 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: 2024-01-21T11:41:01.009 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.error_if_exists: 0 2024-01-21T11:41:01.010 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.create_if_missing: 1 2024-01-21T11:41:01.010 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.paranoid_checks: 1 2024-01-21T11:41:01.010 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.env: 0x556ae35dd1c0 2024-01-21T11:41:01.010 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.fs: Posix File System 2024-01-21T11:41:01.010 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.info_log: 0x556ae58e6ec0 2024-01-21T11:41:01.010 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.max_file_opening_threads: 16 2024-01-21T11:41:01.010 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.statistics: (nil) 2024-01-21T11:41:01.010 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.use_fsync: 0 2024-01-21T11:41:01.010 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.max_log_file_size: 0 2024-01-21T11:41:01.010 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-01-21T11:41:01.010 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.log_file_time_to_roll: 0 2024-01-21T11:41:01.010 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.keep_log_file_num: 1000 2024-01-21T11:41:01.010 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.recycle_log_file_num: 0 2024-01-21T11:41:01.011 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.allow_fallocate: 1 2024-01-21T11:41:01.011 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.allow_mmap_reads: 0 2024-01-21T11:41:01.011 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.allow_mmap_writes: 0 2024-01-21T11:41:01.011 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.use_direct_reads: 0 2024-01-21T11:41:01.011 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-01-21T11:41:01.011 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.create_missing_column_families: 0 2024-01-21T11:41:01.011 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.db_log_dir: 2024-01-21T11:41:01.011 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi029/store.db 2024-01-21T11:41:01.011 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.table_cache_numshardbits: 6 2024-01-21T11:41:01.011 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.max_subcompactions: 1 2024-01-21T11:41:01.011 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.max_background_flushes: -1 2024-01-21T11:41:01.011 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-01-21T11:41:01.012 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-01-21T11:41:01.012 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-01-21T11:41:01.012 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-01-21T11:41:01.012 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-01-21T11:41:01.012 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.advise_random_on_open: 1 2024-01-21T11:41:01.012 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.db_write_buffer_size: 0 2024-01-21T11:41:01.012 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.write_buffer_manager: 0x556ae58ed530 2024-01-21T11:41:01.012 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-01-21T11:41:01.012 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-01-21T11:41:01.012 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-01-21T11:41:01.012 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.use_adaptive_mutex: 0 2024-01-21T11:41:01.012 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.rate_limiter: (nil) 2024-01-21T11:41:01.013 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-01-21T11:41:01.013 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.wal_recovery_mode: 2 2024-01-21T11:41:01.013 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.enable_thread_tracking: 0 2024-01-21T11:41:01.013 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.enable_pipelined_write: 0 2024-01-21T11:41:01.013 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.unordered_write: 0 2024-01-21T11:41:01.013 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-01-21T11:41:01.013 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-01-21T11:41:01.013 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-01-21T11:41:01.013 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-01-21T11:41:01.013 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.row_cache: None 2024-01-21T11:41:01.013 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.wal_filter: None 2024-01-21T11:41:01.013 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-01-21T11:41:01.014 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.allow_ingest_behind: 0 2024-01-21T11:41:01.014 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.preserve_deletes: 0 2024-01-21T11:41:01.014 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.two_write_queues: 0 2024-01-21T11:41:01.014 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.manual_wal_flush: 0 2024-01-21T11:41:01.014 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.atomic_flush: 0 2024-01-21T11:41:01.014 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-01-21T11:41:01.014 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.persist_stats_to_disk: 0 2024-01-21T11:41:01.014 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-01-21T11:41:01.014 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.log_readahead_size: 0 2024-01-21T11:41:01.014 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-01-21T11:41:01.014 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.max_background_jobs: 2 2024-01-21T11:41:01.014 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.max_background_compactions: -1 2024-01-21T11:41:01.015 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-01-21T11:41:01.015 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-01-21T11:41:01.015 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.delayed_write_rate : 16777216 2024-01-21T11:41:01.015 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.max_total_wal_size: 0 2024-01-21T11:41:01.015 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-01-21T11:41:01.015 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.stats_dump_period_sec: 600 2024-01-21T11:41:01.015 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.stats_persist_period_sec: 600 2024-01-21T11:41:01.015 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-01-21T11:41:01.015 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.max_open_files: -1 2024-01-21T11:41:01.015 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.bytes_per_sync: 0 2024-01-21T11:41:01.015 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-01-21T11:41:01.015 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-01-21T11:41:01.016 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.compaction_readahead_size: 0 2024-01-21T11:41:01.016 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Compression algorithms supported: 2024-01-21T11:41:01.016 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-01-21T11:41:01.016 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: kZSTD supported: 0 2024-01-21T11:41:01.016 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: kXpressCompression supported: 0 2024-01-21T11:41:01.016 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: kLZ4HCCompression supported: 1 2024-01-21T11:41:01.016 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: kLZ4Compression supported: 1 2024-01-21T11:41:01.016 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: kBZip2Compression supported: 0 2024-01-21T11:41:01.016 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: kZlibCompression supported: 1 2024-01-21T11:41:01.016 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: kSnappyCompression supported: 1 2024-01-21T11:41:01.016 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-01-21T11:41:01.017 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: [db_impl/db_impl_open.cc:273] Creating manifest 1 2024-01-21T11:41:01.017 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: 2024-01-21T11:41:01.017 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi029/store.db/MANIFEST-000001 2024-01-21T11:41:01.017 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: 2024-01-21T11:41:01.017 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-01-21T11:41:01.017 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: 2024-01-21T11:41:01.017 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-01-21T11:41:01.017 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.merge_operator: 2024-01-21T11:41:01.017 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.compaction_filter: None 2024-01-21T11:41:01.017 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.compaction_filter_factory: None 2024-01-21T11:41:01.017 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-01-21T11:41:01.017 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.table_factory: BlockBasedTable 2024-01-21T11:41:01.017 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x556ae57fe120) 2024-01-21T11:41:01.018 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks: 1 2024-01-21T11:41:01.018 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks_with_high_priority: 0 2024-01-21T11:41:01.018 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: pin_l0_filter_and_index_blocks_in_cache: 0 2024-01-21T11:41:01.018 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: pin_top_level_index_and_filter: 1 2024-01-21T11:41:01.018 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: index_type: 0 2024-01-21T11:41:01.018 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: data_block_index_type: 0 2024-01-21T11:41:01.018 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: index_shortening: 1 2024-01-21T11:41:01.018 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: data_block_hash_table_util_ratio: 0.750000 2024-01-21T11:41:01.018 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: hash_index_allow_collision: 1 2024-01-21T11:41:01.018 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: checksum: 1 2024-01-21T11:41:01.018 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: no_block_cache: 0 2024-01-21T11:41:01.018 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: block_cache: 0x556ae5834d10 2024-01-21T11:41:01.018 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: block_cache_name: BinnedLRUCache 2024-01-21T11:41:01.019 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: block_cache_options: 2024-01-21T11:41:01.019 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: capacity : 536870912 2024-01-21T11:41:01.019 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: num_shard_bits : 4 2024-01-21T11:41:01.019 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: strict_capacity_limit : 0 2024-01-21T11:41:01.019 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: high_pri_pool_ratio: 0.000 2024-01-21T11:41:01.019 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: block_cache_compressed: (nil) 2024-01-21T11:41:01.019 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: persistent_cache: (nil) 2024-01-21T11:41:01.019 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: block_size: 4096 2024-01-21T11:41:01.019 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: block_size_deviation: 10 2024-01-21T11:41:01.019 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: block_restart_interval: 16 2024-01-21T11:41:01.019 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: index_block_restart_interval: 1 2024-01-21T11:41:01.019 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: metadata_block_size: 4096 2024-01-21T11:41:01.019 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: partition_filters: 0 2024-01-21T11:41:01.020 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: use_delta_encoding: 1 2024-01-21T11:41:01.020 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: filter_policy: rocksdb.BuiltinBloomFilter 2024-01-21T11:41:01.020 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: whole_key_filtering: 1 2024-01-21T11:41:01.020 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: verify_compression: 0 2024-01-21T11:41:01.020 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: read_amp_bytes_per_bit: 0 2024-01-21T11:41:01.020 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: format_version: 2 2024-01-21T11:41:01.020 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: enable_index_compression: 1 2024-01-21T11:41:01.020 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: block_align: 0 2024-01-21T11:41:01.020 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: 2024-01-21T11:41:01.020 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.write_buffer_size: 33554432 2024-01-21T11:41:01.020 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.max_write_buffer_number: 2 2024-01-21T11:41:01.020 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.compression: NoCompression 2024-01-21T11:41:01.021 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.bottommost_compression: Disabled 2024-01-21T11:41:01.021 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.prefix_extractor: nullptr 2024-01-21T11:41:01.021 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-01-21T11:41:01.021 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.num_levels: 7 2024-01-21T11:41:01.021 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-01-21T11:41:01.021 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-01-21T11:41:01.021 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-01-21T11:41:01.021 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-01-21T11:41:01.021 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-01-21T11:41:01.021 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-01-21T11:41:01.021 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-01-21T11:41:01.021 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-01-21T11:41:01.021 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-01-21T11:41:01.022 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.compression_opts.window_bits: -14 2024-01-21T11:41:01.022 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.compression_opts.level: 32767 2024-01-21T11:41:01.022 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.compression_opts.strategy: 0 2024-01-21T11:41:01.022 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-01-21T11:41:01.022 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-01-21T11:41:01.022 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.compression_opts.enabled: false 2024-01-21T11:41:01.022 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-01-21T11:41:01.022 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-01-21T11:41:01.022 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-01-21T11:41:01.022 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.target_file_size_base: 67108864 2024-01-21T11:41:01.022 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.target_file_size_multiplier: 1 2024-01-21T11:41:01.022 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-01-21T11:41:01.022 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-01-21T11:41:01.022 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-01-21T11:41:01.023 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-01-21T11:41:01.023 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-01-21T11:41:01.023 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-01-21T11:41:01.023 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-01-21T11:41:01.023 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-01-21T11:41:01.023 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-01-21T11:41:01.023 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-01-21T11:41:01.023 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-01-21T11:41:01.023 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-01-21T11:41:01.023 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.arena_block_size: 4194304 2024-01-21T11:41:01.023 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-01-21T11:41:01.023 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-01-21T11:41:01.023 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-01-21T11:41:01.024 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.disable_auto_compactions: 0 2024-01-21T11:41:01.024 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-01-21T11:41:01.024 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-01-21T11:41:01.024 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-01-21T11:41:01.024 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-01-21T11:41:01.024 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-01-21T11:41:01.024 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-01-21T11:41:01.024 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-01-21T11:41:01.024 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-01-21T11:41:01.024 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-01-21T11:41:01.024 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-01-21T11:41:01.025 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.table_properties_collectors: 2024-01-21T11:41:01.025 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.inplace_update_support: 0 2024-01-21T11:41:01.025 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-01-21T11:41:01.025 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-01-21T11:41:01.025 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-01-21T11:41:01.025 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.memtable_huge_page_size: 0 2024-01-21T11:41:01.025 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.bloom_locality: 0 2024-01-21T11:41:01.025 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.max_successive_merges: 0 2024-01-21T11:41:01.025 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-01-21T11:41:01.025 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.paranoid_file_checks: 0 2024-01-21T11:41:01.025 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.force_consistency_checks: 0 2024-01-21T11:41:01.025 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.report_bg_io_stats: 0 2024-01-21T11:41:01.025 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.ttl: 2592000 2024-01-21T11:41:01.026 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.975+0000 7f05aaa28700 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-01-21T11:41:01.026 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.976+0000 7f05aaa28700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi029/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-01-21T11:41:01.026 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: 2024-01-21T11:41:01.026 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.976+0000 7f05aaa28700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2024-01-21T11:41:01.026 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: 2024-01-21T11:41:01.026 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.977+0000 7f05aaa28700 4 rocksdb: DB pointer 0x556ae58f9800 2024-01-21T11:41:01.026 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.977+0000 7f05938c1700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-01-21T11:41:01.026 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.977+0000 7f05938c1700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-01-21T11:41:01.026 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: ** DB Stats ** 2024-01-21T11:41:01.026 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-01-21T11:41:01.026 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-01-21T11:41:01.026 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-01-21T11:41:01.026 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-21T11:41:01.027 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-01-21T11:41:01.027 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-01-21T11:41:01.027 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-21T11:41:01.027 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: 2024-01-21T11:41:01.027 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-01-21T11:41:01.027 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-21T11:41:01.027 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-21T11:41:01.027 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-21T11:41:01.027 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-21T11:41:01.027 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: 2024-01-21T11:41:01.027 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-01-21T11:41:01.027 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-21T11:41:01.027 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-21T11:41:01.028 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-01-21T11:41:01.028 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2024-01-21T11:41:01.028 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-21T11:41:01.028 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2024-01-21T11:41:01.028 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2024-01-21T11:41:01.028 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2024-01-21T11:41:01.028 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-21T11:41:01.028 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-21T11:41:01.028 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-21T11:41:01.028 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: 2024-01-21T11:41:01.028 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2024-01-21T11:41:01.028 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: 2024-01-21T11:41:01.028 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-01-21T11:41:01.029 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-21T11:41:01.029 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-21T11:41:01.029 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-21T11:41:01.029 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-21T11:41:01.029 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: 2024-01-21T11:41:01.029 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-01-21T11:41:01.029 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-21T11:41:01.029 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-21T11:41:01.029 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-01-21T11:41:01.029 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2024-01-21T11:41:01.029 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-21T11:41:01.029 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2024-01-21T11:41:01.029 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2024-01-21T11:41:01.030 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2024-01-21T11:41:01.030 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-21T11:41:01.030 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-21T11:41:01.030 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-21T11:41:01.030 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: 2024-01-21T11:41:01.030 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2024-01-21T11:41:01.030 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: 2024-01-21T11:41:01.030 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.978+0000 7f05aaa28700 4 rocksdb: [db_impl/db_impl.cc:397] Shutdown: canceling all background work 2024-01-21T11:41:01.030 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.978+0000 7f05aaa28700 4 rocksdb: [db_impl/db_impl.cc:573] Shutdown complete 2024-01-21T11:41:01.030 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-01-21T11:41:00.978+0000 7f05aaa28700 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-smithi029 for mon.smithi029 2024-01-21T11:41:01.291 INFO:teuthology.orchestra.run.smithi029.stderr:create mon.smithi029 on 2024-01-21T11:41:01.291 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: install -d -m0770 -o 167 -g 167 /var/run/ceph/e4084074-b851-11ee-95b0-87774f69a715 2024-01-21T11:41:01.296 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl disable ceph.target 2024-01-21T11:41:01.302 INFO:teuthology.orchestra.run.smithi029.stderr:systemctl: Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2024-01-21T11:41:01.392 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl enable ceph.target 2024-01-21T11:41:01.400 INFO:teuthology.orchestra.run.smithi029.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-01-21T11:41:01.484 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl start ceph.target 2024-01-21T11:41:01.491 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl enable ceph-e4084074-b851-11ee-95b0-87774f69a715.target 2024-01-21T11:41:01.497 INFO:teuthology.orchestra.run.smithi029.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-e4084074-b851-11ee-95b0-87774f69a715.target → /etc/systemd/system/ceph-e4084074-b851-11ee-95b0-87774f69a715.target. 2024-01-21T11:41:01.497 INFO:teuthology.orchestra.run.smithi029.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-e4084074-b851-11ee-95b0-87774f69a715.target → /etc/systemd/system/ceph-e4084074-b851-11ee-95b0-87774f69a715.target. 2024-01-21T11:41:01.581 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl start ceph-e4084074-b851-11ee-95b0-87774f69a715.target 2024-01-21T11:41:01.588 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl daemon-reload 2024-01-21T11:41:01.680 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl stop ceph-e4084074-b851-11ee-95b0-87774f69a715@mon.smithi029 2024-01-21T11:41:01.717 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl reset-failed ceph-e4084074-b851-11ee-95b0-87774f69a715@mon.smithi029 2024-01-21T11:41:01.722 INFO:teuthology.orchestra.run.smithi029.stderr:systemctl: Failed to reset failed state of unit ceph-e4084074-b851-11ee-95b0-87774f69a715@mon.smithi029.service: Unit ceph-e4084074-b851-11ee-95b0-87774f69a715@mon.smithi029.service not loaded. 2024-01-21T11:41:01.723 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl enable ceph-e4084074-b851-11ee-95b0-87774f69a715@mon.smithi029 2024-01-21T11:41:01.730 INFO:teuthology.orchestra.run.smithi029.stderr:systemctl: Created symlink /etc/systemd/system/ceph-e4084074-b851-11ee-95b0-87774f69a715.target.wants/ceph-e4084074-b851-11ee-95b0-87774f69a715@mon.smithi029.service → /etc/systemd/system/ceph-e4084074-b851-11ee-95b0-87774f69a715@.service. 2024-01-21T11:41:01.824 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl start ceph-e4084074-b851-11ee-95b0-87774f69a715@mon.smithi029 2024-01-21T11:41:01.864 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:01 smithi029 systemd[1]: Starting Ceph mon.smithi029 for e4084074-b851-11ee-95b0-87774f69a715... 2024-01-21T11:41:02.132 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl is-enabled firewalld.service 2024-01-21T11:41:02.141 INFO:teuthology.orchestra.run.smithi029.stderr:systemctl: disabled 2024-01-21T11:41:02.141 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl is-active firewalld.service 2024-01-21T11:41:02.149 INFO:teuthology.orchestra.run.smithi029.stderr:systemctl: inactive 2024-01-21T11:41:02.149 INFO:teuthology.orchestra.run.smithi029.stderr:firewalld.service is not enabled 2024-01-21T11:41:02.150 INFO:teuthology.orchestra.run.smithi029.stderr:Not possible to enable service . firewalld.service is not available 2024-01-21T11:41:02.150 INFO:teuthology.orchestra.run.smithi029.stderr:Not possible to open ports <[]>. firewalld.service is not available 2024-01-21T11:41:02.150 INFO:teuthology.orchestra.run.smithi029.stderr:Waiting for mon to start... 2024-01-21T11:41:02.150 INFO:teuthology.orchestra.run.smithi029.stderr:Waiting for mon... 2024-01-21T11:41:02.150 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/lib/ceph/e4084074-b851-11ee-95b0-87774f69a715/mon.smithi029:/var/lib/ceph/mon/ceph-smithi029:z -v /tmp/ceph-tmp77cpgrhz:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpea05xzin:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status 2024-01-21T11:41:02.419 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:02 smithi029 bash[38390]: 61bd58dcdf8fec1b142e075bdad9b005f26d77ef60c329574ccd5d89fb934f3d 2024-01-21T11:41:02.419 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:02 smithi029 systemd[1]: Started Ceph mon.smithi029 for e4084074-b851-11ee-95b0-87774f69a715. 2024-01-21T11:41:02.419 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:02 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38405]: cluster 2024-01-21T11:41:02.157046+0000 mon.smithi029 (mon.0) 0 : [INF] mkfs e4084074-b851-11ee-95b0-87774f69a715 2024-01-21T11:41:02.419 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:02 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38405]: cluster 2024-01-21T11:41:02.141486+0000 mon.smithi029 (mon.0) 1 : cluster [INF] mon.smithi029 is new leader, mons smithi029 in quorum (ranks 0) 2024-01-21T11:41:02.545 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:02.544+0000 7ff158ee2700 1 Processor -- start 2024-01-21T11:41:02.547 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:02.544+0000 7ff158ee2700 1 -- start start 2024-01-21T11:41:02.547 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:02.544+0000 7ff158ee2700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff154072490 0x7ff154070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T11:41:02.547 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:02.544+0000 7ff158ee2700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1540728b0 con 0x7ff154072490 2024-01-21T11:41:02.547 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:02.545+0000 7ff15259c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff154072490 0x7ff154070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:41:02.547 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:02.545+0000 7ff15259c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff154072490 0x7ff154070a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49350/0 (socket says 172.21.15.29:49350) 2024-01-21T11:41:02.547 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:02.545+0000 7ff15259c700 1 -- 172.21.15.29:0/1329642874 learned_addr learned my addr 172.21.15.29:0/1329642874 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-21T11:41:02.547 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:02.545+0000 7ff15259c700 1 -- 172.21.15.29:0/1329642874 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff154070f20 con 0x7ff154072490 2024-01-21T11:41:02.547 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:02.545+0000 7ff15259c700 1 --2- 172.21.15.29:0/1329642874 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff154072490 0x7ff154070a30 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7ff14400b5f0 tx=0x7ff144006ee0).ready entity=mon.0 client_cookie=5ea05ce169e3a558 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:41:02.547 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:02.546+0000 7ff15159a700 1 -- 172.21.15.29:0/1329642874 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff14400d600 con 0x7ff154072490 2024-01-21T11:41:02.547 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:02.546+0000 7ff15159a700 1 -- 172.21.15.29:0/1329642874 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff14400d760 con 0x7ff154072490 2024-01-21T11:41:02.549 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:02.546+0000 7ff15159a700 1 -- 172.21.15.29:0/1329642874 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff14400da30 con 0x7ff154072490 2024-01-21T11:41:02.549 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:02.546+0000 7ff158ee2700 1 -- 172.21.15.29:0/1329642874 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff154072490 msgr2=0x7ff154070a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:41:02.549 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:02.546+0000 7ff158ee2700 1 --2- 172.21.15.29:0/1329642874 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff154072490 0x7ff154070a30 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7ff14400b5f0 tx=0x7ff144006ee0).stop 2024-01-21T11:41:02.549 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:02.546+0000 7ff158ee2700 1 -- 172.21.15.29:0/1329642874 shutdown_connections 2024-01-21T11:41:02.549 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:02.546+0000 7ff158ee2700 1 --2- 172.21.15.29:0/1329642874 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff154072490 0x7ff154070a30 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:41:02.549 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:02.546+0000 7ff158ee2700 1 -- 172.21.15.29:0/1329642874 >> 172.21.15.29:0/1329642874 conn(0x7ff1540f74c0 msgr2=0x7ff1540f98e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:41:02.549 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:02.546+0000 7ff158ee2700 1 -- 172.21.15.29:0/1329642874 shutdown_connections 2024-01-21T11:41:02.549 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:02.546+0000 7ff158ee2700 1 -- 172.21.15.29:0/1329642874 wait complete. 2024-01-21T11:41:02.549 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:02.547+0000 7ff158ee2700 1 Processor -- start 2024-01-21T11:41:02.549 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:02.547+0000 7ff158ee2700 1 -- start start 2024-01-21T11:41:02.549 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:02.547+0000 7ff158ee2700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff154072490 0x7ff1540fbc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:41:02.549 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:02.547+0000 7ff158ee2700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1540fc140 con 0x7ff154072490 2024-01-21T11:41:02.549 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:02.547+0000 7ff15259c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff154072490 0x7ff1540fbc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:41:02.550 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:02.547+0000 7ff15259c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff154072490 0x7ff1540fbc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49358/0 (socket says 172.21.15.29:49358) 2024-01-21T11:41:02.550 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:02.547+0000 7ff15259c700 1 -- 172.21.15.29:0/2032129423 learned_addr learned my addr 172.21.15.29:0/2032129423 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-21T11:41:02.550 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:02.548+0000 7ff15259c700 1 -- 172.21.15.29:0/2032129423 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff14400b200 con 0x7ff154072490 2024-01-21T11:41:02.550 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:02.548+0000 7ff15259c700 1 --2- 172.21.15.29:0/2032129423 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff154072490 0x7ff1540fbc50 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7ff144006c70 tx=0x7ff144016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:41:02.550 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:02.548+0000 7ff142ffd700 1 -- 172.21.15.29:0/2032129423 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff144011b40 con 0x7ff154072490 2024-01-21T11:41:02.550 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:02.548+0000 7ff142ffd700 1 -- 172.21.15.29:0/2032129423 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff144011ca0 con 0x7ff154072490 2024-01-21T11:41:02.550 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:02.548+0000 7ff142ffd700 1 -- 172.21.15.29:0/2032129423 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff14401d430 con 0x7ff154072490 2024-01-21T11:41:02.550 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:02.548+0000 7ff158ee2700 1 -- 172.21.15.29:0/2032129423 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1540fc340 con 0x7ff154072490 2024-01-21T11:41:02.551 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:02.548+0000 7ff158ee2700 1 -- 172.21.15.29:0/2032129423 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1540fc750 con 0x7ff154072490 2024-01-21T11:41:02.551 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:02.548+0000 7ff142ffd700 1 -- 172.21.15.29:0/2032129423 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7ff14400f070 con 0x7ff154072490 2024-01-21T11:41:02.551 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:02.548+0000 7ff142ffd700 1 -- 172.21.15.29:0/2032129423 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7ff1440277d0 con 0x7ff154072490 2024-01-21T11:41:02.551 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:02.549+0000 7ff158ee2700 1 -- 172.21.15.29:0/2032129423 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff154045ad0 con 0x7ff154072490 2024-01-21T11:41:02.552 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:02.551+0000 7ff142ffd700 1 -- 172.21.15.29:0/2032129423 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7ff144016070 con 0x7ff154072490 2024-01-21T11:41:02.623 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:02.620+0000 7ff158ee2700 1 -- 172.21.15.29:0/2032129423 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7ff154105f70 con 0x7ff154072490 2024-01-21T11:41:02.624 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:02.621+0000 7ff142ffd700 1 -- 172.21.15.29:0/2032129423 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7ff144036070 con 0x7ff154072490 2024-01-21T11:41:02.624 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: cluster: 2024-01-21T11:41:02.624 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: id: e4084074-b851-11ee-95b0-87774f69a715 2024-01-21T11:41:02.624 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-01-21T11:41:02.624 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:02.624 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: services: 2024-01-21T11:41:02.624 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: mon: 1 daemons, quorum smithi029 (age 0.466501s) 2024-01-21T11:41:02.624 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: mgr: no daemons active 2024-01-21T11:41:02.624 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-01-21T11:41:02.624 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:02.624 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: data: 2024-01-21T11:41:02.624 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-01-21T11:41:02.624 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-01-21T11:41:02.625 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-01-21T11:41:02.625 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: pgs: 2024-01-21T11:41:02.625 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:02.625 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:02.621+0000 7ff158ee2700 1 -- 172.21.15.29:0/2032129423 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff154072490 msgr2=0x7ff1540fbc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:41:02.625 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:02.621+0000 7ff158ee2700 1 --2- 172.21.15.29:0/2032129423 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff154072490 0x7ff1540fbc50 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7ff144006c70 tx=0x7ff144016000).stop 2024-01-21T11:41:02.625 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:02.621+0000 7ff158ee2700 1 -- 172.21.15.29:0/2032129423 shutdown_connections 2024-01-21T11:41:02.625 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:02.621+0000 7ff158ee2700 1 --2- 172.21.15.29:0/2032129423 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff154072490 0x7ff1540fbc50 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:41:02.625 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:02.621+0000 7ff158ee2700 1 -- 172.21.15.29:0/2032129423 >> 172.21.15.29:0/2032129423 conn(0x7ff1540f74c0 msgr2=0x7ff154100c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:41:02.625 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:02.621+0000 7ff158ee2700 1 -- 172.21.15.29:0/2032129423 shutdown_connections 2024-01-21T11:41:02.625 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:02.621+0000 7ff158ee2700 1 -- 172.21.15.29:0/2032129423 wait complete. 2024-01-21T11:41:02.973 INFO:teuthology.orchestra.run.smithi029.stderr:mon is available 2024-01-21T11:41:02.973 INFO:teuthology.orchestra.run.smithi029.stderr:Assimilating anything we can from ceph.conf... 2024-01-21T11:41:02.973 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e4084074-b851-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp77cpgrhz:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpea05xzin:/etc/ceph/ceph.conf:z -v /var/lib/ceph/e4084074-b851-11ee-95b0-87774f69a715/mon.smithi029:/var/lib/ceph/mon/ceph-smithi029:z docker.io/ceph/ceph:v16.2.4 config assimilate-conf -i /var/lib/ceph/mon/ceph-smithi029/config 2024-01-21T11:41:03.409 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:03.404+0000 7fd2cdd3f700 1 Processor -- start 2024-01-21T11:41:03.409 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:03.404+0000 7fd2cdd3f700 1 -- start start 2024-01-21T11:41:03.409 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:03.404+0000 7fd2cdd3f700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2c80703d0 0x7fd2c80fe940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T11:41:03.409 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:03.404+0000 7fd2cdd3f700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2c80fee30 con 0x7fd2c80703d0 2024-01-21T11:41:03.409 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:03.405+0000 7fd2c77fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2c80703d0 0x7fd2c80fe940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:41:03.409 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:03.405+0000 7fd2c77fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2c80703d0 0x7fd2c80fe940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49368/0 (socket says 172.21.15.29:49368) 2024-01-21T11:41:03.409 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:03.405+0000 7fd2c77fe700 1 -- 172.21.15.29:0/3361061885 learned_addr learned my addr 172.21.15.29:0/3361061885 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-21T11:41:03.409 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:03.405+0000 7fd2c77fe700 1 -- 172.21.15.29:0/3361061885 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2c80fef70 con 0x7fd2c80703d0 2024-01-21T11:41:03.410 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:03.405+0000 7fd2c77fe700 1 --2- 172.21.15.29:0/3361061885 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2c80703d0 0x7fd2c80fe940 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7fd2b800b5f0 tx=0x7fd2b8006ee0).ready entity=mon.0 client_cookie=25a0e426b5425086 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:41:03.410 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:03.405+0000 7fd2c67fc700 1 -- 172.21.15.29:0/3361061885 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd2b800d600 con 0x7fd2c80703d0 2024-01-21T11:41:03.410 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:03.405+0000 7fd2c67fc700 1 -- 172.21.15.29:0/3361061885 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2b800d760 con 0x7fd2c80703d0 2024-01-21T11:41:03.410 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:03.405+0000 7fd2c67fc700 1 -- 172.21.15.29:0/3361061885 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd2b800da30 con 0x7fd2c80703d0 2024-01-21T11:41:03.410 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:03.406+0000 7fd2cdd3f700 1 -- 172.21.15.29:0/3361061885 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2c80703d0 msgr2=0x7fd2c80fe940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:41:03.410 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:03.406+0000 7fd2cdd3f700 1 --2- 172.21.15.29:0/3361061885 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2c80703d0 0x7fd2c80fe940 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7fd2b800b5f0 tx=0x7fd2b8006ee0).stop 2024-01-21T11:41:03.410 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:03.406+0000 7fd2cdd3f700 1 -- 172.21.15.29:0/3361061885 shutdown_connections 2024-01-21T11:41:03.410 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:03.406+0000 7fd2cdd3f700 1 --2- 172.21.15.29:0/3361061885 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2c80703d0 0x7fd2c80fe940 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:41:03.410 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:03.406+0000 7fd2cdd3f700 1 -- 172.21.15.29:0/3361061885 >> 172.21.15.29:0/3361061885 conn(0x7fd2c80f63c0 msgr2=0x7fd2c80f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:41:03.410 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:03.406+0000 7fd2cdd3f700 1 -- 172.21.15.29:0/3361061885 shutdown_connections 2024-01-21T11:41:03.410 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:03.406+0000 7fd2cdd3f700 1 -- 172.21.15.29:0/3361061885 wait complete. 2024-01-21T11:41:03.410 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:03.406+0000 7fd2cdd3f700 1 Processor -- start 2024-01-21T11:41:03.410 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:03.406+0000 7fd2cdd3f700 1 -- start start 2024-01-21T11:41:03.410 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:03.407+0000 7fd2cdd3f700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2c80703d0 0x7fd2c810a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:41:03.410 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:03.407+0000 7fd2cdd3f700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2c810a7b0 con 0x7fd2c80703d0 2024-01-21T11:41:03.411 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:03.407+0000 7fd2c77fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2c80703d0 0x7fd2c810a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:41:03.411 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:03.407+0000 7fd2c77fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2c80703d0 0x7fd2c810a2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49384/0 (socket says 172.21.15.29:49384) 2024-01-21T11:41:03.411 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:03.407+0000 7fd2c77fe700 1 -- 172.21.15.29:0/523198880 learned_addr learned my addr 172.21.15.29:0/523198880 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-21T11:41:03.411 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:03.407+0000 7fd2c77fe700 1 -- 172.21.15.29:0/523198880 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2b800b200 con 0x7fd2c80703d0 2024-01-21T11:41:03.411 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:03.407+0000 7fd2c77fe700 1 --2- 172.21.15.29:0/523198880 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2c80703d0 0x7fd2c810a2c0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7fd2b8006c70 tx=0x7fd2b8016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:41:03.411 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:03.407+0000 7fd2bffff700 1 -- 172.21.15.29:0/523198880 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd2b8011b40 con 0x7fd2c80703d0 2024-01-21T11:41:03.411 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:03.407+0000 7fd2bffff700 1 -- 172.21.15.29:0/523198880 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2b8011ca0 con 0x7fd2c80703d0 2024-01-21T11:41:03.411 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:03.407+0000 7fd2cdd3f700 1 -- 172.21.15.29:0/523198880 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2c810a9b0 con 0x7fd2c80703d0 2024-01-21T11:41:03.411 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:03.407+0000 7fd2bffff700 1 -- 172.21.15.29:0/523198880 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd2b801d430 con 0x7fd2c80703d0 2024-01-21T11:41:03.411 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:03.408+0000 7fd2cdd3f700 1 -- 172.21.15.29:0/523198880 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2c810adc0 con 0x7fd2c80703d0 2024-01-21T11:41:03.411 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:03.408+0000 7fd2bffff700 1 -- 172.21.15.29:0/523198880 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7fd2b800f070 con 0x7fd2c80703d0 2024-01-21T11:41:03.411 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:03.408+0000 7fd2bffff700 1 -- 172.21.15.29:0/523198880 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fd2b8027460 con 0x7fd2c80703d0 2024-01-21T11:41:03.412 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:03.409+0000 7fd2cdd3f700 1 -- 172.21.15.29:0/523198880 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2c8045ad0 con 0x7fd2c80703d0 2024-01-21T11:41:03.412 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:03.411+0000 7fd2bffff700 1 -- 172.21.15.29:0/523198880 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fd2b8016070 con 0x7fd2c80703d0 2024-01-21T11:41:03.418 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:03 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38405]: cluster 2024-01-21T11:41:02.156323+0000 mon.smithi029 (mon.0) 2 : cluster [INF] mon.smithi029 is new leader, mons smithi029 in quorum (ranks 0) 2024-01-21T11:41:03.418 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:03 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38405]: cluster 2024-01-21T11:41:02.156534+0000 mon.smithi029 (mon.0) 3 : cluster [DBG] monmap e1: 1 mons at {smithi029=[v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0]} 2024-01-21T11:41:03.418 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:03 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38405]: cluster 2024-01-21T11:41:02.158675+0000 mon.smithi029 (mon.0) 4 : cluster [DBG] fsmap 2024-01-21T11:41:03.418 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:03 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38405]: cluster 2024-01-21T11:41:02.161148+0000 mon.smithi029 (mon.0) 5 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-01-21T11:41:03.418 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:03 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38405]: cluster 2024-01-21T11:41:02.161639+0000 mon.smithi029 (mon.0) 6 : cluster [DBG] mgrmap e1: no daemons active 2024-01-21T11:41:03.418 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:03 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38405]: audit 2024-01-21T11:41:02.622785+0000 mon.smithi029 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.29:0/2032129423' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-01-21T11:41:03.480 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:03.478+0000 7fd2cdd3f700 1 -- 172.21.15.29:0/523198880 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fd2c80ff160 con 0x7fd2c80703d0 2024-01-21T11:41:03.485 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:03.482+0000 7fd2bffff700 1 -- 172.21.15.29:0/523198880 <== mon.0 v2:172.21.15.29:3300/0 7 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fd2b801d590 con 0x7fd2c80703d0 2024-01-21T11:41:03.485 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:03.482+0000 7fd2bffff700 1 -- 172.21.15.29:0/523198880 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+464 (secure 0 0 0) 0x7fd2b8036020 con 0x7fd2c80703d0 2024-01-21T11:41:03.485 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:03.486 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: [global] 2024-01-21T11:41:03.486 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: fsid = e4084074-b851-11ee-95b0-87774f69a715 2024-01-21T11:41:03.486 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.29:3300,v1:172.21.15.29:6789] 2024-01-21T11:41:03.486 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-01-21T11:41:03.486 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-01-21T11:41:03.486 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-01-21T11:41:03.486 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-01-21T11:41:03.486 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:03.486 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: [mgr] 2024-01-21T11:41:03.486 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-01-21T11:41:03.486 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:03.486 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: [osd] 2024-01-21T11:41:03.486 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: mon_osd_backfillfull_ratio = 0.85 2024-01-21T11:41:03.486 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: mon_osd_full_ratio = 0.9 2024-01-21T11:41:03.486 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: mon_osd_nearfull_ratio = 0.8 2024-01-21T11:41:03.487 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-01-21T11:41:03.487 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: osd_memory_target_autotune = true 2024-01-21T11:41:03.487 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-01-21T11:41:03.487 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:03.483+0000 7fd2cdd3f700 1 -- 172.21.15.29:0/523198880 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2c80703d0 msgr2=0x7fd2c810a2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:41:03.487 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:03.483+0000 7fd2cdd3f700 1 --2- 172.21.15.29:0/523198880 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2c80703d0 0x7fd2c810a2c0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7fd2b8006c70 tx=0x7fd2b8016000).stop 2024-01-21T11:41:03.487 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:03.483+0000 7fd2cdd3f700 1 -- 172.21.15.29:0/523198880 shutdown_connections 2024-01-21T11:41:03.487 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:03.483+0000 7fd2cdd3f700 1 --2- 172.21.15.29:0/523198880 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2c80703d0 0x7fd2c810a2c0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:41:03.487 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:03.483+0000 7fd2cdd3f700 1 -- 172.21.15.29:0/523198880 >> 172.21.15.29:0/523198880 conn(0x7fd2c80f63c0 msgr2=0x7fd2c80f8150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:41:03.487 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:03.483+0000 7fd2cdd3f700 1 -- 172.21.15.29:0/523198880 shutdown_connections 2024-01-21T11:41:03.487 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:03.483+0000 7fd2cdd3f700 1 -- 172.21.15.29:0/523198880 wait complete. 2024-01-21T11:41:03.831 INFO:teuthology.orchestra.run.smithi029.stderr:Generating new minimal ceph.conf... 2024-01-21T11:41:03.831 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e4084074-b851-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp77cpgrhz:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpea05xzin:/etc/ceph/ceph.conf:z -v /var/lib/ceph/e4084074-b851-11ee-95b0-87774f69a715/mon.smithi029:/var/lib/ceph/mon/ceph-smithi029:z docker.io/ceph/ceph:v16.2.4 config generate-minimal-conf -o /var/lib/ceph/mon/ceph-smithi029/config 2024-01-21T11:41:04.206 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:04.204+0000 7f3288f14700 1 Processor -- start 2024-01-21T11:41:04.206 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:04.204+0000 7f3288f14700 1 -- start start 2024-01-21T11:41:04.206 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:04.204+0000 7f3288f14700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32840703d0 0x7f32840fe940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T11:41:04.206 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:04.204+0000 7f3288f14700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32840fee30 con 0x7f32840703d0 2024-01-21T11:41:04.207 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:04.204+0000 7f328259c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32840703d0 0x7f32840fe940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:41:04.207 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:04.204+0000 7f328259c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32840703d0 0x7f32840fe940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49392/0 (socket says 172.21.15.29:49392) 2024-01-21T11:41:04.207 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:04.204+0000 7f328259c700 1 -- 172.21.15.29:0/3079293743 learned_addr learned my addr 172.21.15.29:0/3079293743 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-21T11:41:04.207 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:04.204+0000 7f328259c700 1 -- 172.21.15.29:0/3079293743 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32840fef70 con 0x7f32840703d0 2024-01-21T11:41:04.207 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:04.205+0000 7f328259c700 1 --2- 172.21.15.29:0/3079293743 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32840703d0 0x7f32840fe940 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f327400b5f0 tx=0x7f3274006ee0).ready entity=mon.0 client_cookie=295262bd9a5830f6 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:41:04.207 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:04.205+0000 7f328159a700 1 -- 172.21.15.29:0/3079293743 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f327400d600 con 0x7f32840703d0 2024-01-21T11:41:04.207 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:04.205+0000 7f328159a700 1 -- 172.21.15.29:0/3079293743 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f327400dbd0 con 0x7f32840703d0 2024-01-21T11:41:04.207 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:04.205+0000 7f328159a700 1 -- 172.21.15.29:0/3079293743 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3274011c00 con 0x7f32840703d0 2024-01-21T11:41:04.209 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:04.205+0000 7f3288f14700 1 -- 172.21.15.29:0/3079293743 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32840703d0 msgr2=0x7f32840fe940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:41:04.209 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:04.206+0000 7f3288f14700 1 --2- 172.21.15.29:0/3079293743 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32840703d0 0x7f32840fe940 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f327400b5f0 tx=0x7f3274006ee0).stop 2024-01-21T11:41:04.209 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:04.206+0000 7f3288f14700 1 -- 172.21.15.29:0/3079293743 shutdown_connections 2024-01-21T11:41:04.209 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:04.206+0000 7f3288f14700 1 --2- 172.21.15.29:0/3079293743 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32840703d0 0x7f32840fe940 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:41:04.210 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:04.206+0000 7f3288f14700 1 -- 172.21.15.29:0/3079293743 >> 172.21.15.29:0/3079293743 conn(0x7f32840f63c0 msgr2=0x7f32840f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:41:04.210 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:04.206+0000 7f3288f14700 1 -- 172.21.15.29:0/3079293743 shutdown_connections 2024-01-21T11:41:04.210 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:04.206+0000 7f3288f14700 1 -- 172.21.15.29:0/3079293743 wait complete. 2024-01-21T11:41:04.210 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:04.206+0000 7f3288f14700 1 Processor -- start 2024-01-21T11:41:04.210 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:04.206+0000 7f3288f14700 1 -- start start 2024-01-21T11:41:04.210 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:04.207+0000 7f3288f14700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32840703d0 0x7f328410aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:41:04.210 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:04.207+0000 7f3288f14700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f328410af20 con 0x7f32840703d0 2024-01-21T11:41:04.210 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:04.207+0000 7f328259c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32840703d0 0x7f328410aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:41:04.210 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:04.207+0000 7f328259c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32840703d0 0x7f328410aa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49402/0 (socket says 172.21.15.29:49402) 2024-01-21T11:41:04.210 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:04.207+0000 7f328259c700 1 -- 172.21.15.29:0/3271535833 learned_addr learned my addr 172.21.15.29:0/3271535833 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-21T11:41:04.210 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:04.207+0000 7f328259c700 1 -- 172.21.15.29:0/3271535833 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f327400b200 con 0x7f32840703d0 2024-01-21T11:41:04.210 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:04.207+0000 7f328259c700 1 --2- 172.21.15.29:0/3271535833 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32840703d0 0x7f328410aa30 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f327400e040 tx=0x7f3274006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:41:04.210 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:04.207+0000 7f327affd700 1 -- 172.21.15.29:0/3271535833 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f327400d9f0 con 0x7f32840703d0 2024-01-21T11:41:04.210 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:04.207+0000 7f327affd700 1 -- 172.21.15.29:0/3271535833 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f327401a9f0 con 0x7f32840703d0 2024-01-21T11:41:04.211 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:04.207+0000 7f3288f14700 1 -- 172.21.15.29:0/3271535833 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f328410b120 con 0x7f32840703d0 2024-01-21T11:41:04.211 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:04.207+0000 7f327affd700 1 -- 172.21.15.29:0/3271535833 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f327401b8c0 con 0x7f32840703d0 2024-01-21T11:41:04.211 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:04.207+0000 7f3288f14700 1 -- 172.21.15.29:0/3271535833 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f328410b530 con 0x7f32840703d0 2024-01-21T11:41:04.211 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:04.208+0000 7f327affd700 1 -- 172.21.15.29:0/3271535833 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f327400f070 con 0x7f32840703d0 2024-01-21T11:41:04.211 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:04.208+0000 7f327affd700 1 -- 172.21.15.29:0/3271535833 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f3274025b30 con 0x7f32840703d0 2024-01-21T11:41:04.211 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:04.209+0000 7f3288f14700 1 -- 172.21.15.29:0/3271535833 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3284045ad0 con 0x7f32840703d0 2024-01-21T11:41:04.212 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:04.211+0000 7f327affd700 1 -- 172.21.15.29:0/3271535833 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f3274016070 con 0x7f32840703d0 2024-01-21T11:41:04.279 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:04.278+0000 7f3288f14700 1 -- 172.21.15.29:0/3271535833 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f3284104890 con 0x7f32840703d0 2024-01-21T11:41:04.279 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:04.278+0000 7f327affd700 1 -- 172.21.15.29:0/3271535833 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7f327401bd70 con 0x7f32840703d0 2024-01-21T11:41:04.281 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:04.279+0000 7f3288f14700 1 -- 172.21.15.29:0/3271535833 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32840703d0 msgr2=0x7f328410aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:41:04.281 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:04.279+0000 7f3288f14700 1 --2- 172.21.15.29:0/3271535833 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32840703d0 0x7f328410aa30 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f327400e040 tx=0x7f3274006ee0).stop 2024-01-21T11:41:04.281 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:04.279+0000 7f3288f14700 1 -- 172.21.15.29:0/3271535833 shutdown_connections 2024-01-21T11:41:04.281 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:04.279+0000 7f3288f14700 1 --2- 172.21.15.29:0/3271535833 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32840703d0 0x7f328410aa30 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:41:04.281 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:04.279+0000 7f3288f14700 1 -- 172.21.15.29:0/3271535833 >> 172.21.15.29:0/3271535833 conn(0x7f32840f63c0 msgr2=0x7f32840f7f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:41:04.281 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:04.279+0000 7f3288f14700 1 -- 172.21.15.29:0/3271535833 shutdown_connections 2024-01-21T11:41:04.281 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:04.279+0000 7f3288f14700 1 -- 172.21.15.29:0/3271535833 wait complete. 2024-01-21T11:41:04.418 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:04 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38405]: audit 2024-01-21T11:41:03.480950+0000 mon.smithi029 (mon.0) 8 : audit [INF] from='client.? 172.21.15.29:0/523198880' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-01-21T11:41:04.419 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:04 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38405]: audit 2024-01-21T11:41:03.484034+0000 mon.smithi029 (mon.0) 9 : audit [INF] from='client.? 172.21.15.29:0/523198880' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-01-21T11:41:04.640 INFO:teuthology.orchestra.run.smithi029.stderr:Restarting the monitor... 2024-01-21T11:41:04.640 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl restart ceph-e4084074-b851-11ee-95b0-87774f69a715@mon.smithi029 2024-01-21T11:41:04.720 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:04 smithi029 systemd[1]: Stopping Ceph mon.smithi029 for e4084074-b851-11ee-95b0-87774f69a715... 2024-01-21T11:41:05.038 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:04 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38405]: debug 2024-01-21T11:41:04.719+0000 7fa511649700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi029 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-01-21T11:41:05.038 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:04 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38405]: debug 2024-01-21T11:41:04.719+0000 7fa511649700 -1 mon.smithi029@0(leader) e1 *** Got Signal Terminated *** 2024-01-21T11:41:05.399 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 podman[38727]: ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029 2024-01-21T11:41:05.399 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 systemd[1]: ceph-e4084074-b851-11ee-95b0-87774f69a715@mon.smithi029.service: Succeeded. 2024-01-21T11:41:05.399 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 systemd[1]: Stopped Ceph mon.smithi029 for e4084074-b851-11ee-95b0-87774f69a715. 2024-01-21T11:41:05.399 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 systemd[1]: Starting Ceph mon.smithi029 for e4084074-b851-11ee-95b0-87774f69a715... 2024-01-21T11:41:05.482 INFO:teuthology.orchestra.run.smithi029.stderr:Setting mon public_network to 172.21.0.0/20 2024-01-21T11:41:05.483 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e4084074-b851-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp77cpgrhz:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpea05xzin:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mon public_network 172.21.0.0/20 2024-01-21T11:41:05.826 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:05.822+0000 7f6223e85700 1 Processor -- start 2024-01-21T11:41:05.826 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:05.822+0000 7f6223e85700 1 -- start start 2024-01-21T11:41:05.826 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:05.822+0000 7f6223e85700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f621c070430 0x7f621c0fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T11:41:05.826 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:05.822+0000 7f6223e85700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f621c0fedd0 con 0x7f621c070430 2024-01-21T11:41:05.826 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:05.822+0000 7f6221c21700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f621c070430 0x7f621c0fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:41:05.826 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:05.823+0000 7f6221c21700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f621c070430 0x7f621c0fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49404/0 (socket says 172.21.15.29:49404) 2024-01-21T11:41:05.827 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:05.823+0000 7f6221c21700 1 -- 172.21.15.29:0/1046789736 learned_addr learned my addr 172.21.15.29:0/1046789736 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-21T11:41:05.827 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:05.823+0000 7f6221c21700 1 -- 172.21.15.29:0/1046789736 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f621c0fef10 con 0x7f621c070430 2024-01-21T11:41:05.827 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:05.823+0000 7f6221c21700 1 --2- 172.21.15.29:0/1046789736 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f621c070430 0x7f621c0fe8e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f621800b5f0 tx=0x7f6218006ee0).ready entity=mon.0 client_cookie=58c4f0176fcacef2 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:41:05.827 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:05.823+0000 7f6220c1f700 1 -- 172.21.15.29:0/1046789736 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f621800d600 con 0x7f621c070430 2024-01-21T11:41:05.827 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:05.823+0000 7f6220c1f700 1 -- 172.21.15.29:0/1046789736 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f621800dbd0 con 0x7f621c070430 2024-01-21T11:41:05.827 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:05.823+0000 7f6220c1f700 1 -- 172.21.15.29:0/1046789736 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6218011c00 con 0x7f621c070430 2024-01-21T11:41:05.827 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:05.824+0000 7f6223e85700 1 -- 172.21.15.29:0/1046789736 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f621c070430 msgr2=0x7f621c0fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:41:05.827 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:05.824+0000 7f6223e85700 1 --2- 172.21.15.29:0/1046789736 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f621c070430 0x7f621c0fe8e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f621800b5f0 tx=0x7f6218006ee0).stop 2024-01-21T11:41:05.827 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:05.824+0000 7f6223e85700 1 -- 172.21.15.29:0/1046789736 shutdown_connections 2024-01-21T11:41:05.827 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:05.824+0000 7f6223e85700 1 --2- 172.21.15.29:0/1046789736 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f621c070430 0x7f621c0fe8e0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:41:05.827 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:05.824+0000 7f6223e85700 1 -- 172.21.15.29:0/1046789736 >> 172.21.15.29:0/1046789736 conn(0x7f621c0f63c0 msgr2=0x7f621c0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:41:05.827 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:05.824+0000 7f6223e85700 1 -- 172.21.15.29:0/1046789736 shutdown_connections 2024-01-21T11:41:05.827 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:05.824+0000 7f6223e85700 1 -- 172.21.15.29:0/1046789736 wait complete. 2024-01-21T11:41:05.827 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:05.824+0000 7f6223e85700 1 Processor -- start 2024-01-21T11:41:05.828 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:05.824+0000 7f6223e85700 1 -- start start 2024-01-21T11:41:05.828 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:05.825+0000 7f6223e85700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f621c070430 0x7f621c10ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:41:05.828 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:05.825+0000 7f6223e85700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f621c10d1e0 con 0x7f621c070430 2024-01-21T11:41:05.828 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:05.825+0000 7f6221c21700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f621c070430 0x7f621c10ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:41:05.828 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:05.825+0000 7f6221c21700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f621c070430 0x7f621c10ccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49420/0 (socket says 172.21.15.29:49420) 2024-01-21T11:41:05.828 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:05.825+0000 7f6221c21700 1 -- 172.21.15.29:0/873282698 learned_addr learned my addr 172.21.15.29:0/873282698 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-21T11:41:05.829 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:05.825+0000 7f6221c21700 1 -- 172.21.15.29:0/873282698 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f621800b200 con 0x7f621c070430 2024-01-21T11:41:05.829 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:05.825+0000 7f6221c21700 1 --2- 172.21.15.29:0/873282698 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f621c070430 0x7f621c10ccf0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f621800e040 tx=0x7f6218006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:41:05.829 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:05.825+0000 7f620a7fc700 1 -- 172.21.15.29:0/873282698 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f621800d9f0 con 0x7f621c070430 2024-01-21T11:41:05.830 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:05.825+0000 7f6223e85700 1 -- 172.21.15.29:0/873282698 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f621c10d3e0 con 0x7f621c070430 2024-01-21T11:41:05.830 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:05.825+0000 7f620a7fc700 1 -- 172.21.15.29:0/873282698 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f621801a9f0 con 0x7f621c070430 2024-01-21T11:41:05.830 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:05.825+0000 7f620a7fc700 1 -- 172.21.15.29:0/873282698 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f621801b8c0 con 0x7f621c070430 2024-01-21T11:41:05.830 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:05.825+0000 7f6223e85700 1 -- 172.21.15.29:0/873282698 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f621c10d7f0 con 0x7f621c070430 2024-01-21T11:41:05.830 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:05.825+0000 7f620a7fc700 1 -- 172.21.15.29:0/873282698 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f621800f070 con 0x7f621c070430 2024-01-21T11:41:05.830 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:05.826+0000 7f620a7fc700 1 -- 172.21.15.29:0/873282698 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f6218025b30 con 0x7f621c070430 2024-01-21T11:41:05.830 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:05.826+0000 7f6223e85700 1 -- 172.21.15.29:0/873282698 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f621c045ad0 con 0x7f621c070430 2024-01-21T11:41:05.830 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:05.828+0000 7f620a7fc700 1 -- 172.21.15.29:0/873282698 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f6218016070 con 0x7f621c070430 2024-01-21T11:41:05.896 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:05.895+0000 7f6223e85700 1 -- 172.21.15.29:0/873282698 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f621c10ed20 con 0x7f621c070430 2024-01-21T11:41:05.898 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:05.897+0000 7f620a7fc700 1 -- 172.21.15.29:0/873282698 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f621801bdc0 con 0x7f621c070430 2024-01-21T11:41:05.899 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:05.898+0000 7f6223e85700 1 -- 172.21.15.29:0/873282698 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f621c070430 msgr2=0x7f621c10ccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:41:05.899 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:05.898+0000 7f6223e85700 1 --2- 172.21.15.29:0/873282698 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f621c070430 0x7f621c10ccf0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f621800e040 tx=0x7f6218006ee0).stop 2024-01-21T11:41:05.899 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:05.898+0000 7f6223e85700 1 -- 172.21.15.29:0/873282698 shutdown_connections 2024-01-21T11:41:05.900 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:05.898+0000 7f6223e85700 1 --2- 172.21.15.29:0/873282698 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f621c070430 0x7f621c10ccf0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:41:05.900 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:05.898+0000 7f6223e85700 1 -- 172.21.15.29:0/873282698 >> 172.21.15.29:0/873282698 conn(0x7f621c0f63c0 msgr2=0x7f621c0f7eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:41:05.900 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:05.898+0000 7f6223e85700 1 -- 172.21.15.29:0/873282698 shutdown_connections 2024-01-21T11:41:05.900 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:05.898+0000 7f6223e85700 1 -- 172.21.15.29:0/873282698 wait complete. 2024-01-21T11:41:05.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.471+0000 7fbbfa265700 0 set uid:gid to 167:167 (ceph:ceph) 2024-01-21T11:41:05.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.471+0000 7fbbfa265700 0 ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable), process ceph-mon, pid 7 2024-01-21T11:41:05.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.471+0000 7fbbfa265700 0 pidfile_write: ignore empty --pid-file 2024-01-21T11:41:05.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.476+0000 7fbbfa265700 0 load: jerasure load: lrc load: isa 2024-01-21T11:41:05.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: RocksDB version: 6.8.1 2024-01-21T11:41:05.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: 2024-01-21T11:41:05.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-01-21T11:41:05.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 bash[38800]: 74524d767ca7b550d13d7cef7ddbced65badd305f62a23f7e1e2107152c00eeb 2024-01-21T11:41:05.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Compile date May 13 2021 2024-01-21T11:41:05.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: DB SUMMARY 2024-01-21T11:41:05.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: 2024-01-21T11:41:05.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: CURRENT file: CURRENT 2024-01-21T11:41:05.920 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: 2024-01-21T11:41:05.920 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: IDENTITY file: IDENTITY 2024-01-21T11:41:05.920 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: 2024-01-21T11:41:05.920 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 131 Bytes 2024-01-21T11:41:05.920 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: 2024-01-21T11:41:05.920 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi029/store.db dir, Total Num: 1, files: 000004.sst 2024-01-21T11:41:05.920 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: 2024-01-21T11:41:05.920 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi029/store.db: 000006.log size: 92914 ; 2024-01-21T11:41:05.920 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: 2024-01-21T11:41:05.920 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.error_if_exists: 0 2024-01-21T11:41:05.920 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.create_if_missing: 0 2024-01-21T11:41:05.920 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.paranoid_checks: 1 2024-01-21T11:41:05.920 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.env: 0x5629b33831c0 2024-01-21T11:41:05.921 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.fs: Posix File System 2024-01-21T11:41:05.921 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.info_log: 0x5629b4ea3960 2024-01-21T11:41:05.921 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.max_file_opening_threads: 16 2024-01-21T11:41:05.921 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.statistics: (nil) 2024-01-21T11:41:05.921 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.use_fsync: 0 2024-01-21T11:41:05.921 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.max_log_file_size: 0 2024-01-21T11:41:05.921 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-01-21T11:41:05.921 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.log_file_time_to_roll: 0 2024-01-21T11:41:05.921 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.keep_log_file_num: 1000 2024-01-21T11:41:05.921 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.recycle_log_file_num: 0 2024-01-21T11:41:05.921 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.allow_fallocate: 1 2024-01-21T11:41:05.921 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.allow_mmap_reads: 0 2024-01-21T11:41:05.921 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.allow_mmap_writes: 0 2024-01-21T11:41:05.921 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.use_direct_reads: 0 2024-01-21T11:41:05.921 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-01-21T11:41:05.922 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.create_missing_column_families: 0 2024-01-21T11:41:05.922 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.db_log_dir: 2024-01-21T11:41:05.922 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi029/store.db 2024-01-21T11:41:05.922 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.table_cache_numshardbits: 6 2024-01-21T11:41:05.922 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.max_subcompactions: 1 2024-01-21T11:41:05.922 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.max_background_flushes: -1 2024-01-21T11:41:05.922 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-01-21T11:41:05.922 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-01-21T11:41:05.922 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-01-21T11:41:05.922 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-01-21T11:41:05.922 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-01-21T11:41:05.922 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.advise_random_on_open: 1 2024-01-21T11:41:05.922 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.db_write_buffer_size: 0 2024-01-21T11:41:05.922 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.write_buffer_manager: 0x5629b5b18030 2024-01-21T11:41:05.923 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-01-21T11:41:05.923 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-01-21T11:41:05.923 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-01-21T11:41:05.923 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.use_adaptive_mutex: 0 2024-01-21T11:41:05.923 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.rate_limiter: (nil) 2024-01-21T11:41:05.923 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-01-21T11:41:05.923 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.wal_recovery_mode: 2 2024-01-21T11:41:05.923 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.enable_thread_tracking: 0 2024-01-21T11:41:05.923 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.enable_pipelined_write: 0 2024-01-21T11:41:05.923 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.unordered_write: 0 2024-01-21T11:41:05.923 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-01-21T11:41:05.925 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-01-21T11:41:05.925 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-01-21T11:41:05.925 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-01-21T11:41:05.925 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.row_cache: None 2024-01-21T11:41:05.925 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.wal_filter: None 2024-01-21T11:41:05.925 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-01-21T11:41:05.926 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.allow_ingest_behind: 0 2024-01-21T11:41:05.926 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.preserve_deletes: 0 2024-01-21T11:41:05.926 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.two_write_queues: 0 2024-01-21T11:41:05.926 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.manual_wal_flush: 0 2024-01-21T11:41:05.926 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.atomic_flush: 0 2024-01-21T11:41:05.926 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-01-21T11:41:05.926 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.persist_stats_to_disk: 0 2024-01-21T11:41:05.926 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-01-21T11:41:05.926 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.log_readahead_size: 0 2024-01-21T11:41:05.926 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-01-21T11:41:05.926 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.max_background_jobs: 2 2024-01-21T11:41:05.926 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.max_background_compactions: -1 2024-01-21T11:41:05.926 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-01-21T11:41:05.926 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-01-21T11:41:05.927 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.delayed_write_rate : 16777216 2024-01-21T11:41:05.927 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.max_total_wal_size: 0 2024-01-21T11:41:05.927 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-01-21T11:41:05.927 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.stats_dump_period_sec: 600 2024-01-21T11:41:05.927 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.stats_persist_period_sec: 600 2024-01-21T11:41:05.927 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-01-21T11:41:05.927 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.max_open_files: -1 2024-01-21T11:41:05.927 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.bytes_per_sync: 0 2024-01-21T11:41:05.927 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-01-21T11:41:05.927 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-01-21T11:41:05.927 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.compaction_readahead_size: 0 2024-01-21T11:41:05.927 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Compression algorithms supported: 2024-01-21T11:41:05.927 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-01-21T11:41:05.927 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: kZSTD supported: 0 2024-01-21T11:41:05.927 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: kXpressCompression supported: 0 2024-01-21T11:41:05.928 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: kLZ4HCCompression supported: 1 2024-01-21T11:41:05.928 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: kLZ4Compression supported: 1 2024-01-21T11:41:05.928 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: kBZip2Compression supported: 0 2024-01-21T11:41:05.928 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: kZlibCompression supported: 1 2024-01-21T11:41:05.928 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: kSnappyCompression supported: 1 2024-01-21T11:41:05.928 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-01-21T11:41:05.928 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi029/store.db/MANIFEST-000005 2024-01-21T11:41:05.928 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: 2024-01-21T11:41:05.928 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-01-21T11:41:05.928 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: 2024-01-21T11:41:05.928 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-01-21T11:41:05.928 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.merge_operator: 2024-01-21T11:41:05.928 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.compaction_filter: None 2024-01-21T11:41:05.928 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.compaction_filter_factory: None 2024-01-21T11:41:05.928 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-01-21T11:41:05.929 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.table_factory: BlockBasedTable 2024-01-21T11:41:05.929 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5629b4dba128) 2024-01-21T11:41:05.929 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cache_index_and_filter_blocks: 1 2024-01-21T11:41:05.929 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cache_index_and_filter_blocks_with_high_priority: 0 2024-01-21T11:41:05.929 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: pin_l0_filter_and_index_blocks_in_cache: 0 2024-01-21T11:41:05.929 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: pin_top_level_index_and_filter: 1 2024-01-21T11:41:05.929 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: index_type: 0 2024-01-21T11:41:05.929 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: data_block_index_type: 0 2024-01-21T11:41:05.929 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: index_shortening: 1 2024-01-21T11:41:05.929 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: data_block_hash_table_util_ratio: 0.750000 2024-01-21T11:41:05.929 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: hash_index_allow_collision: 1 2024-01-21T11:41:05.929 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: checksum: 1 2024-01-21T11:41:05.929 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: no_block_cache: 0 2024-01-21T11:41:05.929 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: block_cache: 0x5629b4df0f10 2024-01-21T11:41:05.930 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: block_cache_name: BinnedLRUCache 2024-01-21T11:41:05.930 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: block_cache_options: 2024-01-21T11:41:05.930 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: capacity : 536870912 2024-01-21T11:41:05.930 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: num_shard_bits : 4 2024-01-21T11:41:05.930 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: strict_capacity_limit : 0 2024-01-21T11:41:05.930 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: high_pri_pool_ratio: 0.000 2024-01-21T11:41:05.930 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: block_cache_compressed: (nil) 2024-01-21T11:41:05.930 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: persistent_cache: (nil) 2024-01-21T11:41:05.930 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: block_size: 4096 2024-01-21T11:41:05.930 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: block_size_deviation: 10 2024-01-21T11:41:05.930 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: block_restart_interval: 16 2024-01-21T11:41:05.930 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: index_block_restart_interval: 1 2024-01-21T11:41:05.930 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: metadata_block_size: 4096 2024-01-21T11:41:05.930 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: partition_filters: 0 2024-01-21T11:41:05.931 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: use_delta_encoding: 1 2024-01-21T11:41:05.931 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: filter_policy: rocksdb.BuiltinBloomFilter 2024-01-21T11:41:05.931 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: whole_key_filtering: 1 2024-01-21T11:41:05.931 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: verify_compression: 0 2024-01-21T11:41:05.931 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: read_amp_bytes_per_bit: 0 2024-01-21T11:41:05.931 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: format_version: 2 2024-01-21T11:41:05.931 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: enable_index_compression: 1 2024-01-21T11:41:05.931 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: block_align: 0 2024-01-21T11:41:05.931 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: 2024-01-21T11:41:05.931 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.write_buffer_size: 33554432 2024-01-21T11:41:05.931 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.max_write_buffer_number: 2 2024-01-21T11:41:05.931 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.compression: NoCompression 2024-01-21T11:41:05.931 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.bottommost_compression: Disabled 2024-01-21T11:41:05.931 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.prefix_extractor: nullptr 2024-01-21T11:41:05.931 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-01-21T11:41:05.932 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.num_levels: 7 2024-01-21T11:41:05.932 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-01-21T11:41:05.932 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-01-21T11:41:05.932 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-01-21T11:41:05.932 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-01-21T11:41:05.932 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-01-21T11:41:05.932 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-01-21T11:41:05.932 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-01-21T11:41:05.932 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-01-21T11:41:05.932 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-01-21T11:41:05.932 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.compression_opts.window_bits: -14 2024-01-21T11:41:05.932 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.compression_opts.level: 32767 2024-01-21T11:41:05.932 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.compression_opts.strategy: 0 2024-01-21T11:41:05.932 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-01-21T11:41:05.933 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-01-21T11:41:05.933 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.compression_opts.enabled: false 2024-01-21T11:41:05.933 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-01-21T11:41:05.933 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-01-21T11:41:05.933 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-01-21T11:41:05.933 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.target_file_size_base: 67108864 2024-01-21T11:41:05.933 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.target_file_size_multiplier: 1 2024-01-21T11:41:05.933 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-01-21T11:41:05.933 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-01-21T11:41:05.933 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-01-21T11:41:05.933 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-01-21T11:41:05.933 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-01-21T11:41:05.933 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-01-21T11:41:05.933 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-01-21T11:41:05.934 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-01-21T11:41:05.934 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-01-21T11:41:05.934 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-01-21T11:41:05.934 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-01-21T11:41:05.934 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-01-21T11:41:05.934 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.arena_block_size: 4194304 2024-01-21T11:41:05.934 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-01-21T11:41:05.934 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-01-21T11:41:05.934 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-01-21T11:41:05.934 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.disable_auto_compactions: 0 2024-01-21T11:41:05.934 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-01-21T11:41:05.934 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-01-21T11:41:05.934 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-01-21T11:41:05.934 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-01-21T11:41:05.934 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-01-21T11:41:05.934 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-01-21T11:41:05.935 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-01-21T11:41:05.935 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-01-21T11:41:05.935 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-01-21T11:41:05.935 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-01-21T11:41:05.935 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.table_properties_collectors: 2024-01-21T11:41:05.935 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.inplace_update_support: 0 2024-01-21T11:41:05.935 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-01-21T11:41:05.935 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-01-21T11:41:05.935 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-01-21T11:41:05.935 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.memtable_huge_page_size: 0 2024-01-21T11:41:05.935 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.bloom_locality: 0 2024-01-21T11:41:05.935 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.max_successive_merges: 0 2024-01-21T11:41:05.935 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-01-21T11:41:05.935 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.paranoid_file_checks: 0 2024-01-21T11:41:05.935 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.force_consistency_checks: 0 2024-01-21T11:41:05.936 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.report_bg_io_stats: 0 2024-01-21T11:41:05.936 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.ttl: 2592000 2024-01-21T11:41:05.936 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.477+0000 7fbbfa265700 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-01-21T11:41:05.936 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.478+0000 7fbbfa265700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi029/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 5, log_number is 4,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-01-21T11:41:05.936 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: 2024-01-21T11:41:05.936 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.478+0000 7fbbfa265700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 4 2024-01-21T11:41:05.936 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: 2024-01-21T11:41:05.936 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.478+0000 7fbbfa265700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1705837265480099, "job": 1, "event": "recovery_started", "log_files": [6]} 2024-01-21T11:41:05.936 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.478+0000 7fbbfa265700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #6 mode 2 2024-01-21T11:41:05.936 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.478+0000 7fbbfa265700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-01-21T11:41:05.936 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.479+0000 7fbbfa265700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1705837265481272, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 7, "file_size": 88694, "table_properties": {"data_size": 86809, "index_size": 362, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 709, "raw_key_size": 12384, "raw_average_key_size": 49, "raw_value_size": 79950, "raw_average_value_size": 321, "num_data_blocks": 11, "num_entries": 249, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1705837265, "oldest_key_time": 3, "file_creation_time": 0}} 2024-01-21T11:41:05.936 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.479+0000 7fbbfa265700 4 rocksdb: [version_set.cc:3826] Creating manifest 8 2024-01-21T11:41:05.936 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: 2024-01-21T11:41:05.936 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.479+0000 7fbbfa265700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1705837265481674, "job": 1, "event": "recovery_finished"} 2024-01-21T11:41:05.937 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 systemd[1]: Started Ceph mon.smithi029 for e4084074-b851-11ee-95b0-87774f69a715. 2024-01-21T11:41:05.937 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.480+0000 7fbbfa265700 4 rocksdb: DB pointer 0x5629b4eb5800 2024-01-21T11:41:05.937 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.480+0000 7fbbe107d700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-01-21T11:41:05.937 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.480+0000 7fbbe107d700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-01-21T11:41:05.937 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: ** DB Stats ** 2024-01-21T11:41:05.937 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: Uptime(secs): 0.0 total, 0.0 interval 2024-01-21T11:41:05.937 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-01-21T11:41:05.937 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-01-21T11:41:05.937 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-21T11:41:05.937 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-01-21T11:41:05.937 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-01-21T11:41:05.937 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-21T11:41:05.937 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: 2024-01-21T11:41:05.937 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: ** Compaction Stats [default] ** 2024-01-21T11:41:05.938 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-21T11:41:05.938 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-21T11:41:05.938 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: L0 2/0 88.31 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 117.5 0.00 0.00 1 0.001 0 0 2024-01-21T11:41:05.938 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: Sum 2/0 88.31 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 117.5 0.00 0.00 1 0.001 0 0 2024-01-21T11:41:05.938 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 117.5 0.00 0.00 1 0.001 0 0 2024-01-21T11:41:05.938 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: 2024-01-21T11:41:05.938 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: ** Compaction Stats [default] ** 2024-01-21T11:41:05.938 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-21T11:41:05.938 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-21T11:41:05.938 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 117.5 0.00 0.00 1 0.001 0 0 2024-01-21T11:41:05.938 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: Uptime(secs): 0.0 total, 0.0 interval 2024-01-21T11:41:05.938 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: Flush(GB): cumulative 0.000, interval 0.000 2024-01-21T11:41:05.938 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-21T11:41:05.938 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: AddFile(Total Files): cumulative 0, interval 0 2024-01-21T11:41:05.938 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: AddFile(L0 Files): cumulative 0, interval 0 2024-01-21T11:41:05.938 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: AddFile(Keys): cumulative 0, interval 0 2024-01-21T11:41:05.939 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: Cumulative compaction: 0.00 GB write, 25.66 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-21T11:41:05.939 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: Interval compaction: 0.00 GB write, 25.66 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-21T11:41:05.939 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-21T11:41:05.939 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: 2024-01-21T11:41:05.939 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: ** File Read Latency Histogram By Level [default] ** 2024-01-21T11:41:05.939 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: 2024-01-21T11:41:05.939 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: ** Compaction Stats [default] ** 2024-01-21T11:41:05.939 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-21T11:41:05.939 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-21T11:41:05.939 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: L0 2/0 88.31 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 117.5 0.00 0.00 1 0.001 0 0 2024-01-21T11:41:05.939 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: Sum 2/0 88.31 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 117.5 0.00 0.00 1 0.001 0 0 2024-01-21T11:41:05.939 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-21T11:41:05.939 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: 2024-01-21T11:41:05.939 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: ** Compaction Stats [default] ** 2024-01-21T11:41:05.939 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-21T11:41:05.939 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-21T11:41:05.940 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 117.5 0.00 0.00 1 0.001 0 0 2024-01-21T11:41:05.940 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: Uptime(secs): 0.0 total, 0.0 interval 2024-01-21T11:41:05.940 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: Flush(GB): cumulative 0.000, interval 0.000 2024-01-21T11:41:05.940 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-21T11:41:05.940 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: AddFile(Total Files): cumulative 0, interval 0 2024-01-21T11:41:05.940 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: AddFile(L0 Files): cumulative 0, interval 0 2024-01-21T11:41:05.940 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: AddFile(Keys): cumulative 0, interval 0 2024-01-21T11:41:05.940 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: Cumulative compaction: 0.00 GB write, 25.42 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-21T11:41:05.940 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-21T11:41:05.940 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-21T11:41:05.940 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: 2024-01-21T11:41:05.940 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: ** File Read Latency Histogram By Level [default] ** 2024-01-21T11:41:05.940 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: 2024-01-21T11:41:05.940 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.481+0000 7fbbfa265700 0 starting mon.smithi029 rank 0 at public addrs [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] at bind addrs [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi029 fsid e4084074-b851-11ee-95b0-87774f69a715 2024-01-21T11:41:05.940 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.482+0000 7fbbfa265700 1 mon.smithi029@-1(???) e1 preinit fsid e4084074-b851-11ee-95b0-87774f69a715 2024-01-21T11:41:05.941 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.482+0000 7fbbfa265700 0 mon.smithi029@-1(???).mds e1 new map 2024-01-21T11:41:05.941 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.482+0000 7fbbfa265700 0 mon.smithi029@-1(???).mds e1 print_map 2024-01-21T11:41:05.941 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: e1 2024-01-21T11:41:05.941 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: enable_multiple, ever_enabled_multiple: 1,1 2024-01-21T11:41:05.941 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-01-21T11:41:05.941 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: legacy client fscid: -1 2024-01-21T11:41:05.941 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: 2024-01-21T11:41:05.941 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: No filesystems configured 2024-01-21T11:41:05.941 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: 2024-01-21T11:41:05.941 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.483+0000 7fbbfa265700 0 mon.smithi029@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-01-21T11:41:05.941 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.483+0000 7fbbfa265700 0 mon.smithi029@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-01-21T11:41:05.941 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.483+0000 7fbbfa265700 0 mon.smithi029@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-01-21T11:41:05.941 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.483+0000 7fbbfa265700 0 mon.smithi029@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-01-21T11:41:05.942 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:41:03.480950+0000 mon.smithi029 (mon.0) 8 : audit [INF] from='client.? 172.21.15.29:0/523198880' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-01-21T11:41:05.942 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:41:03.484034+0000 mon.smithi029 (mon.0) 9 : audit [INF] from='client.? 172.21.15.29:0/523198880' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-01-21T11:41:05.942 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.483+0000 7fbbfa265700 1 mon.smithi029@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-01-21T11:41:05.942 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.484+0000 7fbbfa265700 20 expand_channel_meta expand map: {default=false} 2024-01-21T11:41:05.942 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.484+0000 7fbbfa265700 20 expand_channel_meta from 'false' to 'false' 2024-01-21T11:41:05.942 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.484+0000 7fbbfa265700 20 expand_channel_meta expanded map: {default=false} 2024-01-21T11:41:05.942 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.484+0000 7fbbfa265700 20 expand_channel_meta expand map: {default=info} 2024-01-21T11:41:05.942 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.484+0000 7fbbfa265700 20 expand_channel_meta from 'info' to 'info' 2024-01-21T11:41:05.942 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.484+0000 7fbbfa265700 20 expand_channel_meta expanded map: {default=info} 2024-01-21T11:41:05.942 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.484+0000 7fbbfa265700 20 expand_channel_meta expand map: {default=daemon} 2024-01-21T11:41:05.942 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.484+0000 7fbbfa265700 20 expand_channel_meta from 'daemon' to 'daemon' 2024-01-21T11:41:05.942 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.484+0000 7fbbfa265700 20 expand_channel_meta expanded map: {default=daemon} 2024-01-21T11:41:05.942 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.484+0000 7fbbfa265700 20 expand_channel_meta expand map: {default=debug} 2024-01-21T11:41:05.942 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.484+0000 7fbbfa265700 20 expand_channel_meta from 'debug' to 'debug' 2024-01-21T11:41:05.942 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:41:05.484+0000 7fbbfa265700 20 expand_channel_meta expanded map: {default=debug} 2024-01-21T11:41:05.942 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:41:05.500595+0000 mon.smithi029 (mon.0) 1 : cluster [INF] mon.smithi029 is new leader, mons smithi029 in quorum (ranks 0) 2024-01-21T11:41:05.943 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:41:05.500662+0000 mon.smithi029 (mon.0) 2 : cluster [DBG] monmap e1: 1 mons at {smithi029=[v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0]} 2024-01-21T11:41:05.943 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:41:05.500713+0000 mon.smithi029 (mon.0) 3 : cluster [DBG] fsmap 2024-01-21T11:41:05.943 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:41:05.500741+0000 mon.smithi029 (mon.0) 4 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-01-21T11:41:05.943 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:41:05.501121+0000 mon.smithi029 (mon.0) 5 : cluster [DBG] mgrmap e1: no daemons active 2024-01-21T11:41:06.289 INFO:teuthology.orchestra.run.smithi029.stderr:Wrote config to /etc/ceph/ceph.conf 2024-01-21T11:41:06.290 INFO:teuthology.orchestra.run.smithi029.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-01-21T11:41:06.290 INFO:teuthology.orchestra.run.smithi029.stderr:Creating mgr... 2024-01-21T11:41:06.290 INFO:teuthology.orchestra.run.smithi029.stderr:Verifying port 9283 ... 2024-01-21T11:41:06.291 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl daemon-reload 2024-01-21T11:41:06.390 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl stop ceph-e4084074-b851-11ee-95b0-87774f69a715@mgr.smithi029.blhxxk 2024-01-21T11:41:06.415 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl reset-failed ceph-e4084074-b851-11ee-95b0-87774f69a715@mgr.smithi029.blhxxk 2024-01-21T11:41:06.421 INFO:teuthology.orchestra.run.smithi029.stderr:systemctl: Failed to reset failed state of unit ceph-e4084074-b851-11ee-95b0-87774f69a715@mgr.smithi029.blhxxk.service: Unit ceph-e4084074-b851-11ee-95b0-87774f69a715@mgr.smithi029.blhxxk.service not loaded. 2024-01-21T11:41:06.421 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl enable ceph-e4084074-b851-11ee-95b0-87774f69a715@mgr.smithi029.blhxxk 2024-01-21T11:41:06.427 INFO:teuthology.orchestra.run.smithi029.stderr:systemctl: Created symlink /etc/systemd/system/ceph-e4084074-b851-11ee-95b0-87774f69a715.target.wants/ceph-e4084074-b851-11ee-95b0-87774f69a715@mgr.smithi029.blhxxk.service → /etc/systemd/system/ceph-e4084074-b851-11ee-95b0-87774f69a715@.service. 2024-01-21T11:41:06.523 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl start ceph-e4084074-b851-11ee-95b0-87774f69a715@mgr.smithi029.blhxxk 2024-01-21T11:41:06.883 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl is-enabled firewalld.service 2024-01-21T11:41:06.892 INFO:teuthology.orchestra.run.smithi029.stderr:systemctl: disabled 2024-01-21T11:41:06.893 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl is-active firewalld.service 2024-01-21T11:41:06.901 INFO:teuthology.orchestra.run.smithi029.stderr:systemctl: inactive 2024-01-21T11:41:06.902 INFO:teuthology.orchestra.run.smithi029.stderr:firewalld.service is not enabled 2024-01-21T11:41:06.902 INFO:teuthology.orchestra.run.smithi029.stderr:Not possible to enable service . firewalld.service is not available 2024-01-21T11:41:06.902 INFO:teuthology.orchestra.run.smithi029.stderr:Not possible to open ports <[]>. firewalld.service is not available 2024-01-21T11:41:06.903 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl is-enabled firewalld.service 2024-01-21T11:41:06.911 INFO:teuthology.orchestra.run.smithi029.stderr:systemctl: disabled 2024-01-21T11:41:06.912 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl is-active firewalld.service 2024-01-21T11:41:06.920 INFO:teuthology.orchestra.run.smithi029.stderr:systemctl: inactive 2024-01-21T11:41:06.921 INFO:teuthology.orchestra.run.smithi029.stderr:firewalld.service is not enabled 2024-01-21T11:41:06.921 INFO:teuthology.orchestra.run.smithi029.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-01-21T11:41:06.921 INFO:teuthology.orchestra.run.smithi029.stderr:Waiting for mgr to start... 2024-01-21T11:41:06.921 INFO:teuthology.orchestra.run.smithi029.stderr:Waiting for mgr... 2024-01-21T11:41:06.921 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e4084074-b851-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp77cpgrhz:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpea05xzin:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2024-01-21T11:41:07.169 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:06 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:41:05.899152+0000 mon.smithi029 (mon.0) 6 : audit [INF] from='client.? 172.21.15.29:0/873282698' entity='client.admin' 2024-01-21T11:41:07.308 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:07.306+0000 7fcfb5d60700 1 Processor -- start 2024-01-21T11:41:07.309 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:07.306+0000 7fcfb5d60700 1 -- start start 2024-01-21T11:41:07.309 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:07.307+0000 7fcfb5d60700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfb00fab80 0x7fcfb00fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T11:41:07.309 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:07.307+0000 7fcfb5d60700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfb00fb490 con 0x7fcfb00fab80 2024-01-21T11:41:07.311 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:07.307+0000 7fcfaf7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfb00fab80 0x7fcfb00fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:41:07.311 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:07.307+0000 7fcfaf7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfb00fab80 0x7fcfb00fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49456/0 (socket says 172.21.15.29:49456) 2024-01-21T11:41:07.311 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:07.307+0000 7fcfaf7fe700 1 -- 172.21.15.29:0/1077289790 learned_addr learned my addr 172.21.15.29:0/1077289790 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-21T11:41:07.311 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:07.307+0000 7fcfaf7fe700 1 -- 172.21.15.29:0/1077289790 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfb00fb5d0 con 0x7fcfb00fab80 2024-01-21T11:41:07.311 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:07.307+0000 7fcfaf7fe700 1 --2- 172.21.15.29:0/1077289790 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfb00fab80 0x7fcfb00fafa0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fcfa000b5f0 tx=0x7fcfa0006ee0).ready entity=mon.0 client_cookie=ffdf50d73cef7d64 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:41:07.311 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:07.308+0000 7fcfae7fc700 1 -- 172.21.15.29:0/1077289790 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcfa000d600 con 0x7fcfb00fab80 2024-01-21T11:41:07.311 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:07.308+0000 7fcfae7fc700 1 -- 172.21.15.29:0/1077289790 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fcfa000dbd0 con 0x7fcfb00fab80 2024-01-21T11:41:07.311 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:07.308+0000 7fcfae7fc700 1 -- 172.21.15.29:0/1077289790 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcfa0011c00 con 0x7fcfb00fab80 2024-01-21T11:41:07.311 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:07.308+0000 7fcfb5d60700 1 -- 172.21.15.29:0/1077289790 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfb00fab80 msgr2=0x7fcfb00fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:41:07.311 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:07.308+0000 7fcfb5d60700 1 --2- 172.21.15.29:0/1077289790 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfb00fab80 0x7fcfb00fafa0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fcfa000b5f0 tx=0x7fcfa0006ee0).stop 2024-01-21T11:41:07.311 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:07.308+0000 7fcfb5d60700 1 -- 172.21.15.29:0/1077289790 shutdown_connections 2024-01-21T11:41:07.312 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:07.308+0000 7fcfb5d60700 1 --2- 172.21.15.29:0/1077289790 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfb00fab80 0x7fcfb00fafa0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:41:07.313 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:07.308+0000 7fcfb5d60700 1 -- 172.21.15.29:0/1077289790 >> 172.21.15.29:0/1077289790 conn(0x7fcfb00f63c0 msgr2=0x7fcfb00f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:41:07.313 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:07.309+0000 7fcfb5d60700 1 -- 172.21.15.29:0/1077289790 shutdown_connections 2024-01-21T11:41:07.313 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:07.309+0000 7fcfb5d60700 1 -- 172.21.15.29:0/1077289790 wait complete. 2024-01-21T11:41:07.313 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:07.309+0000 7fcfb5d60700 1 Processor -- start 2024-01-21T11:41:07.313 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:07.309+0000 7fcfb5d60700 1 -- start start 2024-01-21T11:41:07.314 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:07.309+0000 7fcfb5d60700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfb00fab80 0x7fcfb010aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:41:07.314 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:07.309+0000 7fcfb5d60700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfb010aef0 con 0x7fcfb00fab80 2024-01-21T11:41:07.314 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:07.310+0000 7fcfaf7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfb00fab80 0x7fcfb010aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:41:07.314 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:07.310+0000 7fcfaf7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfb00fab80 0x7fcfb010aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49468/0 (socket says 172.21.15.29:49468) 2024-01-21T11:41:07.314 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:07.310+0000 7fcfaf7fe700 1 -- 172.21.15.29:0/2774323774 learned_addr learned my addr 172.21.15.29:0/2774323774 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-21T11:41:07.314 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:07.310+0000 7fcfaf7fe700 1 -- 172.21.15.29:0/2774323774 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfa000b200 con 0x7fcfb00fab80 2024-01-21T11:41:07.314 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:07.310+0000 7fcfaf7fe700 1 --2- 172.21.15.29:0/2774323774 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfb00fab80 0x7fcfb010aa00 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fcfa000b1d0 tx=0x7fcfa0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:41:07.314 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:07.310+0000 7fcf97fff700 1 -- 172.21.15.29:0/2774323774 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcfa001b410 con 0x7fcfb00fab80 2024-01-21T11:41:07.314 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:07.310+0000 7fcf97fff700 1 -- 172.21.15.29:0/2774323774 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fcfa001b9e0 con 0x7fcfb00fab80 2024-01-21T11:41:07.314 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:07.310+0000 7fcfb5d60700 1 -- 172.21.15.29:0/2774323774 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfb010b0f0 con 0x7fcfb00fab80 2024-01-21T11:41:07.314 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:07.310+0000 7fcf97fff700 1 -- 172.21.15.29:0/2774323774 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcfa001a840 con 0x7fcfb00fab80 2024-01-21T11:41:07.314 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:07.310+0000 7fcfb5d60700 1 -- 172.21.15.29:0/2774323774 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfb010b500 con 0x7fcfb00fab80 2024-01-21T11:41:07.314 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:07.311+0000 7fcf97fff700 1 -- 172.21.15.29:0/2774323774 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7fcfa000f070 con 0x7fcfb00fab80 2024-01-21T11:41:07.315 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:07.311+0000 7fcf97fff700 1 -- 172.21.15.29:0/2774323774 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fcfa0025c90 con 0x7fcfb00fab80 2024-01-21T11:41:07.315 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:07.311+0000 7fcfb5d60700 1 -- 172.21.15.29:0/2774323774 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfb0045ad0 con 0x7fcfb00fab80 2024-01-21T11:41:07.315 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:07.313+0000 7fcf97fff700 1 -- 172.21.15.29:0/2774323774 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fcfa0016070 con 0x7fcfb00fab80 2024-01-21T11:41:07.391 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:07.384+0000 7fcfb5d60700 1 -- 172.21.15.29:0/2774323774 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fcfb00fc890 con 0x7fcfb00fab80 2024-01-21T11:41:07.391 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:07.384+0000 7fcf97fff700 1 -- 172.21.15.29:0/2774323774 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7fcfa001aea0 con 0x7fcfb00fab80 2024-01-21T11:41:07.391 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:07.391 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: { 2024-01-21T11:41:07.391 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "fsid": "e4084074-b851-11ee-95b0-87774f69a715", 2024-01-21T11:41:07.392 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "health": { 2024-01-21T11:41:07.392 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-01-21T11:41:07.392 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "checks": {}, 2024-01-21T11:41:07.392 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "mutes": [] 2024-01-21T11:41:07.392 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: }, 2024-01-21T11:41:07.392 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-01-21T11:41:07.392 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "quorum": [ 2024-01-21T11:41:07.392 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 0 2024-01-21T11:41:07.392 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: ], 2024-01-21T11:41:07.392 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "quorum_names": [ 2024-01-21T11:41:07.392 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "smithi029" 2024-01-21T11:41:07.392 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: ], 2024-01-21T11:41:07.392 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "quorum_age": 1, 2024-01-21T11:41:07.392 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "monmap": { 2024-01-21T11:41:07.393 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T11:41:07.393 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-01-21T11:41:07.393 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_mons": 1 2024-01-21T11:41:07.393 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: }, 2024-01-21T11:41:07.393 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "osdmap": { 2024-01-21T11:41:07.393 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T11:41:07.393 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_osds": 0, 2024-01-21T11:41:07.393 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-01-21T11:41:07.393 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-01-21T11:41:07.393 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-01-21T11:41:07.393 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-01-21T11:41:07.393 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-01-21T11:41:07.393 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: }, 2024-01-21T11:41:07.393 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "pgmap": { 2024-01-21T11:41:07.393 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-01-21T11:41:07.393 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-01-21T11:41:07.394 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_pools": 0, 2024-01-21T11:41:07.394 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_objects": 0, 2024-01-21T11:41:07.394 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-01-21T11:41:07.394 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-01-21T11:41:07.394 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-01-21T11:41:07.394 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "bytes_total": 0 2024-01-21T11:41:07.394 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: }, 2024-01-21T11:41:07.394 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "fsmap": { 2024-01-21T11:41:07.394 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T11:41:07.394 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "by_rank": [], 2024-01-21T11:41:07.394 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "up:standby": 0 2024-01-21T11:41:07.394 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: }, 2024-01-21T11:41:07.394 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "mgrmap": { 2024-01-21T11:41:07.394 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "available": false, 2024-01-21T11:41:07.394 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-01-21T11:41:07.395 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "modules": [ 2024-01-21T11:41:07.395 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "iostat", 2024-01-21T11:41:07.395 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "restful" 2024-01-21T11:41:07.395 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: ], 2024-01-21T11:41:07.395 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "services": {} 2024-01-21T11:41:07.395 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: }, 2024-01-21T11:41:07.395 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "servicemap": { 2024-01-21T11:41:07.395 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T11:41:07.395 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "modified": "2024-01-21T11:41:02.157560+0000", 2024-01-21T11:41:07.395 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "services": {} 2024-01-21T11:41:07.395 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: }, 2024-01-21T11:41:07.395 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "progress_events": {} 2024-01-21T11:41:07.395 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: } 2024-01-21T11:41:07.395 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:07.385+0000 7fcfb5d60700 1 -- 172.21.15.29:0/2774323774 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfb00fab80 msgr2=0x7fcfb010aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:41:07.395 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:07.385+0000 7fcfb5d60700 1 --2- 172.21.15.29:0/2774323774 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfb00fab80 0x7fcfb010aa00 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fcfa000b1d0 tx=0x7fcfa0006ee0).stop 2024-01-21T11:41:07.395 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:07.385+0000 7fcfb5d60700 1 -- 172.21.15.29:0/2774323774 shutdown_connections 2024-01-21T11:41:07.396 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:07.385+0000 7fcfb5d60700 1 --2- 172.21.15.29:0/2774323774 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfb00fab80 0x7fcfb010aa00 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:41:07.396 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:07.385+0000 7fcfb5d60700 1 -- 172.21.15.29:0/2774323774 >> 172.21.15.29:0/2774323774 conn(0x7fcfb00f63c0 msgr2=0x7fcfb0070090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:41:07.396 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:07.385+0000 7fcfb5d60700 1 -- 172.21.15.29:0/2774323774 shutdown_connections 2024-01-21T11:41:07.396 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:07.385+0000 7fcfb5d60700 1 -- 172.21.15.29:0/2774323774 wait complete. 2024-01-21T11:41:07.764 INFO:teuthology.orchestra.run.smithi029.stderr:mgr not available, waiting (1/15)... 2024-01-21T11:41:08.168 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:07 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:41:07.386419+0000 mon.smithi029 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.29:0/2774323774' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-01-21T11:41:09.766 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e4084074-b851-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp77cpgrhz:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpea05xzin:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2024-01-21T11:41:10.149 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:10.146+0000 7f0a49817700 1 Processor -- start 2024-01-21T11:41:10.149 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:10.146+0000 7f0a49817700 1 -- start start 2024-01-21T11:41:10.149 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:10.146+0000 7f0a49817700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a44070430 0x7f0a440fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T11:41:10.150 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:10.146+0000 7f0a49817700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a440fedd0 con 0x7f0a44070430 2024-01-21T11:41:10.150 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:10.146+0000 7f0a42ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a44070430 0x7f0a440fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:41:10.150 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:10.146+0000 7f0a42ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a44070430 0x7f0a440fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49472/0 (socket says 172.21.15.29:49472) 2024-01-21T11:41:10.150 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:10.146+0000 7f0a42ffd700 1 -- 172.21.15.29:0/1928774816 learned_addr learned my addr 172.21.15.29:0/1928774816 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-21T11:41:10.150 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:10.147+0000 7f0a42ffd700 1 -- 172.21.15.29:0/1928774816 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a440fef10 con 0x7f0a44070430 2024-01-21T11:41:10.150 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:10.147+0000 7f0a42ffd700 1 --2- 172.21.15.29:0/1928774816 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a44070430 0x7f0a440fe8e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f0a3400b5f0 tx=0x7f0a34006ee0).ready entity=mon.0 client_cookie=90112f157011fc09 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:41:10.150 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:10.147+0000 7f0a41ffb700 1 -- 172.21.15.29:0/1928774816 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a3400d600 con 0x7f0a44070430 2024-01-21T11:41:10.150 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:10.147+0000 7f0a41ffb700 1 -- 172.21.15.29:0/1928774816 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f0a3400dbd0 con 0x7f0a44070430 2024-01-21T11:41:10.150 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:10.147+0000 7f0a41ffb700 1 -- 172.21.15.29:0/1928774816 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a34011c00 con 0x7f0a44070430 2024-01-21T11:41:10.150 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:10.148+0000 7f0a49817700 1 -- 172.21.15.29:0/1928774816 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a44070430 msgr2=0x7f0a440fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:41:10.150 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:10.148+0000 7f0a49817700 1 --2- 172.21.15.29:0/1928774816 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a44070430 0x7f0a440fe8e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f0a3400b5f0 tx=0x7f0a34006ee0).stop 2024-01-21T11:41:10.150 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:10.148+0000 7f0a49817700 1 -- 172.21.15.29:0/1928774816 shutdown_connections 2024-01-21T11:41:10.150 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:10.148+0000 7f0a49817700 1 --2- 172.21.15.29:0/1928774816 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a44070430 0x7f0a440fe8e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:41:10.150 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:10.148+0000 7f0a49817700 1 -- 172.21.15.29:0/1928774816 >> 172.21.15.29:0/1928774816 conn(0x7f0a440f63c0 msgr2=0x7f0a440f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:41:10.151 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:10.148+0000 7f0a49817700 1 -- 172.21.15.29:0/1928774816 shutdown_connections 2024-01-21T11:41:10.151 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:10.148+0000 7f0a49817700 1 -- 172.21.15.29:0/1928774816 wait complete. 2024-01-21T11:41:10.152 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:10.149+0000 7f0a49817700 1 Processor -- start 2024-01-21T11:41:10.152 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:10.149+0000 7f0a49817700 1 -- start start 2024-01-21T11:41:10.152 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:10.149+0000 7f0a49817700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a44070430 0x7f0a4410aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:41:10.153 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:10.149+0000 7f0a49817700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a4410aef0 con 0x7f0a44070430 2024-01-21T11:41:10.153 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:10.149+0000 7f0a42ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a44070430 0x7f0a4410aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:41:10.153 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:10.149+0000 7f0a42ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a44070430 0x7f0a4410aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49478/0 (socket says 172.21.15.29:49478) 2024-01-21T11:41:10.153 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:10.149+0000 7f0a42ffd700 1 -- 172.21.15.29:0/2392042485 learned_addr learned my addr 172.21.15.29:0/2392042485 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-21T11:41:10.153 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:10.150+0000 7f0a42ffd700 1 -- 172.21.15.29:0/2392042485 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a3400b200 con 0x7f0a44070430 2024-01-21T11:41:10.153 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:10.150+0000 7f0a42ffd700 1 --2- 172.21.15.29:0/2392042485 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a44070430 0x7f0a4410aa00 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f0a3400b190 tx=0x7f0a34006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:41:10.153 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:10.150+0000 7f0a2b7fe700 1 -- 172.21.15.29:0/2392042485 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a3401b410 con 0x7f0a44070430 2024-01-21T11:41:10.153 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:10.150+0000 7f0a49817700 1 -- 172.21.15.29:0/2392042485 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a4410b0f0 con 0x7f0a44070430 2024-01-21T11:41:10.153 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:10.150+0000 7f0a2b7fe700 1 -- 172.21.15.29:0/2392042485 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f0a3401b9e0 con 0x7f0a44070430 2024-01-21T11:41:10.153 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:10.150+0000 7f0a2b7fe700 1 -- 172.21.15.29:0/2392042485 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a3401a880 con 0x7f0a44070430 2024-01-21T11:41:10.153 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:10.150+0000 7f0a49817700 1 -- 172.21.15.29:0/2392042485 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a4410b500 con 0x7f0a44070430 2024-01-21T11:41:10.153 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:10.150+0000 7f0a2b7fe700 1 -- 172.21.15.29:0/2392042485 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f0a3400f070 con 0x7f0a44070430 2024-01-21T11:41:10.153 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:10.150+0000 7f0a2b7fe700 1 -- 172.21.15.29:0/2392042485 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f0a34025c90 con 0x7f0a44070430 2024-01-21T11:41:10.154 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:10.151+0000 7f0a49817700 1 -- 172.21.15.29:0/2392042485 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a44045ad0 con 0x7f0a44070430 2024-01-21T11:41:10.154 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:10.153+0000 7f0a2b7fe700 1 -- 172.21.15.29:0/2392042485 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f0a34016070 con 0x7f0a44070430 2024-01-21T11:41:10.251 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:10.250+0000 7f0a49817700 1 -- 172.21.15.29:0/2392042485 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f0a44104890 con 0x7f0a44070430 2024-01-21T11:41:10.260 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:10.251+0000 7f0a2b7fe700 1 -- 172.21.15.29:0/2392042485 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f0a3401bb50 con 0x7f0a44070430 2024-01-21T11:41:10.260 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:10.260 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: { 2024-01-21T11:41:10.260 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "fsid": "e4084074-b851-11ee-95b0-87774f69a715", 2024-01-21T11:41:10.260 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "health": { 2024-01-21T11:41:10.260 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-01-21T11:41:10.260 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "checks": {}, 2024-01-21T11:41:10.260 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "mutes": [] 2024-01-21T11:41:10.260 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: }, 2024-01-21T11:41:10.260 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-01-21T11:41:10.260 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "quorum": [ 2024-01-21T11:41:10.260 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 0 2024-01-21T11:41:10.261 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: ], 2024-01-21T11:41:10.261 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "quorum_names": [ 2024-01-21T11:41:10.261 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "smithi029" 2024-01-21T11:41:10.261 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: ], 2024-01-21T11:41:10.261 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "quorum_age": 4, 2024-01-21T11:41:10.261 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "monmap": { 2024-01-21T11:41:10.261 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T11:41:10.261 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-01-21T11:41:10.261 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_mons": 1 2024-01-21T11:41:10.261 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: }, 2024-01-21T11:41:10.261 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "osdmap": { 2024-01-21T11:41:10.261 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T11:41:10.261 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_osds": 0, 2024-01-21T11:41:10.261 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-01-21T11:41:10.261 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-01-21T11:41:10.261 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-01-21T11:41:10.262 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-01-21T11:41:10.262 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-01-21T11:41:10.262 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: }, 2024-01-21T11:41:10.262 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "pgmap": { 2024-01-21T11:41:10.262 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-01-21T11:41:10.262 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-01-21T11:41:10.262 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_pools": 0, 2024-01-21T11:41:10.262 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_objects": 0, 2024-01-21T11:41:10.262 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-01-21T11:41:10.262 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-01-21T11:41:10.262 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-01-21T11:41:10.262 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "bytes_total": 0 2024-01-21T11:41:10.262 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: }, 2024-01-21T11:41:10.262 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "fsmap": { 2024-01-21T11:41:10.262 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T11:41:10.262 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "by_rank": [], 2024-01-21T11:41:10.262 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "up:standby": 0 2024-01-21T11:41:10.263 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: }, 2024-01-21T11:41:10.263 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "mgrmap": { 2024-01-21T11:41:10.263 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "available": false, 2024-01-21T11:41:10.263 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-01-21T11:41:10.263 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "modules": [ 2024-01-21T11:41:10.263 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "iostat", 2024-01-21T11:41:10.263 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "restful" 2024-01-21T11:41:10.263 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: ], 2024-01-21T11:41:10.263 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "services": {} 2024-01-21T11:41:10.263 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: }, 2024-01-21T11:41:10.263 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "servicemap": { 2024-01-21T11:41:10.263 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T11:41:10.263 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "modified": "2024-01-21T11:41:02.157560+0000", 2024-01-21T11:41:10.263 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "services": {} 2024-01-21T11:41:10.263 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: }, 2024-01-21T11:41:10.263 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "progress_events": {} 2024-01-21T11:41:10.264 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: } 2024-01-21T11:41:10.264 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:10.252+0000 7f0a49817700 1 -- 172.21.15.29:0/2392042485 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a44070430 msgr2=0x7f0a4410aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:41:10.264 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:10.252+0000 7f0a49817700 1 --2- 172.21.15.29:0/2392042485 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a44070430 0x7f0a4410aa00 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f0a3400b190 tx=0x7f0a34006ee0).stop 2024-01-21T11:41:10.264 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:10.252+0000 7f0a49817700 1 -- 172.21.15.29:0/2392042485 shutdown_connections 2024-01-21T11:41:10.264 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:10.252+0000 7f0a49817700 1 --2- 172.21.15.29:0/2392042485 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a44070430 0x7f0a4410aa00 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:41:10.264 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:10.252+0000 7f0a49817700 1 -- 172.21.15.29:0/2392042485 >> 172.21.15.29:0/2392042485 conn(0x7f0a440f63c0 msgr2=0x7f0a440f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:41:10.264 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:10.252+0000 7f0a49817700 1 -- 172.21.15.29:0/2392042485 shutdown_connections 2024-01-21T11:41:10.264 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:10.253+0000 7f0a49817700 1 -- 172.21.15.29:0/2392042485 wait complete. 2024-01-21T11:41:10.540 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:10 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:41:10.252924+0000 mon.smithi029 (mon.0) 8 : audit [DBG] from='client.? 172.21.15.29:0/2392042485' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-01-21T11:41:10.604 INFO:teuthology.orchestra.run.smithi029.stderr:mgr not available, waiting (2/15)... 2024-01-21T11:41:12.418 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:12 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:41:12.064730+0000 mon.smithi029 (mon.0) 9 : cluster [INF] Activating manager daemon smithi029.blhxxk 2024-01-21T11:41:12.419 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:12 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:41:12.067417+0000 mon.smithi029 (mon.0) 10 : cluster [DBG] mgrmap e2: smithi029.blhxxk(active, starting, since 0.00280549s) 2024-01-21T11:41:12.419 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:12 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:41:12.068193+0000 mon.smithi029 (mon.0) 11 : audit [DBG] from='mgr.14100 172.21.15.29:0/3237516849' entity='mgr.smithi029.blhxxk' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-21T11:41:12.419 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:12 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:41:12.068324+0000 mon.smithi029 (mon.0) 12 : audit [DBG] from='mgr.14100 172.21.15.29:0/3237516849' entity='mgr.smithi029.blhxxk' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-21T11:41:12.419 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:12 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:41:12.068430+0000 mon.smithi029 (mon.0) 13 : audit [DBG] from='mgr.14100 172.21.15.29:0/3237516849' entity='mgr.smithi029.blhxxk' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-21T11:41:12.419 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:12 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:41:12.068580+0000 mon.smithi029 (mon.0) 14 : audit [DBG] from='mgr.14100 172.21.15.29:0/3237516849' entity='mgr.smithi029.blhxxk' cmd=[{"prefix": "mon metadata", "id": "smithi029"}]: dispatch 2024-01-21T11:41:12.419 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:12 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:41:12.068751+0000 mon.smithi029 (mon.0) 15 : audit [DBG] from='mgr.14100 172.21.15.29:0/3237516849' entity='mgr.smithi029.blhxxk' cmd=[{"prefix": "mgr metadata", "who": "smithi029.blhxxk", "id": "smithi029.blhxxk"}]: dispatch 2024-01-21T11:41:12.419 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:12 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:41:12.082230+0000 mon.smithi029 (mon.0) 16 : cluster [INF] Manager daemon smithi029.blhxxk is now available 2024-01-21T11:41:12.419 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:12 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:41:12.089856+0000 mon.smithi029 (mon.0) 17 : audit [INF] from='mgr.14100 172.21.15.29:0/3237516849' entity='mgr.smithi029.blhxxk' 2024-01-21T11:41:12.419 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:12 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:41:12.094151+0000 mon.smithi029 (mon.0) 18 : audit [INF] from='mgr.14100 172.21.15.29:0/3237516849' entity='mgr.smithi029.blhxxk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi029.blhxxk/mirror_snapshot_schedule"}]: dispatch 2024-01-21T11:41:12.419 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:12 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:41:12.095318+0000 mon.smithi029 (mon.0) 19 : audit [INF] from='mgr.14100 172.21.15.29:0/3237516849' entity='mgr.smithi029.blhxxk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi029.blhxxk/trash_purge_schedule"}]: dispatch 2024-01-21T11:41:12.419 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:12 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:41:12.099923+0000 mon.smithi029 (mon.0) 20 : audit [INF] from='mgr.14100 172.21.15.29:0/3237516849' entity='mgr.smithi029.blhxxk' 2024-01-21T11:41:12.419 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:12 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:41:12.102049+0000 mon.smithi029 (mon.0) 21 : audit [INF] from='mgr.14100 172.21.15.29:0/3237516849' entity='mgr.smithi029.blhxxk' 2024-01-21T11:41:12.607 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e4084074-b851-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp77cpgrhz:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpea05xzin:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2024-01-21T11:41:12.962 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:12.960+0000 7fa8e5a71700 1 Processor -- start 2024-01-21T11:41:12.962 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:12.960+0000 7fa8e5a71700 1 -- start start 2024-01-21T11:41:12.962 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:12.960+0000 7fa8e5a71700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8e00fbcb0 0x7fa8e00fc0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T11:41:12.962 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:12.960+0000 7fa8e5a71700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e00fc5c0 con 0x7fa8e00fbcb0 2024-01-21T11:41:12.962 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:12.960+0000 7fa8df7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8e00fbcb0 0x7fa8e00fc0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:41:12.962 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:12.960+0000 7fa8df7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8e00fbcb0 0x7fa8e00fc0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58170/0 (socket says 172.21.15.29:58170) 2024-01-21T11:41:12.963 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:12.960+0000 7fa8df7fe700 1 -- 172.21.15.29:0/2856323752 learned_addr learned my addr 172.21.15.29:0/2856323752 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-21T11:41:12.963 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:12.960+0000 7fa8df7fe700 1 -- 172.21.15.29:0/2856323752 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8e00fc700 con 0x7fa8e00fbcb0 2024-01-21T11:41:12.963 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:12.961+0000 7fa8df7fe700 1 --2- 172.21.15.29:0/2856323752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8e00fbcb0 0x7fa8e00fc0d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fa8d000b5f0 tx=0x7fa8d0006ee0).ready entity=mon.0 client_cookie=5d64ed3924926585 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:41:12.963 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:12.961+0000 7fa8de7fc700 1 -- 172.21.15.29:0/2856323752 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa8d000d600 con 0x7fa8e00fbcb0 2024-01-21T11:41:12.963 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:12.961+0000 7fa8de7fc700 1 -- 172.21.15.29:0/2856323752 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fa8d000dbd0 con 0x7fa8e00fbcb0 2024-01-21T11:41:12.963 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:12.961+0000 7fa8de7fc700 1 -- 172.21.15.29:0/2856323752 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa8d0011c00 con 0x7fa8e00fbcb0 2024-01-21T11:41:12.963 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:12.961+0000 7fa8e5a71700 1 -- 172.21.15.29:0/2856323752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8e00fbcb0 msgr2=0x7fa8e00fc0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:41:12.963 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:12.961+0000 7fa8e5a71700 1 --2- 172.21.15.29:0/2856323752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8e00fbcb0 0x7fa8e00fc0d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fa8d000b5f0 tx=0x7fa8d0006ee0).stop 2024-01-21T11:41:12.963 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:12.961+0000 7fa8e5a71700 1 -- 172.21.15.29:0/2856323752 shutdown_connections 2024-01-21T11:41:12.963 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:12.961+0000 7fa8e5a71700 1 --2- 172.21.15.29:0/2856323752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8e00fbcb0 0x7fa8e00fc0d0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:41:12.963 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:12.961+0000 7fa8e5a71700 1 -- 172.21.15.29:0/2856323752 >> 172.21.15.29:0/2856323752 conn(0x7fa8e00f74b0 msgr2=0x7fa8e00f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:41:12.963 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:12.961+0000 7fa8e5a71700 1 -- 172.21.15.29:0/2856323752 shutdown_connections 2024-01-21T11:41:12.963 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:12.961+0000 7fa8e5a71700 1 -- 172.21.15.29:0/2856323752 wait complete. 2024-01-21T11:41:12.963 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:12.962+0000 7fa8e5a71700 1 Processor -- start 2024-01-21T11:41:12.963 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:12.962+0000 7fa8e5a71700 1 -- start start 2024-01-21T11:41:12.965 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:12.963+0000 7fa8e5a71700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8e00fbcb0 0x7fa8e010aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:41:12.965 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:12.963+0000 7fa8e5a71700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e010af50 con 0x7fa8e00fbcb0 2024-01-21T11:41:12.965 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:12.963+0000 7fa8df7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8e00fbcb0 0x7fa8e010aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:41:12.965 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:12.963+0000 7fa8df7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8e00fbcb0 0x7fa8e010aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58176/0 (socket says 172.21.15.29:58176) 2024-01-21T11:41:12.965 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:12.963+0000 7fa8df7fe700 1 -- 172.21.15.29:0/3256215286 learned_addr learned my addr 172.21.15.29:0/3256215286 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-21T11:41:12.965 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:12.963+0000 7fa8df7fe700 1 -- 172.21.15.29:0/3256215286 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8d000b200 con 0x7fa8e00fbcb0 2024-01-21T11:41:12.965 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:12.963+0000 7fa8df7fe700 1 --2- 172.21.15.29:0/3256215286 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8e00fbcb0 0x7fa8e010aa60 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fa8d000e040 tx=0x7fa8d0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:41:12.965 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:12.963+0000 7fa8c7fff700 1 -- 172.21.15.29:0/3256215286 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa8d001b410 con 0x7fa8e00fbcb0 2024-01-21T11:41:12.965 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:12.963+0000 7fa8e5a71700 1 -- 172.21.15.29:0/3256215286 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8e010b150 con 0x7fa8e00fbcb0 2024-01-21T11:41:12.965 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:12.963+0000 7fa8c7fff700 1 -- 172.21.15.29:0/3256215286 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fa8d001b9e0 con 0x7fa8e00fbcb0 2024-01-21T11:41:12.965 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:12.963+0000 7fa8c7fff700 1 -- 172.21.15.29:0/3256215286 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa8d001a820 con 0x7fa8e00fbcb0 2024-01-21T11:41:12.966 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:12.963+0000 7fa8e5a71700 1 -- 172.21.15.29:0/3256215286 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8e010b560 con 0x7fa8e00fbcb0 2024-01-21T11:41:12.966 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:12.964+0000 7fa8c7fff700 1 -- 172.21.15.29:0/3256215286 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 2) v1 ==== 35959+0+0 (secure 0 0 0) 0x7fa8d000f070 con 0x7fa8e00fbcb0 2024-01-21T11:41:12.966 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:12.964+0000 7fa8c7fff700 1 -- 172.21.15.29:0/3256215286 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fa8d00411e0 con 0x7fa8e00fbcb0 2024-01-21T11:41:12.966 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:12.964+0000 7fa8e5a71700 1 -- 172.21.15.29:0/3256215286 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8e0045ad0 con 0x7fa8e00fbcb0 2024-01-21T11:41:12.966 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:12.967+0000 7fa8c7fff700 1 -- 172.21.15.29:0/3256215286 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fa8d0016070 con 0x7fa8e00fbcb0 2024-01-21T11:41:13.042 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:13.036+0000 7fa8e5a71700 1 -- 172.21.15.29:0/3256215286 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fa8e00fd9c0 con 0x7fa8e00fbcb0 2024-01-21T11:41:13.042 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:13.037+0000 7fa8c7fff700 1 -- 172.21.15.29:0/3256215286 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7fa8d0022b70 con 0x7fa8e00fbcb0 2024-01-21T11:41:13.042 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:13.042 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: { 2024-01-21T11:41:13.042 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "fsid": "e4084074-b851-11ee-95b0-87774f69a715", 2024-01-21T11:41:13.042 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "health": { 2024-01-21T11:41:13.043 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-01-21T11:41:13.043 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "checks": {}, 2024-01-21T11:41:13.043 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "mutes": [] 2024-01-21T11:41:13.043 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: }, 2024-01-21T11:41:13.043 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-01-21T11:41:13.043 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "quorum": [ 2024-01-21T11:41:13.043 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 0 2024-01-21T11:41:13.043 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: ], 2024-01-21T11:41:13.043 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "quorum_names": [ 2024-01-21T11:41:13.043 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "smithi029" 2024-01-21T11:41:13.043 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: ], 2024-01-21T11:41:13.043 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "quorum_age": 7, 2024-01-21T11:41:13.043 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "monmap": { 2024-01-21T11:41:13.043 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T11:41:13.043 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-01-21T11:41:13.043 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_mons": 1 2024-01-21T11:41:13.043 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: }, 2024-01-21T11:41:13.044 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "osdmap": { 2024-01-21T11:41:13.044 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T11:41:13.044 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_osds": 0, 2024-01-21T11:41:13.044 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-01-21T11:41:13.044 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-01-21T11:41:13.044 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-01-21T11:41:13.044 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-01-21T11:41:13.044 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-01-21T11:41:13.044 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: }, 2024-01-21T11:41:13.044 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "pgmap": { 2024-01-21T11:41:13.044 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-01-21T11:41:13.044 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-01-21T11:41:13.044 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_pools": 0, 2024-01-21T11:41:13.044 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_objects": 0, 2024-01-21T11:41:13.044 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-01-21T11:41:13.044 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-01-21T11:41:13.045 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-01-21T11:41:13.045 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "bytes_total": 0 2024-01-21T11:41:13.045 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: }, 2024-01-21T11:41:13.045 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "fsmap": { 2024-01-21T11:41:13.045 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T11:41:13.045 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "by_rank": [], 2024-01-21T11:41:13.045 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "up:standby": 0 2024-01-21T11:41:13.045 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: }, 2024-01-21T11:41:13.045 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "mgrmap": { 2024-01-21T11:41:13.045 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "available": false, 2024-01-21T11:41:13.045 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-01-21T11:41:13.045 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "modules": [ 2024-01-21T11:41:13.045 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "iostat", 2024-01-21T11:41:13.045 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "restful" 2024-01-21T11:41:13.045 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: ], 2024-01-21T11:41:13.045 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "services": {} 2024-01-21T11:41:13.045 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: }, 2024-01-21T11:41:13.046 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "servicemap": { 2024-01-21T11:41:13.046 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T11:41:13.046 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "modified": "2024-01-21T11:41:02.157560+0000", 2024-01-21T11:41:13.046 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "services": {} 2024-01-21T11:41:13.046 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: }, 2024-01-21T11:41:13.046 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "progress_events": {} 2024-01-21T11:41:13.046 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: } 2024-01-21T11:41:13.046 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:13.037+0000 7fa8e5a71700 1 -- 172.21.15.29:0/3256215286 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8e00fbcb0 msgr2=0x7fa8e010aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:41:13.046 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:13.037+0000 7fa8e5a71700 1 --2- 172.21.15.29:0/3256215286 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8e00fbcb0 0x7fa8e010aa60 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fa8d000e040 tx=0x7fa8d0006ee0).stop 2024-01-21T11:41:13.046 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:13.037+0000 7fa8e5a71700 1 -- 172.21.15.29:0/3256215286 shutdown_connections 2024-01-21T11:41:13.046 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:13.037+0000 7fa8e5a71700 1 --2- 172.21.15.29:0/3256215286 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8e00fbcb0 0x7fa8e010aa60 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:41:13.046 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:13.037+0000 7fa8e5a71700 1 -- 172.21.15.29:0/3256215286 >> 172.21.15.29:0/3256215286 conn(0x7fa8e00f74b0 msgr2=0x7fa8e00f8f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:41:13.046 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:13.037+0000 7fa8e5a71700 1 -- 172.21.15.29:0/3256215286 shutdown_connections 2024-01-21T11:41:13.046 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:13.037+0000 7fa8e5a71700 1 -- 172.21.15.29:0/3256215286 wait complete. 2024-01-21T11:41:13.307 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:13 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:41:13.037717+0000 mon.smithi029 (mon.0) 22 : audit [DBG] from='client.? 172.21.15.29:0/3256215286' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-01-21T11:41:13.307 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:13 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:41:13.069714+0000 mon.smithi029 (mon.0) 23 : cluster [DBG] mgrmap e3: smithi029.blhxxk(active, since 1.0051s) 2024-01-21T11:41:13.371 INFO:teuthology.orchestra.run.smithi029.stderr:mgr not available, waiting (3/15)... 2024-01-21T11:41:15.374 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e4084074-b851-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp77cpgrhz:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpea05xzin:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2024-01-21T11:41:15.418 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:15 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:41:14.119954+0000 mon.smithi029 (mon.0) 24 : cluster [DBG] mgrmap e4: smithi029.blhxxk(active, since 2s) 2024-01-21T11:41:15.770 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:15.767+0000 7f994eda6700 1 Processor -- start 2024-01-21T11:41:15.770 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:15.767+0000 7f994eda6700 1 -- start start 2024-01-21T11:41:15.770 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:15.767+0000 7f994eda6700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99480fa9d0 0x7f99480fadf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T11:41:15.770 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:15.767+0000 7f994eda6700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99480fb2e0 con 0x7f99480fa9d0 2024-01-21T11:41:15.770 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:15.767+0000 7f994dda4700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99480fa9d0 0x7f99480fadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:41:15.770 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:15.767+0000 7f994dda4700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99480fa9d0 0x7f99480fadf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58184/0 (socket says 172.21.15.29:58184) 2024-01-21T11:41:15.770 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:15.767+0000 7f994dda4700 1 -- 172.21.15.29:0/3036029438 learned_addr learned my addr 172.21.15.29:0/3036029438 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-21T11:41:15.770 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:15.768+0000 7f994dda4700 1 -- 172.21.15.29:0/3036029438 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99480fb420 con 0x7f99480fa9d0 2024-01-21T11:41:15.770 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:15.768+0000 7f994dda4700 1 --2- 172.21.15.29:0/3036029438 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99480fa9d0 0x7f99480fadf0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f994400b5f0 tx=0x7f9944006ee0).ready entity=mon.0 client_cookie=8ae84adc0576919f server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:41:15.770 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:15.768+0000 7f994cda2700 1 -- 172.21.15.29:0/3036029438 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f994400d600 con 0x7f99480fa9d0 2024-01-21T11:41:15.771 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:15.768+0000 7f994cda2700 1 -- 172.21.15.29:0/3036029438 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f994400dbd0 con 0x7f99480fa9d0 2024-01-21T11:41:15.771 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:15.768+0000 7f994cda2700 1 -- 172.21.15.29:0/3036029438 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9944011c00 con 0x7f99480fa9d0 2024-01-21T11:41:15.771 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:15.768+0000 7f994eda6700 1 -- 172.21.15.29:0/3036029438 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99480fa9d0 msgr2=0x7f99480fadf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:41:15.771 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:15.768+0000 7f994eda6700 1 --2- 172.21.15.29:0/3036029438 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99480fa9d0 0x7f99480fadf0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f994400b5f0 tx=0x7f9944006ee0).stop 2024-01-21T11:41:15.771 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:15.769+0000 7f994eda6700 1 -- 172.21.15.29:0/3036029438 shutdown_connections 2024-01-21T11:41:15.771 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:15.769+0000 7f994eda6700 1 --2- 172.21.15.29:0/3036029438 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99480fa9d0 0x7f99480fadf0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:41:15.771 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:15.769+0000 7f994eda6700 1 -- 172.21.15.29:0/3036029438 >> 172.21.15.29:0/3036029438 conn(0x7f99480f6210 msgr2=0x7f99480f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:41:15.771 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:15.769+0000 7f994eda6700 1 -- 172.21.15.29:0/3036029438 shutdown_connections 2024-01-21T11:41:15.771 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:15.769+0000 7f994eda6700 1 -- 172.21.15.29:0/3036029438 wait complete. 2024-01-21T11:41:15.771 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:15.769+0000 7f994eda6700 1 Processor -- start 2024-01-21T11:41:15.773 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:15.769+0000 7f994eda6700 1 -- start start 2024-01-21T11:41:15.773 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:15.769+0000 7f994eda6700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99480fa9d0 0x7f9948100c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:41:15.773 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:15.770+0000 7f994eda6700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99480ff0e0 con 0x7f99480fa9d0 2024-01-21T11:41:15.773 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:15.770+0000 7f994dda4700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99480fa9d0 0x7f9948100c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:41:15.773 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:15.770+0000 7f994dda4700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99480fa9d0 0x7f9948100c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58186/0 (socket says 172.21.15.29:58186) 2024-01-21T11:41:15.774 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:15.770+0000 7f994dda4700 1 -- 172.21.15.29:0/247220789 learned_addr learned my addr 172.21.15.29:0/247220789 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-21T11:41:15.774 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:15.770+0000 7f994dda4700 1 -- 172.21.15.29:0/247220789 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f994400b200 con 0x7f99480fa9d0 2024-01-21T11:41:15.774 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:15.770+0000 7f994dda4700 1 --2- 172.21.15.29:0/247220789 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99480fa9d0 0x7f9948100c60 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f994400e040 tx=0x7f9944006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:41:15.774 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:15.770+0000 7f99367fc700 1 -- 172.21.15.29:0/247220789 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f994401b410 con 0x7f99480fa9d0 2024-01-21T11:41:15.774 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:15.770+0000 7f99367fc700 1 -- 172.21.15.29:0/247220789 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f994401b9e0 con 0x7f99480fa9d0 2024-01-21T11:41:15.774 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:15.770+0000 7f994eda6700 1 -- 172.21.15.29:0/247220789 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99480ff280 con 0x7f99480fa9d0 2024-01-21T11:41:15.774 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:15.770+0000 7f99367fc700 1 -- 172.21.15.29:0/247220789 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f994401a820 con 0x7f99480fa9d0 2024-01-21T11:41:15.774 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:15.770+0000 7f994eda6700 1 -- 172.21.15.29:0/247220789 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99480ff690 con 0x7f99480fa9d0 2024-01-21T11:41:15.774 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:15.771+0000 7f99367fc700 1 -- 172.21.15.29:0/247220789 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 4) v1 ==== 36109+0+0 (secure 0 0 0) 0x7f994400f070 con 0x7f99480fa9d0 2024-01-21T11:41:15.774 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:15.771+0000 7f99367fc700 1 --2- 172.21.15.29:0/247220789 >> [v2:172.21.15.29:6800/616654539,v1:172.21.15.29:6801/616654539] conn(0x7f993002fd50 0x7f9930032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:41:15.774 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:15.771+0000 7f99367fc700 1 -- 172.21.15.29:0/247220789 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f9944041350 con 0x7f99480fa9d0 2024-01-21T11:41:15.774 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:15.771+0000 7f994eda6700 1 -- 172.21.15.29:0/247220789 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9948045ad0 con 0x7f99480fa9d0 2024-01-21T11:41:15.775 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:15.771+0000 7f994d5a3700 1 --2- 172.21.15.29:0/247220789 >> [v2:172.21.15.29:6800/616654539,v1:172.21.15.29:6801/616654539] conn(0x7f993002fd50 0x7f9930032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:41:15.775 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:15.772+0000 7f994d5a3700 1 --2- 172.21.15.29:0/247220789 >> [v2:172.21.15.29:6800/616654539,v1:172.21.15.29:6801/616654539] conn(0x7f993002fd50 0x7f9930032210 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f99380096a0 tx=0x7f9938006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:41:15.776 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:15.776+0000 7f99367fc700 1 -- 172.21.15.29:0/247220789 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9944016070 con 0x7f99480fa9d0 2024-01-21T11:41:16.000 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:15.993+0000 7f994eda6700 1 -- 172.21.15.29:0/247220789 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f9948027a00 con 0x7f99480fa9d0 2024-01-21T11:41:16.000 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:15.993+0000 7f99367fc700 1 -- 172.21.15.29:0/247220789 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f9944025460 con 0x7f99480fa9d0 2024-01-21T11:41:16.000 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:16.000 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: { 2024-01-21T11:41:16.000 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "fsid": "e4084074-b851-11ee-95b0-87774f69a715", 2024-01-21T11:41:16.000 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "health": { 2024-01-21T11:41:16.000 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-01-21T11:41:16.001 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "checks": {}, 2024-01-21T11:41:16.001 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "mutes": [] 2024-01-21T11:41:16.001 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: }, 2024-01-21T11:41:16.001 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-01-21T11:41:16.001 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "quorum": [ 2024-01-21T11:41:16.001 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 0 2024-01-21T11:41:16.001 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: ], 2024-01-21T11:41:16.001 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "quorum_names": [ 2024-01-21T11:41:16.001 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "smithi029" 2024-01-21T11:41:16.001 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: ], 2024-01-21T11:41:16.001 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "quorum_age": 10, 2024-01-21T11:41:16.001 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "monmap": { 2024-01-21T11:41:16.001 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T11:41:16.001 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-01-21T11:41:16.001 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_mons": 1 2024-01-21T11:41:16.001 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: }, 2024-01-21T11:41:16.001 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "osdmap": { 2024-01-21T11:41:16.002 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T11:41:16.002 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_osds": 0, 2024-01-21T11:41:16.002 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-01-21T11:41:16.002 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-01-21T11:41:16.002 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-01-21T11:41:16.002 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-01-21T11:41:16.002 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-01-21T11:41:16.002 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: }, 2024-01-21T11:41:16.002 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "pgmap": { 2024-01-21T11:41:16.002 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-01-21T11:41:16.002 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-01-21T11:41:16.002 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_pools": 0, 2024-01-21T11:41:16.002 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_objects": 0, 2024-01-21T11:41:16.002 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-01-21T11:41:16.002 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-01-21T11:41:16.002 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-01-21T11:41:16.003 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "bytes_total": 0 2024-01-21T11:41:16.003 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: }, 2024-01-21T11:41:16.003 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "fsmap": { 2024-01-21T11:41:16.003 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T11:41:16.003 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "by_rank": [], 2024-01-21T11:41:16.003 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "up:standby": 0 2024-01-21T11:41:16.003 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: }, 2024-01-21T11:41:16.003 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "mgrmap": { 2024-01-21T11:41:16.003 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "available": true, 2024-01-21T11:41:16.003 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-01-21T11:41:16.003 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "modules": [ 2024-01-21T11:41:16.003 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "iostat", 2024-01-21T11:41:16.003 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "restful" 2024-01-21T11:41:16.003 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: ], 2024-01-21T11:41:16.003 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "services": {} 2024-01-21T11:41:16.004 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: }, 2024-01-21T11:41:16.004 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "servicemap": { 2024-01-21T11:41:16.004 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T11:41:16.004 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "modified": "2024-01-21T11:41:02.157560+0000", 2024-01-21T11:41:16.004 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "services": {} 2024-01-21T11:41:16.004 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: }, 2024-01-21T11:41:16.004 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "progress_events": {} 2024-01-21T11:41:16.004 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: } 2024-01-21T11:41:16.004 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:15.995+0000 7f994eda6700 1 -- 172.21.15.29:0/247220789 >> [v2:172.21.15.29:6800/616654539,v1:172.21.15.29:6801/616654539] conn(0x7f993002fd50 msgr2=0x7f9930032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:41:16.004 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:15.995+0000 7f994eda6700 1 --2- 172.21.15.29:0/247220789 >> [v2:172.21.15.29:6800/616654539,v1:172.21.15.29:6801/616654539] conn(0x7f993002fd50 0x7f9930032210 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f99380096a0 tx=0x7f9938006b40).stop 2024-01-21T11:41:16.004 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:15.995+0000 7f994eda6700 1 -- 172.21.15.29:0/247220789 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99480fa9d0 msgr2=0x7f9948100c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:41:16.004 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:15.995+0000 7f994eda6700 1 --2- 172.21.15.29:0/247220789 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99480fa9d0 0x7f9948100c60 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f994400e040 tx=0x7f9944006ee0).stop 2024-01-21T11:41:16.004 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:15.995+0000 7f994eda6700 1 -- 172.21.15.29:0/247220789 shutdown_connections 2024-01-21T11:41:16.004 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:15.995+0000 7f994eda6700 1 --2- 172.21.15.29:0/247220789 >> [v2:172.21.15.29:6800/616654539,v1:172.21.15.29:6801/616654539] conn(0x7f993002fd50 0x7f9930032210 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:41:16.004 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:15.995+0000 7f994eda6700 1 --2- 172.21.15.29:0/247220789 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99480fa9d0 0x7f9948100c60 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:41:16.004 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:15.995+0000 7f994eda6700 1 -- 172.21.15.29:0/247220789 >> 172.21.15.29:0/247220789 conn(0x7f99480f6210 msgr2=0x7f9948070090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:41:16.005 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:15.995+0000 7f994eda6700 1 -- 172.21.15.29:0/247220789 shutdown_connections 2024-01-21T11:41:16.005 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:15.995+0000 7f994eda6700 1 -- 172.21.15.29:0/247220789 wait complete. 2024-01-21T11:41:16.354 INFO:teuthology.orchestra.run.smithi029.stderr:mgr is available 2024-01-21T11:41:16.355 INFO:teuthology.orchestra.run.smithi029.stderr:Enabling cephadm module... 2024-01-21T11:41:16.355 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e4084074-b851-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp77cpgrhz:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpea05xzin:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable cephadm 2024-01-21T11:41:16.668 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:16 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:41:15.994584+0000 mon.smithi029 (mon.0) 25 : audit [DBG] from='client.? 172.21.15.29:0/247220789' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-01-21T11:41:16.714 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:16.713+0000 7facf8ad3700 1 Processor -- start 2024-01-21T11:41:16.714 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:16.713+0000 7facf8ad3700 1 -- start start 2024-01-21T11:41:16.714 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:16.713+0000 7facf8ad3700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7facf40703d0 0x7facf4100d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T11:41:16.714 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:16.713+0000 7facf8ad3700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facf40fc730 con 0x7facf40703d0 2024-01-21T11:41:16.714 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:16.714+0000 7facf259c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7facf40703d0 0x7facf4100d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:41:16.714 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:16.714+0000 7facf259c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7facf40703d0 0x7facf4100d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58194/0 (socket says 172.21.15.29:58194) 2024-01-21T11:41:16.714 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:16.714+0000 7facf259c700 1 -- 172.21.15.29:0/2766544849 learned_addr learned my addr 172.21.15.29:0/2766544849 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-21T11:41:16.718 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:16.714+0000 7facf259c700 1 -- 172.21.15.29:0/2766544849 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facf4101230 con 0x7facf40703d0 2024-01-21T11:41:16.718 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:16.714+0000 7facf259c700 1 --2- 172.21.15.29:0/2766544849 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7facf40703d0 0x7facf4100d40 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7face800b5f0 tx=0x7face8006ee0).ready entity=mon.0 client_cookie=3b9ad841776f9a1 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:41:16.718 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:16.714+0000 7facf159a700 1 -- 172.21.15.29:0/2766544849 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7face800d600 con 0x7facf40703d0 2024-01-21T11:41:16.718 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:16.714+0000 7facf159a700 1 -- 172.21.15.29:0/2766544849 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7face800dbd0 con 0x7facf40703d0 2024-01-21T11:41:16.718 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:16.714+0000 7facf159a700 1 -- 172.21.15.29:0/2766544849 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7face8011c00 con 0x7facf40703d0 2024-01-21T11:41:16.718 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:16.715+0000 7facf8ad3700 1 -- 172.21.15.29:0/2766544849 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7facf40703d0 msgr2=0x7facf4100d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:41:16.718 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:16.715+0000 7facf8ad3700 1 --2- 172.21.15.29:0/2766544849 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7facf40703d0 0x7facf4100d40 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7face800b5f0 tx=0x7face8006ee0).stop 2024-01-21T11:41:16.718 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:16.715+0000 7facf8ad3700 1 -- 172.21.15.29:0/2766544849 shutdown_connections 2024-01-21T11:41:16.718 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:16.715+0000 7facf8ad3700 1 --2- 172.21.15.29:0/2766544849 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7facf40703d0 0x7facf4100d40 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:41:16.718 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:16.715+0000 7facf8ad3700 1 -- 172.21.15.29:0/2766544849 >> 172.21.15.29:0/2766544849 conn(0x7facf40f63c0 msgr2=0x7facf40f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:41:16.718 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:16.715+0000 7facf8ad3700 1 -- 172.21.15.29:0/2766544849 shutdown_connections 2024-01-21T11:41:16.719 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:16.715+0000 7facf8ad3700 1 -- 172.21.15.29:0/2766544849 wait complete. 2024-01-21T11:41:16.719 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:16.715+0000 7facf8ad3700 1 Processor -- start 2024-01-21T11:41:16.719 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:16.715+0000 7facf8ad3700 1 -- start start 2024-01-21T11:41:16.719 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:16.716+0000 7facf8ad3700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7facf40703d0 0x7facf410cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:41:16.719 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:16.716+0000 7facf8ad3700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facf410d0a0 con 0x7facf40703d0 2024-01-21T11:41:16.719 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:16.716+0000 7facf259c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7facf40703d0 0x7facf410cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:41:16.719 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:16.716+0000 7facf259c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7facf40703d0 0x7facf410cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58198/0 (socket says 172.21.15.29:58198) 2024-01-21T11:41:16.719 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:16.716+0000 7facf259c700 1 -- 172.21.15.29:0/3139669672 learned_addr learned my addr 172.21.15.29:0/3139669672 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-21T11:41:16.719 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:16.716+0000 7facf259c700 1 -- 172.21.15.29:0/3139669672 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7face800b200 con 0x7facf40703d0 2024-01-21T11:41:16.719 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:16.716+0000 7facf259c700 1 --2- 172.21.15.29:0/3139669672 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7facf40703d0 0x7facf410cbb0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7face800b190 tx=0x7face8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:41:16.719 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:16.716+0000 7face2ffd700 1 -- 172.21.15.29:0/3139669672 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7face801b410 con 0x7facf40703d0 2024-01-21T11:41:16.719 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:16.716+0000 7face2ffd700 1 -- 172.21.15.29:0/3139669672 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7face801b9e0 con 0x7facf40703d0 2024-01-21T11:41:16.719 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:16.716+0000 7facf8ad3700 1 -- 172.21.15.29:0/3139669672 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facf410d2a0 con 0x7facf40703d0 2024-01-21T11:41:16.719 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:16.716+0000 7face2ffd700 1 -- 172.21.15.29:0/3139669672 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7face801a880 con 0x7facf40703d0 2024-01-21T11:41:16.719 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:16.716+0000 7facf8ad3700 1 -- 172.21.15.29:0/3139669672 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facf410d6b0 con 0x7facf40703d0 2024-01-21T11:41:16.719 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:16.717+0000 7face2ffd700 1 -- 172.21.15.29:0/3139669672 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 4) v1 ==== 36109+0+0 (secure 0 0 0) 0x7face800f070 con 0x7facf40703d0 2024-01-21T11:41:16.720 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:16.717+0000 7face2ffd700 1 --2- 172.21.15.29:0/3139669672 >> [v2:172.21.15.29:6800/616654539,v1:172.21.15.29:6801/616654539] conn(0x7facd802fd00 0x7facd80321c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:41:16.720 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:16.717+0000 7face2ffd700 1 -- 172.21.15.29:0/3139669672 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7face80411b0 con 0x7facf40703d0 2024-01-21T11:41:16.720 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:16.717+0000 7facf1d9b700 1 --2- 172.21.15.29:0/3139669672 >> [v2:172.21.15.29:6800/616654539,v1:172.21.15.29:6801/616654539] conn(0x7facd802fd00 0x7facd80321c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:41:16.720 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:16.717+0000 7facf8ad3700 1 -- 172.21.15.29:0/3139669672 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facf4059100 con 0x7facf40703d0 2024-01-21T11:41:16.720 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:16.718+0000 7facf1d9b700 1 --2- 172.21.15.29:0/3139669672 >> [v2:172.21.15.29:6800/616654539,v1:172.21.15.29:6801/616654539] conn(0x7facd802fd00 0x7facd80321c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7face40096a0 tx=0x7face4006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:41:16.722 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:16.722+0000 7face2ffd700 1 -- 172.21.15.29:0/3139669672 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7face8016070 con 0x7facf40703d0 2024-01-21T11:41:16.920 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:16.920+0000 7facf8ad3700 1 -- 172.21.15.29:0/3139669672 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7facf4106a20 con 0x7facf40703d0 2024-01-21T11:41:17.493 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:17.492+0000 7face2ffd700 1 -- 172.21.15.29:0/3139669672 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7face8025460 con 0x7facf40703d0 2024-01-21T11:41:17.493 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:17.494+0000 7face2ffd700 1 -- 172.21.15.29:0/3139669672 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7face8019730 con 0x7facf40703d0 2024-01-21T11:41:17.497 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:17.496+0000 7facf8ad3700 1 -- 172.21.15.29:0/3139669672 >> [v2:172.21.15.29:6800/616654539,v1:172.21.15.29:6801/616654539] conn(0x7facd802fd00 msgr2=0x7facd80321c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:41:17.497 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:17.496+0000 7facf8ad3700 1 --2- 172.21.15.29:0/3139669672 >> [v2:172.21.15.29:6800/616654539,v1:172.21.15.29:6801/616654539] conn(0x7facd802fd00 0x7facd80321c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7face40096a0 tx=0x7face4006b40).stop 2024-01-21T11:41:17.497 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:17.496+0000 7facf8ad3700 1 -- 172.21.15.29:0/3139669672 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7facf40703d0 msgr2=0x7facf410cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:41:17.498 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:17.496+0000 7facf8ad3700 1 --2- 172.21.15.29:0/3139669672 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7facf40703d0 0x7facf410cbb0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7face800b190 tx=0x7face8006ee0).stop 2024-01-21T11:41:17.498 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:17.496+0000 7facf8ad3700 1 -- 172.21.15.29:0/3139669672 shutdown_connections 2024-01-21T11:41:17.498 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:17.496+0000 7facf8ad3700 1 --2- 172.21.15.29:0/3139669672 >> [v2:172.21.15.29:6800/616654539,v1:172.21.15.29:6801/616654539] conn(0x7facd802fd00 0x7facd80321c0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:41:17.498 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:17.496+0000 7facf8ad3700 1 --2- 172.21.15.29:0/3139669672 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7facf40703d0 0x7facf410cbb0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:41:17.498 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:17.496+0000 7facf8ad3700 1 -- 172.21.15.29:0/3139669672 >> 172.21.15.29:0/3139669672 conn(0x7facf40f63c0 msgr2=0x7facf40f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:41:17.498 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:17.497+0000 7facf8ad3700 1 -- 172.21.15.29:0/3139669672 shutdown_connections 2024-01-21T11:41:17.498 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:17.497+0000 7facf8ad3700 1 -- 172.21.15.29:0/3139669672 wait complete. 2024-01-21T11:41:17.789 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:17 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:41:16.921166+0000 mon.smithi029 (mon.0) 26 : audit [INF] from='client.? 172.21.15.29:0/3139669672' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-01-21T11:41:17.837 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e4084074-b851-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp77cpgrhz:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpea05xzin:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2024-01-21T11:41:18.220 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:18.219+0000 7fb5ebbc3700 1 Processor -- start 2024-01-21T11:41:18.221 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:18.219+0000 7fb5ebbc3700 1 -- start start 2024-01-21T11:41:18.221 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:18.219+0000 7fb5ebbc3700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5e40703b0 0x7fb5e4100d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T11:41:18.221 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:18.219+0000 7fb5ebbc3700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e40fc750 con 0x7fb5e40703b0 2024-01-21T11:41:18.221 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:18.220+0000 7fb5e995f700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5e40703b0 0x7fb5e4100d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:41:18.221 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:18.220+0000 7fb5e995f700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5e40703b0 0x7fb5e4100d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58214/0 (socket says 172.21.15.29:58214) 2024-01-21T11:41:18.221 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:18.220+0000 7fb5e995f700 1 -- 172.21.15.29:0/1532225330 learned_addr learned my addr 172.21.15.29:0/1532225330 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-21T11:41:18.221 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:18.220+0000 7fb5e995f700 1 -- 172.21.15.29:0/1532225330 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5e4101250 con 0x7fb5e40703b0 2024-01-21T11:41:18.221 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:18.220+0000 7fb5e995f700 1 --2- 172.21.15.29:0/1532225330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5e40703b0 0x7fb5e4100d60 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7fb5e000b5f0 tx=0x7fb5e0006ee0).ready entity=mon.0 client_cookie=12987c80e7ca5504 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:41:18.221 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:18.220+0000 7fb5e895d700 1 -- 172.21.15.29:0/1532225330 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb5e000d600 con 0x7fb5e40703b0 2024-01-21T11:41:18.221 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:18.220+0000 7fb5e895d700 1 -- 172.21.15.29:0/1532225330 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fb5e000dbd0 con 0x7fb5e40703b0 2024-01-21T11:41:18.221 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:18.220+0000 7fb5e895d700 1 -- 172.21.15.29:0/1532225330 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb5e0011c00 con 0x7fb5e40703b0 2024-01-21T11:41:18.223 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:18.221+0000 7fb5ebbc3700 1 -- 172.21.15.29:0/1532225330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5e40703b0 msgr2=0x7fb5e4100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:41:18.223 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:18.221+0000 7fb5ebbc3700 1 --2- 172.21.15.29:0/1532225330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5e40703b0 0x7fb5e4100d60 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7fb5e000b5f0 tx=0x7fb5e0006ee0).stop 2024-01-21T11:41:18.223 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:18.221+0000 7fb5ebbc3700 1 -- 172.21.15.29:0/1532225330 shutdown_connections 2024-01-21T11:41:18.224 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:18.221+0000 7fb5ebbc3700 1 --2- 172.21.15.29:0/1532225330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5e40703b0 0x7fb5e4100d60 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:41:18.224 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:18.221+0000 7fb5ebbc3700 1 -- 172.21.15.29:0/1532225330 >> 172.21.15.29:0/1532225330 conn(0x7fb5e40f63c0 msgr2=0x7fb5e40f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:41:18.224 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:18.221+0000 7fb5ebbc3700 1 -- 172.21.15.29:0/1532225330 shutdown_connections 2024-01-21T11:41:18.224 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:18.221+0000 7fb5ebbc3700 1 -- 172.21.15.29:0/1532225330 wait complete. 2024-01-21T11:41:18.224 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:18.222+0000 7fb5ebbc3700 1 Processor -- start 2024-01-21T11:41:18.224 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:18.222+0000 7fb5ebbc3700 1 -- start start 2024-01-21T11:41:18.224 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:18.222+0000 7fb5ebbc3700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5e40703b0 0x7fb5e410cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:41:18.224 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:18.222+0000 7fb5ebbc3700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e410d100 con 0x7fb5e40703b0 2024-01-21T11:41:18.224 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:18.222+0000 7fb5e995f700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5e40703b0 0x7fb5e410cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:41:18.224 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:18.222+0000 7fb5e995f700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5e40703b0 0x7fb5e410cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58216/0 (socket says 172.21.15.29:58216) 2024-01-21T11:41:18.224 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:18.222+0000 7fb5e995f700 1 -- 172.21.15.29:0/3374913584 learned_addr learned my addr 172.21.15.29:0/3374913584 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-21T11:41:18.224 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:18.223+0000 7fb5e995f700 1 -- 172.21.15.29:0/3374913584 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5e000b200 con 0x7fb5e40703b0 2024-01-21T11:41:18.224 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:18.223+0000 7fb5e995f700 1 --2- 172.21.15.29:0/3374913584 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5e40703b0 0x7fb5e410cc10 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7fb5e000b190 tx=0x7fb5e0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:41:18.224 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:18.223+0000 7fb5d27fc700 1 -- 172.21.15.29:0/3374913584 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb5e001b410 con 0x7fb5e40703b0 2024-01-21T11:41:18.224 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:18.223+0000 7fb5d27fc700 1 -- 172.21.15.29:0/3374913584 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fb5e001b9e0 con 0x7fb5e40703b0 2024-01-21T11:41:18.224 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:18.223+0000 7fb5ebbc3700 1 -- 172.21.15.29:0/3374913584 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5e410d300 con 0x7fb5e40703b0 2024-01-21T11:41:18.225 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:18.223+0000 7fb5d27fc700 1 -- 172.21.15.29:0/3374913584 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb5e001a880 con 0x7fb5e40703b0 2024-01-21T11:41:18.225 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:18.223+0000 7fb5ebbc3700 1 -- 172.21.15.29:0/3374913584 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5e410d710 con 0x7fb5e40703b0 2024-01-21T11:41:18.225 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:18.224+0000 7fb5ebbc3700 1 -- 172.21.15.29:0/3374913584 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5e4045ad0 con 0x7fb5e40703b0 2024-01-21T11:41:18.225 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:18.224+0000 7fb5d27fc700 1 -- 172.21.15.29:0/3374913584 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fb5e000f070 con 0x7fb5e40703b0 2024-01-21T11:41:18.225 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:18.225+0000 7fb5d27fc700 1 --2- 172.21.15.29:0/3374913584 >> [v2:172.21.15.29:6800/616654539,v1:172.21.15.29:6801/616654539] conn(0x7fb5cc02fd50 0x7fb5cc032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:41:18.226 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:18.225+0000 7fb5d27fc700 1 -- 172.21.15.29:0/3374913584 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fb5e0041270 con 0x7fb5e40703b0 2024-01-21T11:41:18.226 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:18.225+0000 7fb5e915e700 1 -- 172.21.15.29:0/3374913584 >> [v2:172.21.15.29:6800/616654539,v1:172.21.15.29:6801/616654539] conn(0x7fb5cc02fd50 msgr2=0x7fb5cc032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/616654539 2024-01-21T11:41:18.226 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:18.225+0000 7fb5e915e700 1 --2- 172.21.15.29:0/3374913584 >> [v2:172.21.15.29:6800/616654539,v1:172.21.15.29:6801/616654539] conn(0x7fb5cc02fd50 0x7fb5cc032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-01-21T11:41:18.229 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:18.229+0000 7fb5d27fc700 1 -- 172.21.15.29:0/3374913584 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb5e0019cd0 con 0x7fb5e40703b0 2024-01-21T11:41:18.426 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:18.426+0000 7fb5e915e700 1 -- 172.21.15.29:0/3374913584 >> [v2:172.21.15.29:6800/616654539,v1:172.21.15.29:6801/616654539] conn(0x7fb5cc02fd50 msgr2=0x7fb5cc032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/616654539 2024-01-21T11:41:18.426 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:18.426+0000 7fb5e915e700 1 --2- 172.21.15.29:0/3374913584 >> [v2:172.21.15.29:6800/616654539,v1:172.21.15.29:6801/616654539] conn(0x7fb5cc02fd50 0x7fb5cc032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-01-21T11:41:18.429 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:18.429+0000 7fb5ebbc3700 1 -- 172.21.15.29:0/3374913584 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fb5e410e820 con 0x7fb5e40703b0 2024-01-21T11:41:18.430 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:18.430+0000 7fb5d27fc700 1 -- 172.21.15.29:0/3374913584 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7fb5e0016070 con 0x7fb5e40703b0 2024-01-21T11:41:18.430 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: { 2024-01-21T11:41:18.430 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "epoch": 5, 2024-01-21T11:41:18.430 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "available": true, 2024-01-21T11:41:18.430 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "active_name": "smithi029.blhxxk", 2024-01-21T11:41:18.430 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_standby": 0 2024-01-21T11:41:18.430 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: } 2024-01-21T11:41:18.432 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:18.431+0000 7fb5ebbc3700 1 -- 172.21.15.29:0/3374913584 >> [v2:172.21.15.29:6800/616654539,v1:172.21.15.29:6801/616654539] conn(0x7fb5cc02fd50 msgr2=0x7fb5cc032210 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-21T11:41:18.432 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:18.431+0000 7fb5ebbc3700 1 --2- 172.21.15.29:0/3374913584 >> [v2:172.21.15.29:6800/616654539,v1:172.21.15.29:6801/616654539] conn(0x7fb5cc02fd50 0x7fb5cc032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-21T11:41:18.432 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:18.431+0000 7fb5ebbc3700 1 -- 172.21.15.29:0/3374913584 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5e40703b0 msgr2=0x7fb5e410cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:41:18.432 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:18.431+0000 7fb5ebbc3700 1 --2- 172.21.15.29:0/3374913584 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5e40703b0 0x7fb5e410cc10 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7fb5e000b190 tx=0x7fb5e0006ee0).stop 2024-01-21T11:41:18.432 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:18.431+0000 7fb5ebbc3700 1 -- 172.21.15.29:0/3374913584 shutdown_connections 2024-01-21T11:41:18.432 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:18.431+0000 7fb5ebbc3700 1 --2- 172.21.15.29:0/3374913584 >> [v2:172.21.15.29:6800/616654539,v1:172.21.15.29:6801/616654539] conn(0x7fb5cc02fd50 0x7fb5cc032210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-21T11:41:18.432 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:18.431+0000 7fb5ebbc3700 1 --2- 172.21.15.29:0/3374913584 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5e40703b0 0x7fb5e410cc10 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:41:18.433 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:18.431+0000 7fb5ebbc3700 1 -- 172.21.15.29:0/3374913584 >> 172.21.15.29:0/3374913584 conn(0x7fb5e40f63c0 msgr2=0x7fb5e40f7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:41:18.433 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:18.432+0000 7fb5ebbc3700 1 -- 172.21.15.29:0/3374913584 shutdown_connections 2024-01-21T11:41:18.433 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:18.432+0000 7fb5ebbc3700 1 -- 172.21.15.29:0/3374913584 wait complete. 2024-01-21T11:41:18.722 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:18 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:41:17.494729+0000 mon.smithi029 (mon.0) 27 : audit [INF] from='client.? 172.21.15.29:0/3139669672' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-01-21T11:41:18.723 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:18 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:41:17.494785+0000 mon.smithi029 (mon.0) 28 : cluster [DBG] mgrmap e5: smithi029.blhxxk(active, since 5s) 2024-01-21T11:41:18.723 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:18 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:41:18.430722+0000 mon.smithi029 (mon.0) 29 : audit [DBG] from='client.? 172.21.15.29:0/3374913584' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-01-21T11:41:18.791 INFO:teuthology.orchestra.run.smithi029.stderr:Waiting for the mgr to restart... 2024-01-21T11:41:18.791 INFO:teuthology.orchestra.run.smithi029.stderr:Waiting for mgr epoch 5... 2024-01-21T11:41:18.791 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e4084074-b851-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp77cpgrhz:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpea05xzin:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2024-01-21T11:41:19.172 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:19.163+0000 7fd3f8915700 1 Processor -- start 2024-01-21T11:41:19.172 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:19.163+0000 7fd3f8915700 1 -- start start 2024-01-21T11:41:19.172 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:19.163+0000 7fd3f8915700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3f40703d0 0x7fd3f40fe790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T11:41:19.172 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:19.163+0000 7fd3f8915700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f40fec80 con 0x7fd3f40703d0 2024-01-21T11:41:19.172 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:19.164+0000 7fd3f2d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3f40703d0 0x7fd3f40fe790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:41:19.173 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:19.164+0000 7fd3f2d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3f40703d0 0x7fd3f40fe790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58232/0 (socket says 172.21.15.29:58232) 2024-01-21T11:41:19.173 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:19.164+0000 7fd3f2d9d700 1 -- 172.21.15.29:0/3240256824 learned_addr learned my addr 172.21.15.29:0/3240256824 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-21T11:41:19.173 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:19.164+0000 7fd3f2d9d700 1 -- 172.21.15.29:0/3240256824 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3f40fedc0 con 0x7fd3f40703d0 2024-01-21T11:41:19.173 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:19.164+0000 7fd3f2d9d700 1 --2- 172.21.15.29:0/3240256824 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3f40703d0 0x7fd3f40fe790 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7fd3e400b5f0 tx=0x7fd3e4006ee0).ready entity=mon.0 client_cookie=ff9da5480f29b8cb server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:41:19.173 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:19.164+0000 7fd3f1d9b700 1 -- 172.21.15.29:0/3240256824 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd3e400d600 con 0x7fd3f40703d0 2024-01-21T11:41:19.173 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:19.164+0000 7fd3f1d9b700 1 -- 172.21.15.29:0/3240256824 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fd3e400dbd0 con 0x7fd3f40703d0 2024-01-21T11:41:19.174 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:19.164+0000 7fd3f1d9b700 1 -- 172.21.15.29:0/3240256824 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd3e4011c00 con 0x7fd3f40703d0 2024-01-21T11:41:19.174 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:19.165+0000 7fd3f8915700 1 -- 172.21.15.29:0/3240256824 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3f40703d0 msgr2=0x7fd3f40fe790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:41:19.174 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:19.165+0000 7fd3f8915700 1 --2- 172.21.15.29:0/3240256824 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3f40703d0 0x7fd3f40fe790 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7fd3e400b5f0 tx=0x7fd3e4006ee0).stop 2024-01-21T11:41:19.174 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:19.165+0000 7fd3f8915700 1 -- 172.21.15.29:0/3240256824 shutdown_connections 2024-01-21T11:41:19.174 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:19.165+0000 7fd3f8915700 1 --2- 172.21.15.29:0/3240256824 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3f40703d0 0x7fd3f40fe790 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:41:19.174 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:19.165+0000 7fd3f8915700 1 -- 172.21.15.29:0/3240256824 >> 172.21.15.29:0/3240256824 conn(0x7fd3f40f6210 msgr2=0x7fd3f40f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:41:19.175 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:19.165+0000 7fd3f8915700 1 -- 172.21.15.29:0/3240256824 shutdown_connections 2024-01-21T11:41:19.175 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:19.165+0000 7fd3f8915700 1 -- 172.21.15.29:0/3240256824 wait complete. 2024-01-21T11:41:19.175 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:19.166+0000 7fd3f8915700 1 Processor -- start 2024-01-21T11:41:19.175 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:19.166+0000 7fd3f8915700 1 -- start start 2024-01-21T11:41:19.175 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:19.166+0000 7fd3f8915700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3f40703d0 0x7fd3f410a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:41:19.175 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:19.166+0000 7fd3f8915700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f410ad20 con 0x7fd3f40703d0 2024-01-21T11:41:19.175 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:19.166+0000 7fd3f2d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3f40703d0 0x7fd3f410a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:41:19.176 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:19.166+0000 7fd3f2d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3f40703d0 0x7fd3f410a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58236/0 (socket says 172.21.15.29:58236) 2024-01-21T11:41:19.176 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:19.166+0000 7fd3f2d9d700 1 -- 172.21.15.29:0/4256236216 learned_addr learned my addr 172.21.15.29:0/4256236216 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-21T11:41:19.176 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:19.167+0000 7fd3f2d9d700 1 -- 172.21.15.29:0/4256236216 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3e400b200 con 0x7fd3f40703d0 2024-01-21T11:41:19.176 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:19.167+0000 7fd3f2d9d700 1 --2- 172.21.15.29:0/4256236216 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3f40703d0 0x7fd3f410a830 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7fd3e400b190 tx=0x7fd3e4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:41:19.177 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:19.167+0000 7fd3db7fe700 1 -- 172.21.15.29:0/4256236216 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd3e401b410 con 0x7fd3f40703d0 2024-01-21T11:41:19.177 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:19.167+0000 7fd3db7fe700 1 -- 172.21.15.29:0/4256236216 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fd3e401b9e0 con 0x7fd3f40703d0 2024-01-21T11:41:19.177 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:19.167+0000 7fd3db7fe700 1 -- 172.21.15.29:0/4256236216 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd3e401a880 con 0x7fd3f40703d0 2024-01-21T11:41:19.177 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:19.167+0000 7fd3f8915700 1 -- 172.21.15.29:0/4256236216 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3f410af20 con 0x7fd3f40703d0 2024-01-21T11:41:19.177 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:19.167+0000 7fd3f8915700 1 -- 172.21.15.29:0/4256236216 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3f410b330 con 0x7fd3f40703d0 2024-01-21T11:41:19.177 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:19.168+0000 7fd3db7fe700 1 -- 172.21.15.29:0/4256236216 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fd3e400f070 con 0x7fd3f40703d0 2024-01-21T11:41:19.177 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:19.168+0000 7fd3db7fe700 1 --2- 172.21.15.29:0/4256236216 >> [v2:172.21.15.29:6800/616654539,v1:172.21.15.29:6801/616654539] conn(0x7fd3e002fdb0 0x7fd3e0032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:41:19.178 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:19.168+0000 7fd3db7fe700 1 -- 172.21.15.29:0/4256236216 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fd3e4041110 con 0x7fd3f40703d0 2024-01-21T11:41:19.178 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:19.168+0000 7fd3f259c700 1 -- 172.21.15.29:0/4256236216 >> [v2:172.21.15.29:6800/616654539,v1:172.21.15.29:6801/616654539] conn(0x7fd3e002fdb0 msgr2=0x7fd3e0032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/616654539 2024-01-21T11:41:19.178 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:19.168+0000 7fd3f259c700 1 --2- 172.21.15.29:0/4256236216 >> [v2:172.21.15.29:6800/616654539,v1:172.21.15.29:6801/616654539] conn(0x7fd3e002fdb0 0x7fd3e0032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-01-21T11:41:19.178 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:19.168+0000 7fd3f8915700 1 -- 172.21.15.29:0/4256236216 --> [v2:172.21.15.29:6800/616654539,v1:172.21.15.29:6801/616654539] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd3f4059100 con 0x7fd3e002fdb0 2024-01-21T11:41:19.369 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:19.369+0000 7fd3f259c700 1 -- 172.21.15.29:0/4256236216 >> [v2:172.21.15.29:6800/616654539,v1:172.21.15.29:6801/616654539] conn(0x7fd3e002fdb0 msgr2=0x7fd3e0032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/616654539 2024-01-21T11:41:19.369 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:19.369+0000 7fd3f259c700 1 --2- 172.21.15.29:0/4256236216 >> [v2:172.21.15.29:6800/616654539,v1:172.21.15.29:6801/616654539] conn(0x7fd3e002fdb0 0x7fd3e0032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-01-21T11:41:19.770 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:19.770+0000 7fd3f259c700 1 -- 172.21.15.29:0/4256236216 >> [v2:172.21.15.29:6800/616654539,v1:172.21.15.29:6801/616654539] conn(0x7fd3e002fdb0 msgr2=0x7fd3e0032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/616654539 2024-01-21T11:41:19.770 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:19.770+0000 7fd3f259c700 1 --2- 172.21.15.29:0/4256236216 >> [v2:172.21.15.29:6800/616654539,v1:172.21.15.29:6801/616654539] conn(0x7fd3e002fdb0 0x7fd3e0032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2024-01-21T11:41:20.571 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:20.571+0000 7fd3f259c700 1 -- 172.21.15.29:0/4256236216 >> [v2:172.21.15.29:6800/616654539,v1:172.21.15.29:6801/616654539] conn(0x7fd3e002fdb0 msgr2=0x7fd3e0032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/616654539 2024-01-21T11:41:20.571 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:20.571+0000 7fd3f259c700 1 --2- 172.21.15.29:0/4256236216 >> [v2:172.21.15.29:6800/616654539,v1:172.21.15.29:6801/616654539] conn(0x7fd3e002fdb0 0x7fd3e0032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2024-01-21T11:41:22.173 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:22.172+0000 7fd3f259c700 1 -- 172.21.15.29:0/4256236216 >> [v2:172.21.15.29:6800/616654539,v1:172.21.15.29:6801/616654539] conn(0x7fd3e002fdb0 msgr2=0x7fd3e0032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/616654539 2024-01-21T11:41:22.173 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:22.172+0000 7fd3f259c700 1 --2- 172.21.15.29:0/4256236216 >> [v2:172.21.15.29:6800/616654539,v1:172.21.15.29:6801/616654539] conn(0x7fd3e002fdb0 0x7fd3e0032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2024-01-21T11:41:22.725 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:22.725+0000 7fd3db7fe700 1 -- 172.21.15.29:0/4256236216 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mgrmap(e 6) v1 ==== 35970+0+0 (secure 0 0 0) 0x7fd3e40257e0 con 0x7fd3f40703d0 2024-01-21T11:41:22.725 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:22.725+0000 7fd3db7fe700 1 -- 172.21.15.29:0/4256236216 >> [v2:172.21.15.29:6800/616654539,v1:172.21.15.29:6801/616654539] conn(0x7fd3e002fdb0 msgr2=0x7fd3e0032270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-21T11:41:22.725 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:22.725+0000 7fd3db7fe700 1 --2- 172.21.15.29:0/4256236216 >> [v2:172.21.15.29:6800/616654539,v1:172.21.15.29:6801/616654539] conn(0x7fd3e002fdb0 0x7fd3e0032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-21T11:41:23.727 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:23.726+0000 7fd3db7fe700 1 -- 172.21.15.29:0/4256236216 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 7) v1 ==== 36040+0+0 (secure 0 0 0) 0x7fd3e4042850 con 0x7fd3f40703d0 2024-01-21T11:41:23.727 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:23.726+0000 7fd3db7fe700 1 --2- 172.21.15.29:0/4256236216 >> [v2:172.21.15.29:6800/1563185523,v1:172.21.15.29:6801/1563185523] conn(0x7fd3e00336d0 0x7fd3e0035ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:41:23.727 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:23.727+0000 7fd3db7fe700 1 -- 172.21.15.29:0/4256236216 --> [v2:172.21.15.29:6800/1563185523,v1:172.21.15.29:6801/1563185523] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd3f4059100 con 0x7fd3e00336d0 2024-01-21T11:41:23.728 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:23.727+0000 7fd3f259c700 1 --2- 172.21.15.29:0/4256236216 >> [v2:172.21.15.29:6800/1563185523,v1:172.21.15.29:6801/1563185523] conn(0x7fd3e00336d0 0x7fd3e0035ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:41:23.728 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:23.727+0000 7fd3f259c700 1 --2- 172.21.15.29:0/4256236216 >> [v2:172.21.15.29:6800/1563185523,v1:172.21.15.29:6801/1563185523] conn(0x7fd3e00336d0 0x7fd3e0035ac0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fd3e8006fd0 tx=0x7fd3e8006d70).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:41:23.728 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:23.728+0000 7fd3db7fe700 1 -- 172.21.15.29:0/4256236216 <== mgr.14116 v2:172.21.15.29:6800/1563185523 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7fd3f4059100 con 0x7fd3e00336d0 2024-01-21T11:41:23.735 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:23.735+0000 7fd3f8915700 1 -- 172.21.15.29:0/4256236216 --> [v2:172.21.15.29:6800/1563185523,v1:172.21.15.29:6801/1563185523] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fd3f4045ad0 con 0x7fd3e00336d0 2024-01-21T11:41:23.735 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:23.735+0000 7fd3db7fe700 1 -- 172.21.15.29:0/4256236216 <== mgr.14116 v2:172.21.15.29:6800/1563185523 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fd3f4045ad0 con 0x7fd3e00336d0 2024-01-21T11:41:23.736 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: { 2024-01-21T11:41:23.736 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2024-01-21T11:41:23.736 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "initialized": true 2024-01-21T11:41:23.736 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: } 2024-01-21T11:41:23.737 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:23.736+0000 7fd3f8915700 1 -- 172.21.15.29:0/4256236216 >> [v2:172.21.15.29:6800/1563185523,v1:172.21.15.29:6801/1563185523] conn(0x7fd3e00336d0 msgr2=0x7fd3e0035ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:41:23.737 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:23.736+0000 7fd3f8915700 1 --2- 172.21.15.29:0/4256236216 >> [v2:172.21.15.29:6800/1563185523,v1:172.21.15.29:6801/1563185523] conn(0x7fd3e00336d0 0x7fd3e0035ac0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fd3e8006fd0 tx=0x7fd3e8006d70).stop 2024-01-21T11:41:23.737 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:23.736+0000 7fd3f8915700 1 -- 172.21.15.29:0/4256236216 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3f40703d0 msgr2=0x7fd3f410a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:41:23.738 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:23.736+0000 7fd3f8915700 1 --2- 172.21.15.29:0/4256236216 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3f40703d0 0x7fd3f410a830 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7fd3e400b190 tx=0x7fd3e4006ee0).stop 2024-01-21T11:41:23.738 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:23.736+0000 7fd3f8915700 1 -- 172.21.15.29:0/4256236216 shutdown_connections 2024-01-21T11:41:23.738 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:23.736+0000 7fd3f8915700 1 --2- 172.21.15.29:0/4256236216 >> [v2:172.21.15.29:6800/1563185523,v1:172.21.15.29:6801/1563185523] conn(0x7fd3e00336d0 0x7fd3e0035ac0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:41:23.738 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:23.736+0000 7fd3f8915700 1 --2- 172.21.15.29:0/4256236216 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3f40703d0 0x7fd3f410a830 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:41:23.738 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:23.736+0000 7fd3f8915700 1 --2- 172.21.15.29:0/4256236216 >> [v2:172.21.15.29:6800/616654539,v1:172.21.15.29:6801/616654539] conn(0x7fd3e002fdb0 0x7fd3e0032270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-21T11:41:23.738 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:23.736+0000 7fd3f8915700 1 -- 172.21.15.29:0/4256236216 >> 172.21.15.29:0/4256236216 conn(0x7fd3f40f6210 msgr2=0x7fd3f40f7cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:41:23.738 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:23.736+0000 7fd3f8915700 1 -- 172.21.15.29:0/4256236216 shutdown_connections 2024-01-21T11:41:23.738 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:23.736+0000 7fd3f8915700 1 -- 172.21.15.29:0/4256236216 wait complete. 2024-01-21T11:41:24.169 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:23 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:41:22.673762+0000 mon.smithi029 (mon.0) 30 : cluster [INF] Active manager daemon smithi029.blhxxk restarted 2024-01-21T11:41:24.169 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:23 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:41:22.674148+0000 mon.smithi029 (mon.0) 31 : cluster [INF] Activating manager daemon smithi029.blhxxk 2024-01-21T11:41:24.169 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:23 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:41:22.675875+0000 mon.smithi029 (mon.0) 32 : cluster [DBG] osdmap e2: 0 total, 0 up, 0 in 2024-01-21T11:41:24.169 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:23 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:41:22.726708+0000 mon.smithi029 (mon.0) 33 : cluster [DBG] mgrmap e6: smithi029.blhxxk(active, starting, since 0.0526285s) 2024-01-21T11:41:24.169 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:23 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:41:22.727768+0000 mon.smithi029 (mon.0) 34 : audit [DBG] from='mgr.14116 172.21.15.29:0/1084824939' entity='mgr.smithi029.blhxxk' cmd=[{"prefix": "mon metadata", "id": "smithi029"}]: dispatch 2024-01-21T11:41:24.169 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:23 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:41:22.728321+0000 mon.smithi029 (mon.0) 35 : audit [DBG] from='mgr.14116 172.21.15.29:0/1084824939' entity='mgr.smithi029.blhxxk' cmd=[{"prefix": "mgr metadata", "who": "smithi029.blhxxk", "id": "smithi029.blhxxk"}]: dispatch 2024-01-21T11:41:24.169 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:23 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:41:22.729017+0000 mon.smithi029 (mon.0) 36 : audit [DBG] from='mgr.14116 172.21.15.29:0/1084824939' entity='mgr.smithi029.blhxxk' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-21T11:41:24.169 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:23 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:41:22.729235+0000 mon.smithi029 (mon.0) 37 : audit [DBG] from='mgr.14116 172.21.15.29:0/1084824939' entity='mgr.smithi029.blhxxk' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-21T11:41:24.169 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:23 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:41:22.729439+0000 mon.smithi029 (mon.0) 38 : audit [DBG] from='mgr.14116 172.21.15.29:0/1084824939' entity='mgr.smithi029.blhxxk' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-21T11:41:24.170 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:23 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:41:22.743437+0000 mon.smithi029 (mon.0) 39 : cluster [INF] Manager daemon smithi029.blhxxk is now available 2024-01-21T11:41:24.170 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:23 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:41:22.756929+0000 mon.smithi029 (mon.0) 40 : audit [INF] from='mgr.14116 172.21.15.29:0/1084824939' entity='mgr.smithi029.blhxxk' 2024-01-21T11:41:24.170 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:23 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:41:22.759614+0000 mon.smithi029 (mon.0) 41 : audit [INF] from='mgr.14116 172.21.15.29:0/1084824939' entity='mgr.smithi029.blhxxk' 2024-01-21T11:41:24.170 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:23 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:41:22.762130+0000 mon.smithi029 (mon.0) 42 : audit [INF] from='mgr.14116 172.21.15.29:0/1084824939' entity='mgr.smithi029.blhxxk' 2024-01-21T11:41:24.170 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:23 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:41:22.856178+0000 mon.smithi029 (mon.0) 43 : audit [INF] from='mgr.14116 172.21.15.29:0/1084824939' entity='mgr.smithi029.blhxxk' 2024-01-21T11:41:24.170 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:23 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:41:22.856878+0000 mon.smithi029 (mon.0) 44 : audit [DBG] from='mgr.14116 172.21.15.29:0/1084824939' entity='mgr.smithi029.blhxxk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-21T11:41:24.170 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:23 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:41:22.858153+0000 mon.smithi029 (mon.0) 45 : audit [DBG] from='mgr.14116 172.21.15.29:0/1084824939' entity='mgr.smithi029.blhxxk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-21T11:41:24.170 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:23 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:41:22.859957+0000 mon.smithi029 (mon.0) 46 : audit [DBG] from='mgr.14116 172.21.15.29:0/1084824939' entity='mgr.smithi029.blhxxk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-21T11:41:24.170 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:23 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:41:22.870787+0000 mon.smithi029 (mon.0) 47 : audit [INF] from='mgr.14116 172.21.15.29:0/1084824939' entity='mgr.smithi029.blhxxk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi029.blhxxk/mirror_snapshot_schedule"}]: dispatch 2024-01-21T11:41:24.170 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:23 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:41:22.872019+0000 mon.smithi029 (mon.0) 48 : audit [INF] from='mgr.14116 172.21.15.29:0/1084824939' entity='mgr.smithi029.blhxxk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi029.blhxxk/trash_purge_schedule"}]: dispatch 2024-01-21T11:41:25.168 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:24 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:41:23.729075+0000 mgr.smithi029.blhxxk (mgr.14116) 1 : audit [DBG] from='client.14120 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-01-21T11:41:25.168 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:24 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:41:23.729855+0000 mon.smithi029 (mon.0) 49 : cluster [DBG] mgrmap e7: smithi029.blhxxk(active, since 1.05577s) 2024-01-21T11:41:25.169 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:24 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:41:23.736394+0000 mgr.smithi029.blhxxk (mgr.14116) 2 : audit [DBG] from='client.14120 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-01-21T11:41:26.168 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:25 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:41:24.731346+0000 mon.smithi029 (mon.0) 50 : cluster [DBG] mgrmap e8: smithi029.blhxxk(active, since 2s) 2024-01-21T11:41:26.979 INFO:teuthology.orchestra.run.smithi029.stderr:mgr epoch 5 is available 2024-01-21T11:41:26.979 INFO:teuthology.orchestra.run.smithi029.stderr:Setting orchestrator backend to cephadm... 2024-01-21T11:41:26.979 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e4084074-b851-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp77cpgrhz:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpea05xzin:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch set backend cephadm 2024-01-21T11:41:28.719 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:28.484+0000 7ff01bd20700 1 Processor -- start 2024-01-21T11:41:28.719 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:28.485+0000 7ff01bd20700 1 -- start start 2024-01-21T11:41:28.719 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:28.485+0000 7ff01bd20700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0140fbcb0 0x7ff0140fc0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T11:41:28.719 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:28.485+0000 7ff01bd20700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0140fc5c0 con 0x7ff0140fbcb0 2024-01-21T11:41:28.719 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:28.485+0000 7ff019abc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0140fbcb0 0x7ff0140fc0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:41:28.719 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:28.485+0000 7ff019abc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0140fbcb0 0x7ff0140fc0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39866/0 (socket says 172.21.15.29:39866) 2024-01-21T11:41:28.719 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:28.485+0000 7ff019abc700 1 -- 172.21.15.29:0/219700045 learned_addr learned my addr 172.21.15.29:0/219700045 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-21T11:41:28.720 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:28.485+0000 7ff019abc700 1 -- 172.21.15.29:0/219700045 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0140fc700 con 0x7ff0140fbcb0 2024-01-21T11:41:28.720 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:28.485+0000 7ff019abc700 1 --2- 172.21.15.29:0/219700045 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0140fbcb0 0x7ff0140fc0d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7ff01000b5f0 tx=0x7ff010006ee0).ready entity=mon.0 client_cookie=c6bb71fa5d853c9a server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:41:28.720 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:28.485+0000 7ff018aba700 1 -- 172.21.15.29:0/219700045 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff01000d600 con 0x7ff0140fbcb0 2024-01-21T11:41:28.720 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:28.486+0000 7ff018aba700 1 -- 172.21.15.29:0/219700045 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7ff01000dbd0 con 0x7ff0140fbcb0 2024-01-21T11:41:28.720 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:28.486+0000 7ff018aba700 1 -- 172.21.15.29:0/219700045 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff010011c00 con 0x7ff0140fbcb0 2024-01-21T11:41:28.720 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:28.486+0000 7ff01bd20700 1 -- 172.21.15.29:0/219700045 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0140fbcb0 msgr2=0x7ff0140fc0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:41:28.720 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:28.486+0000 7ff01bd20700 1 --2- 172.21.15.29:0/219700045 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0140fbcb0 0x7ff0140fc0d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7ff01000b5f0 tx=0x7ff010006ee0).stop 2024-01-21T11:41:28.720 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:28.486+0000 7ff01bd20700 1 -- 172.21.15.29:0/219700045 shutdown_connections 2024-01-21T11:41:28.720 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:28.486+0000 7ff01bd20700 1 --2- 172.21.15.29:0/219700045 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0140fbcb0 0x7ff0140fc0d0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:41:28.720 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:28.486+0000 7ff01bd20700 1 -- 172.21.15.29:0/219700045 >> 172.21.15.29:0/219700045 conn(0x7ff0140f74b0 msgr2=0x7ff0140f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:41:28.720 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:28.486+0000 7ff01bd20700 1 -- 172.21.15.29:0/219700045 shutdown_connections 2024-01-21T11:41:28.720 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:28.486+0000 7ff01bd20700 1 -- 172.21.15.29:0/219700045 wait complete. 2024-01-21T11:41:28.720 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:28.487+0000 7ff01bd20700 1 Processor -- start 2024-01-21T11:41:28.720 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:28.487+0000 7ff01bd20700 1 -- start start 2024-01-21T11:41:28.721 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:28.487+0000 7ff01bd20700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0140fbcb0 0x7ff0140725e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:41:28.721 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:28.487+0000 7ff01bd20700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff014070a60 con 0x7ff0140fbcb0 2024-01-21T11:41:28.721 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:28.487+0000 7ff019abc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0140fbcb0 0x7ff0140725e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:41:28.721 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:28.487+0000 7ff019abc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0140fbcb0 0x7ff0140725e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39870/0 (socket says 172.21.15.29:39870) 2024-01-21T11:41:28.721 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:28.487+0000 7ff019abc700 1 -- 172.21.15.29:0/3816674706 learned_addr learned my addr 172.21.15.29:0/3816674706 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-21T11:41:28.721 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:28.487+0000 7ff019abc700 1 -- 172.21.15.29:0/3816674706 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff01000b200 con 0x7ff0140fbcb0 2024-01-21T11:41:28.721 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:28.488+0000 7ff019abc700 1 --2- 172.21.15.29:0/3816674706 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0140fbcb0 0x7ff0140725e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7ff01000e040 tx=0x7ff010006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:41:28.721 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:28.488+0000 7ff0067fc700 1 -- 172.21.15.29:0/3816674706 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff01001b410 con 0x7ff0140fbcb0 2024-01-21T11:41:28.721 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:28.488+0000 7ff0067fc700 1 -- 172.21.15.29:0/3816674706 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7ff01001b9e0 con 0x7ff0140fbcb0 2024-01-21T11:41:28.721 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:28.488+0000 7ff01bd20700 1 -- 172.21.15.29:0/3816674706 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff014070c00 con 0x7ff0140fbcb0 2024-01-21T11:41:28.721 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:28.488+0000 7ff0067fc700 1 -- 172.21.15.29:0/3816674706 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff01001a820 con 0x7ff0140fbcb0 2024-01-21T11:41:28.721 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:28.488+0000 7ff01bd20700 1 -- 172.21.15.29:0/3816674706 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff014071010 con 0x7ff0140fbcb0 2024-01-21T11:41:28.721 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:28.488+0000 7ff0067fc700 1 -- 172.21.15.29:0/3816674706 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7ff01000f070 con 0x7ff0140fbcb0 2024-01-21T11:41:28.721 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:28.489+0000 7ff0067fc700 1 --2- 172.21.15.29:0/3816674706 >> [v2:172.21.15.29:6800/1563185523,v1:172.21.15.29:6801/1563185523] conn(0x7feffc02fdb0 0x7feffc032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:41:28.721 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:28.489+0000 7ff0067fc700 1 -- 172.21.15.29:0/3816674706 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7ff0100413f0 con 0x7ff0140fbcb0 2024-01-21T11:41:28.722 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:28.489+0000 7ff0192bb700 1 --2- 172.21.15.29:0/3816674706 >> [v2:172.21.15.29:6800/1563185523,v1:172.21.15.29:6801/1563185523] conn(0x7feffc02fdb0 0x7feffc032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:41:28.722 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:28.489+0000 7ff01bd20700 1 -- 172.21.15.29:0/3816674706 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff014059100 con 0x7ff0140fbcb0 2024-01-21T11:41:28.722 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:28.489+0000 7ff0192bb700 1 --2- 172.21.15.29:0/3816674706 >> [v2:172.21.15.29:6800/1563185523,v1:172.21.15.29:6801/1563185523] conn(0x7feffc02fdb0 0x7feffc032270 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7ff0080096a0 tx=0x7ff008006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:41:28.722 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:28.494+0000 7ff0067fc700 1 -- 172.21.15.29:0/3816674706 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff010016070 con 0x7ff0140fbcb0 2024-01-21T11:41:28.722 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:28.682+0000 7ff01bd20700 1 -- 172.21.15.29:0/3816674706 --> [v2:172.21.15.29:6800/1563185523,v1:172.21.15.29:6801/1563185523] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7ff0140fda30 con 0x7feffc02fdb0 2024-01-21T11:41:28.722 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:28.687+0000 7ff0067fc700 1 -- 172.21.15.29:0/3816674706 <== mgr.14116 v2:172.21.15.29:6800/1563185523 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7ff0140fda30 con 0x7feffc02fdb0 2024-01-21T11:41:28.722 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:28.689+0000 7ff01bd20700 1 -- 172.21.15.29:0/3816674706 >> [v2:172.21.15.29:6800/1563185523,v1:172.21.15.29:6801/1563185523] conn(0x7feffc02fdb0 msgr2=0x7feffc032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:41:28.722 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:28.689+0000 7ff01bd20700 1 --2- 172.21.15.29:0/3816674706 >> [v2:172.21.15.29:6800/1563185523,v1:172.21.15.29:6801/1563185523] conn(0x7feffc02fdb0 0x7feffc032270 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7ff0080096a0 tx=0x7ff008006b40).stop 2024-01-21T11:41:28.722 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:28.689+0000 7ff01bd20700 1 -- 172.21.15.29:0/3816674706 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0140fbcb0 msgr2=0x7ff0140725e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:41:28.722 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:28.689+0000 7ff01bd20700 1 --2- 172.21.15.29:0/3816674706 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0140fbcb0 0x7ff0140725e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7ff01000e040 tx=0x7ff010006ee0).stop 2024-01-21T11:41:28.722 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:28.689+0000 7ff01bd20700 1 -- 172.21.15.29:0/3816674706 shutdown_connections 2024-01-21T11:41:28.722 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:28.689+0000 7ff01bd20700 1 --2- 172.21.15.29:0/3816674706 >> [v2:172.21.15.29:6800/1563185523,v1:172.21.15.29:6801/1563185523] conn(0x7feffc02fdb0 0x7feffc032270 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:41:28.722 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:28.689+0000 7ff01bd20700 1 --2- 172.21.15.29:0/3816674706 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0140fbcb0 0x7ff0140725e0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:41:28.722 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:28.689+0000 7ff01bd20700 1 -- 172.21.15.29:0/3816674706 >> 172.21.15.29:0/3816674706 conn(0x7ff0140f74b0 msgr2=0x7ff0140f8fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:41:28.722 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:28.689+0000 7ff01bd20700 1 -- 172.21.15.29:0/3816674706 shutdown_connections 2024-01-21T11:41:28.722 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:28.689+0000 7ff01bd20700 1 -- 172.21.15.29:0/3816674706 wait complete. 2024-01-21T11:41:30.168 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:29 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:41:28.684495+0000 mgr.smithi029.blhxxk (mgr.14116) 3 : audit [DBG] from='client.14126 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T11:41:30.169 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:29 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:41:28.688389+0000 mon.smithi029 (mon.0) 51 : audit [INF] from='mgr.14116 172.21.15.29:0/1084824939' entity='mgr.smithi029.blhxxk' 2024-01-21T11:41:30.169 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:29 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:41:28.691983+0000 mon.smithi029 (mon.0) 52 : audit [DBG] from='mgr.14116 172.21.15.29:0/1084824939' entity='mgr.smithi029.blhxxk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-21T11:41:31.970 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e4084074-b851-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp77cpgrhz:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpea05xzin:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm set-user root 2024-01-21T11:41:33.254 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:33.173+0000 7f3192b0f700 1 Processor -- start 2024-01-21T11:41:33.254 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:33.173+0000 7f3192b0f700 1 -- start start 2024-01-21T11:41:33.254 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:33.173+0000 7f3192b0f700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f318c0fa9d0 0x7f318c0fadf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T11:41:33.254 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:33.173+0000 7f3192b0f700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f318c0fb2e0 con 0x7f318c0fa9d0 2024-01-21T11:41:33.254 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:33.173+0000 7f3191b0d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f318c0fa9d0 0x7f318c0fadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:41:33.254 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:33.173+0000 7f3191b0d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f318c0fa9d0 0x7f318c0fadf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46586/0 (socket says 172.21.15.29:46586) 2024-01-21T11:41:33.254 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:33.173+0000 7f3191b0d700 1 -- 172.21.15.29:0/984026332 learned_addr learned my addr 172.21.15.29:0/984026332 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-21T11:41:33.254 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:33.174+0000 7f3191b0d700 1 -- 172.21.15.29:0/984026332 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f318c0fb420 con 0x7f318c0fa9d0 2024-01-21T11:41:33.255 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:33.174+0000 7f3191b0d700 1 --2- 172.21.15.29:0/984026332 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f318c0fa9d0 0x7f318c0fadf0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f318800b5f0 tx=0x7f3188006ee0).ready entity=mon.0 client_cookie=d05748b963755851 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:41:33.255 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:33.174+0000 7f3190b0b700 1 -- 172.21.15.29:0/984026332 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f318800d600 con 0x7f318c0fa9d0 2024-01-21T11:41:33.255 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:33.174+0000 7f3190b0b700 1 -- 172.21.15.29:0/984026332 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f318800dbd0 con 0x7f318c0fa9d0 2024-01-21T11:41:33.255 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:33.174+0000 7f3190b0b700 1 -- 172.21.15.29:0/984026332 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3188011c00 con 0x7f318c0fa9d0 2024-01-21T11:41:33.255 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:33.174+0000 7f3192b0f700 1 -- 172.21.15.29:0/984026332 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f318c0fa9d0 msgr2=0x7f318c0fadf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:41:33.255 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:33.174+0000 7f3192b0f700 1 --2- 172.21.15.29:0/984026332 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f318c0fa9d0 0x7f318c0fadf0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f318800b5f0 tx=0x7f3188006ee0).stop 2024-01-21T11:41:33.255 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:33.174+0000 7f3192b0f700 1 -- 172.21.15.29:0/984026332 shutdown_connections 2024-01-21T11:41:33.255 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:33.174+0000 7f3192b0f700 1 --2- 172.21.15.29:0/984026332 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f318c0fa9d0 0x7f318c0fadf0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:41:33.255 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:33.174+0000 7f3192b0f700 1 -- 172.21.15.29:0/984026332 >> 172.21.15.29:0/984026332 conn(0x7f318c0f6210 msgr2=0x7f318c0f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:41:33.255 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:33.175+0000 7f3192b0f700 1 -- 172.21.15.29:0/984026332 shutdown_connections 2024-01-21T11:41:33.255 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:33.175+0000 7f3192b0f700 1 -- 172.21.15.29:0/984026332 wait complete. 2024-01-21T11:41:33.255 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:33.175+0000 7f3192b0f700 1 Processor -- start 2024-01-21T11:41:33.256 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:33.175+0000 7f3192b0f700 1 -- start start 2024-01-21T11:41:33.256 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:33.175+0000 7f3192b0f700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f318c0fa9d0 0x7f318c10a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:41:33.256 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:33.175+0000 7f3192b0f700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f318c10ad70 con 0x7f318c0fa9d0 2024-01-21T11:41:33.256 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:33.175+0000 7f3191b0d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f318c0fa9d0 0x7f318c10a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:41:33.256 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:33.175+0000 7f3191b0d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f318c0fa9d0 0x7f318c10a880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46598/0 (socket says 172.21.15.29:46598) 2024-01-21T11:41:33.256 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:33.175+0000 7f3191b0d700 1 -- 172.21.15.29:0/178678190 learned_addr learned my addr 172.21.15.29:0/178678190 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-21T11:41:33.256 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:33.176+0000 7f3191b0d700 1 -- 172.21.15.29:0/178678190 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f318800b200 con 0x7f318c0fa9d0 2024-01-21T11:41:33.256 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:33.176+0000 7f3191b0d700 1 --2- 172.21.15.29:0/178678190 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f318c0fa9d0 0x7f318c10a880 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f318800e040 tx=0x7f3188006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:41:33.256 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:33.176+0000 7f317a7fc700 1 -- 172.21.15.29:0/178678190 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f318801b410 con 0x7f318c0fa9d0 2024-01-21T11:41:33.256 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:33.176+0000 7f317a7fc700 1 -- 172.21.15.29:0/178678190 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f318801b9e0 con 0x7f318c0fa9d0 2024-01-21T11:41:33.256 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:33.176+0000 7f3192b0f700 1 -- 172.21.15.29:0/178678190 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f318c10af70 con 0x7f318c0fa9d0 2024-01-21T11:41:33.256 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:33.176+0000 7f317a7fc700 1 -- 172.21.15.29:0/178678190 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f318801a820 con 0x7f318c0fa9d0 2024-01-21T11:41:33.257 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:33.176+0000 7f3192b0f700 1 -- 172.21.15.29:0/178678190 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f318c10b380 con 0x7f318c0fa9d0 2024-01-21T11:41:33.257 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:33.177+0000 7f317a7fc700 1 -- 172.21.15.29:0/178678190 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f318800f070 con 0x7f318c0fa9d0 2024-01-21T11:41:33.257 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:33.177+0000 7f317a7fc700 1 --2- 172.21.15.29:0/178678190 >> [v2:172.21.15.29:6800/1563185523,v1:172.21.15.29:6801/1563185523] conn(0x7f317402fdb0 0x7f3174032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:41:33.257 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:33.177+0000 7f317a7fc700 1 -- 172.21.15.29:0/178678190 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f31880414c0 con 0x7f318c0fa9d0 2024-01-21T11:41:33.257 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:33.177+0000 7f319130c700 1 --2- 172.21.15.29:0/178678190 >> [v2:172.21.15.29:6800/1563185523,v1:172.21.15.29:6801/1563185523] conn(0x7f317402fdb0 0x7f3174032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:41:33.257 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:33.177+0000 7f3192b0f700 1 -- 172.21.15.29:0/178678190 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f318c059100 con 0x7f318c0fa9d0 2024-01-21T11:41:33.257 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:33.178+0000 7f319130c700 1 --2- 172.21.15.29:0/178678190 >> [v2:172.21.15.29:6800/1563185523,v1:172.21.15.29:6801/1563185523] conn(0x7f317402fdb0 0x7f3174032270 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f317c0096a0 tx=0x7f317c006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:41:33.257 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:33.182+0000 7f317a7fc700 1 -- 172.21.15.29:0/178678190 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3188016070 con 0x7f318c0fa9d0 2024-01-21T11:41:33.344 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:33.344+0000 7f3192b0f700 1 -- 172.21.15.29:0/178678190 --> [v2:172.21.15.29:6800/1563185523,v1:172.21.15.29:6801/1563185523] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f318c104820 con 0x7f317402fdb0 2024-01-21T11:41:33.345 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:33.344+0000 7f317a7fc700 1 -- 172.21.15.29:0/178678190 <== mgr.14116 v2:172.21.15.29:6800/1563185523 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f318c104820 con 0x7f317402fdb0 2024-01-21T11:41:33.345 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: value unchanged 2024-01-21T11:41:33.347 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:33.346+0000 7f3192b0f700 1 -- 172.21.15.29:0/178678190 >> [v2:172.21.15.29:6800/1563185523,v1:172.21.15.29:6801/1563185523] conn(0x7f317402fdb0 msgr2=0x7f3174032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:41:33.347 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:33.346+0000 7f3192b0f700 1 --2- 172.21.15.29:0/178678190 >> [v2:172.21.15.29:6800/1563185523,v1:172.21.15.29:6801/1563185523] conn(0x7f317402fdb0 0x7f3174032270 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f317c0096a0 tx=0x7f317c006b40).stop 2024-01-21T11:41:33.347 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:33.346+0000 7f3192b0f700 1 -- 172.21.15.29:0/178678190 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f318c0fa9d0 msgr2=0x7f318c10a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:41:33.347 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:33.346+0000 7f3192b0f700 1 --2- 172.21.15.29:0/178678190 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f318c0fa9d0 0x7f318c10a880 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f318800e040 tx=0x7f3188006ee0).stop 2024-01-21T11:41:33.347 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:33.346+0000 7f3192b0f700 1 -- 172.21.15.29:0/178678190 shutdown_connections 2024-01-21T11:41:33.347 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:33.346+0000 7f3192b0f700 1 --2- 172.21.15.29:0/178678190 >> [v2:172.21.15.29:6800/1563185523,v1:172.21.15.29:6801/1563185523] conn(0x7f317402fdb0 0x7f3174032270 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:41:33.347 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:33.346+0000 7f3192b0f700 1 --2- 172.21.15.29:0/178678190 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f318c0fa9d0 0x7f318c10a880 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:41:33.348 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:33.346+0000 7f3192b0f700 1 -- 172.21.15.29:0/178678190 >> 172.21.15.29:0/178678190 conn(0x7f318c0f6210 msgr2=0x7f318c070050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:41:33.348 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:33.346+0000 7f3192b0f700 1 -- 172.21.15.29:0/178678190 shutdown_connections 2024-01-21T11:41:33.348 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:33.346+0000 7f3192b0f700 1 -- 172.21.15.29:0/178678190 wait complete. 2024-01-21T11:41:33.668 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:33 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:41:33.345543+0000 mgr.smithi029.blhxxk (mgr.14116) 4 : audit [DBG] from='client.14128 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T11:41:35.852 INFO:teuthology.orchestra.run.smithi029.stderr:Generating ssh key... 2024-01-21T11:41:35.852 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e4084074-b851-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp77cpgrhz:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpea05xzin:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm generate-key 2024-01-21T11:41:37.130 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:37.129+0000 7f55d937d700 1 Processor -- start 2024-01-21T11:41:37.130 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:37.129+0000 7f55d937d700 1 -- start start 2024-01-21T11:41:37.130 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:37.129+0000 7f55d937d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55d40fbcb0 0x7f55d40fc0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T11:41:37.130 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:37.129+0000 7f55d937d700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55d40fc5c0 con 0x7f55d40fbcb0 2024-01-21T11:41:37.130 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:37.129+0000 7f55d2ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55d40fbcb0 0x7f55d40fc0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:41:37.130 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:37.129+0000 7f55d2ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55d40fbcb0 0x7f55d40fc0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46612/0 (socket says 172.21.15.29:46612) 2024-01-21T11:41:37.130 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:37.129+0000 7f55d2ffd700 1 -- 172.21.15.29:0/2962826512 learned_addr learned my addr 172.21.15.29:0/2962826512 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-21T11:41:37.132 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:37.130+0000 7f55d2ffd700 1 -- 172.21.15.29:0/2962826512 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55d40fc700 con 0x7f55d40fbcb0 2024-01-21T11:41:37.132 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:37.130+0000 7f55d2ffd700 1 --2- 172.21.15.29:0/2962826512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55d40fbcb0 0x7f55d40fc0d0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f55c400b5f0 tx=0x7f55c4006ee0).ready entity=mon.0 client_cookie=e2ed8014a1353e55 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:41:37.132 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:37.130+0000 7f55d1ffb700 1 -- 172.21.15.29:0/2962826512 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f55c400d600 con 0x7f55d40fbcb0 2024-01-21T11:41:37.132 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:37.130+0000 7f55d1ffb700 1 -- 172.21.15.29:0/2962826512 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f55c400dbd0 con 0x7f55d40fbcb0 2024-01-21T11:41:37.132 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:37.130+0000 7f55d1ffb700 1 -- 172.21.15.29:0/2962826512 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f55c4011c00 con 0x7f55d40fbcb0 2024-01-21T11:41:37.132 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:37.131+0000 7f55d937d700 1 -- 172.21.15.29:0/2962826512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55d40fbcb0 msgr2=0x7f55d40fc0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:41:37.132 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:37.131+0000 7f55d937d700 1 --2- 172.21.15.29:0/2962826512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55d40fbcb0 0x7f55d40fc0d0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f55c400b5f0 tx=0x7f55c4006ee0).stop 2024-01-21T11:41:37.132 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:37.131+0000 7f55d937d700 1 -- 172.21.15.29:0/2962826512 shutdown_connections 2024-01-21T11:41:37.132 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:37.131+0000 7f55d937d700 1 --2- 172.21.15.29:0/2962826512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55d40fbcb0 0x7f55d40fc0d0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:41:37.132 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:37.131+0000 7f55d937d700 1 -- 172.21.15.29:0/2962826512 >> 172.21.15.29:0/2962826512 conn(0x7f55d40f74b0 msgr2=0x7f55d40f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:41:37.132 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:37.131+0000 7f55d937d700 1 -- 172.21.15.29:0/2962826512 shutdown_connections 2024-01-21T11:41:37.132 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:37.131+0000 7f55d937d700 1 -- 172.21.15.29:0/2962826512 wait complete. 2024-01-21T11:41:37.132 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:37.132+0000 7f55d937d700 1 Processor -- start 2024-01-21T11:41:37.135 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:37.132+0000 7f55d937d700 1 -- start start 2024-01-21T11:41:37.135 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:37.132+0000 7f55d937d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55d40fbcb0 0x7f55d40725e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:41:37.135 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:37.132+0000 7f55d937d700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55d4070a60 con 0x7f55d40fbcb0 2024-01-21T11:41:37.135 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:37.132+0000 7f55d2ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55d40fbcb0 0x7f55d40725e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:41:37.135 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:37.132+0000 7f55d2ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55d40fbcb0 0x7f55d40725e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46614/0 (socket says 172.21.15.29:46614) 2024-01-21T11:41:37.135 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:37.132+0000 7f55d2ffd700 1 -- 172.21.15.29:0/1924334192 learned_addr learned my addr 172.21.15.29:0/1924334192 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-21T11:41:37.135 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:37.133+0000 7f55d2ffd700 1 -- 172.21.15.29:0/1924334192 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55c400b200 con 0x7f55d40fbcb0 2024-01-21T11:41:37.135 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:37.133+0000 7f55d2ffd700 1 --2- 172.21.15.29:0/1924334192 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55d40fbcb0 0x7f55d40725e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f55c400e040 tx=0x7f55c4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:41:37.135 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:37.133+0000 7f55bb7fe700 1 -- 172.21.15.29:0/1924334192 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f55c401b410 con 0x7f55d40fbcb0 2024-01-21T11:41:37.135 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:37.133+0000 7f55bb7fe700 1 -- 172.21.15.29:0/1924334192 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f55c401b9e0 con 0x7f55d40fbcb0 2024-01-21T11:41:37.135 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:37.133+0000 7f55bb7fe700 1 -- 172.21.15.29:0/1924334192 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f55c401a820 con 0x7f55d40fbcb0 2024-01-21T11:41:37.135 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:37.133+0000 7f55d937d700 1 -- 172.21.15.29:0/1924334192 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55d4070c00 con 0x7f55d40fbcb0 2024-01-21T11:41:37.135 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:37.133+0000 7f55d937d700 1 -- 172.21.15.29:0/1924334192 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55d4071010 con 0x7f55d40fbcb0 2024-01-21T11:41:37.135 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:37.134+0000 7f55bb7fe700 1 -- 172.21.15.29:0/1924334192 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f55c400f070 con 0x7f55d40fbcb0 2024-01-21T11:41:37.136 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:37.134+0000 7f55bb7fe700 1 --2- 172.21.15.29:0/1924334192 >> [v2:172.21.15.29:6800/1563185523,v1:172.21.15.29:6801/1563185523] conn(0x7f55c002fdb0 0x7f55c0032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:41:37.136 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:37.134+0000 7f55bb7fe700 1 -- 172.21.15.29:0/1924334192 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f55c40420e0 con 0x7f55d40fbcb0 2024-01-21T11:41:37.136 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:37.134+0000 7f55d27fc700 1 --2- 172.21.15.29:0/1924334192 >> [v2:172.21.15.29:6800/1563185523,v1:172.21.15.29:6801/1563185523] conn(0x7f55c002fdb0 0x7f55c0032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:41:37.136 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:37.134+0000 7f55d937d700 1 -- 172.21.15.29:0/1924334192 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55d4059100 con 0x7f55d40fbcb0 2024-01-21T11:41:37.136 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:37.134+0000 7f55d27fc700 1 --2- 172.21.15.29:0/1924334192 >> [v2:172.21.15.29:6800/1563185523,v1:172.21.15.29:6801/1563185523] conn(0x7f55c002fdb0 0x7f55c0032270 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f55c80096a0 tx=0x7f55c8006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:41:37.139 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:37.139+0000 7f55bb7fe700 1 -- 172.21.15.29:0/1924334192 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f55c4016070 con 0x7f55d40fbcb0 2024-01-21T11:41:37.302 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:37.301+0000 7f55d937d700 1 -- 172.21.15.29:0/1924334192 --> [v2:172.21.15.29:6800/1563185523,v1:172.21.15.29:6801/1563185523] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f55d4104930 con 0x7f55c002fdb0 2024-01-21T11:41:37.607 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:37.607+0000 7f55bb7fe700 1 -- 172.21.15.29:0/1924334192 <== mgr.14116 v2:172.21.15.29:6800/1563185523 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f55d4104930 con 0x7f55c002fdb0 2024-01-21T11:41:37.609 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:37.609+0000 7f55d937d700 1 -- 172.21.15.29:0/1924334192 >> [v2:172.21.15.29:6800/1563185523,v1:172.21.15.29:6801/1563185523] conn(0x7f55c002fdb0 msgr2=0x7f55c0032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:41:37.609 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:37.609+0000 7f55d937d700 1 --2- 172.21.15.29:0/1924334192 >> [v2:172.21.15.29:6800/1563185523,v1:172.21.15.29:6801/1563185523] conn(0x7f55c002fdb0 0x7f55c0032270 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f55c80096a0 tx=0x7f55c8006b40).stop 2024-01-21T11:41:37.610 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:37.609+0000 7f55d937d700 1 -- 172.21.15.29:0/1924334192 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55d40fbcb0 msgr2=0x7f55d40725e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:41:37.610 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:37.609+0000 7f55d937d700 1 --2- 172.21.15.29:0/1924334192 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55d40fbcb0 0x7f55d40725e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f55c400e040 tx=0x7f55c4006ee0).stop 2024-01-21T11:41:37.610 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:37.609+0000 7f55d937d700 1 -- 172.21.15.29:0/1924334192 shutdown_connections 2024-01-21T11:41:37.610 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:37.609+0000 7f55d937d700 1 --2- 172.21.15.29:0/1924334192 >> [v2:172.21.15.29:6800/1563185523,v1:172.21.15.29:6801/1563185523] conn(0x7f55c002fdb0 0x7f55c0032270 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:41:37.610 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:37.609+0000 7f55d937d700 1 --2- 172.21.15.29:0/1924334192 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55d40fbcb0 0x7f55d40725e0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:41:37.610 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:37.609+0000 7f55d937d700 1 -- 172.21.15.29:0/1924334192 >> 172.21.15.29:0/1924334192 conn(0x7f55d40f74b0 msgr2=0x7f55d40f8f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:41:37.610 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:37.609+0000 7f55d937d700 1 -- 172.21.15.29:0/1924334192 shutdown_connections 2024-01-21T11:41:37.610 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:37.609+0000 7f55d937d700 1 -- 172.21.15.29:0/1924334192 wait complete. 2024-01-21T11:41:37.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:37 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cephadm 2024-01-21T11:41:37.303104+0000 mgr.smithi029.blhxxk (mgr.14116) 6 : cephadm [INF] Generating ssh key... 2024-01-21T11:41:37.943 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e4084074-b851-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp77cpgrhz:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpea05xzin:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm get-pub-key 2024-01-21T11:41:38.285 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:38.284+0000 7fe3c4208700 1 Processor -- start 2024-01-21T11:41:38.289 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:38.284+0000 7fe3c4208700 1 -- start start 2024-01-21T11:41:38.289 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:38.284+0000 7fe3c4208700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3bc070430 0x7fe3bc0fc6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T11:41:38.289 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:38.284+0000 7fe3c4208700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3bc0fcbc0 con 0x7fe3bc070430 2024-01-21T11:41:38.289 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:38.285+0000 7fe3c1fa4700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3bc070430 0x7fe3bc0fc6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:41:38.289 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:38.285+0000 7fe3c1fa4700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3bc070430 0x7fe3bc0fc6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46628/0 (socket says 172.21.15.29:46628) 2024-01-21T11:41:38.289 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:38.285+0000 7fe3c1fa4700 1 -- 172.21.15.29:0/3804132444 learned_addr learned my addr 172.21.15.29:0/3804132444 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-21T11:41:38.290 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:38.285+0000 7fe3c1fa4700 1 -- 172.21.15.29:0/3804132444 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3bc0fcd00 con 0x7fe3bc070430 2024-01-21T11:41:38.290 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:38.285+0000 7fe3c1fa4700 1 --2- 172.21.15.29:0/3804132444 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3bc070430 0x7fe3bc0fc6d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fe3b800b5f0 tx=0x7fe3b8006ee0).ready entity=mon.0 client_cookie=497faf6995af51e server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:41:38.290 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:38.285+0000 7fe3c0fa2700 1 -- 172.21.15.29:0/3804132444 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe3b800d600 con 0x7fe3bc070430 2024-01-21T11:41:38.290 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:38.285+0000 7fe3c0fa2700 1 -- 172.21.15.29:0/3804132444 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fe3b800dbd0 con 0x7fe3bc070430 2024-01-21T11:41:38.290 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:38.285+0000 7fe3c0fa2700 1 -- 172.21.15.29:0/3804132444 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe3b8011c00 con 0x7fe3bc070430 2024-01-21T11:41:38.290 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:38.286+0000 7fe3c4208700 1 -- 172.21.15.29:0/3804132444 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3bc070430 msgr2=0x7fe3bc0fc6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:41:38.290 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:38.286+0000 7fe3c4208700 1 --2- 172.21.15.29:0/3804132444 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3bc070430 0x7fe3bc0fc6d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fe3b800b5f0 tx=0x7fe3b8006ee0).stop 2024-01-21T11:41:38.290 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:38.286+0000 7fe3c4208700 1 -- 172.21.15.29:0/3804132444 shutdown_connections 2024-01-21T11:41:38.290 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:38.286+0000 7fe3c4208700 1 --2- 172.21.15.29:0/3804132444 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3bc070430 0x7fe3bc0fc6d0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:41:38.290 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:38.286+0000 7fe3c4208700 1 -- 172.21.15.29:0/3804132444 >> 172.21.15.29:0/3804132444 conn(0x7fe3bc0f63c0 msgr2=0x7fe3bc0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:41:38.290 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:38.286+0000 7fe3c4208700 1 -- 172.21.15.29:0/3804132444 shutdown_connections 2024-01-21T11:41:38.290 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:38.286+0000 7fe3c4208700 1 -- 172.21.15.29:0/3804132444 wait complete. 2024-01-21T11:41:38.290 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:38.287+0000 7fe3c4208700 1 Processor -- start 2024-01-21T11:41:38.290 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:38.287+0000 7fe3c4208700 1 -- start start 2024-01-21T11:41:38.290 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:38.287+0000 7fe3c4208700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3bc070430 0x7fe3bc111050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:41:38.290 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:38.287+0000 7fe3c4208700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3bc111540 con 0x7fe3bc070430 2024-01-21T11:41:38.290 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:38.287+0000 7fe3c1fa4700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3bc070430 0x7fe3bc111050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:41:38.291 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:38.287+0000 7fe3c1fa4700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3bc070430 0x7fe3bc111050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46634/0 (socket says 172.21.15.29:46634) 2024-01-21T11:41:38.291 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:38.287+0000 7fe3c1fa4700 1 -- 172.21.15.29:0/2000411174 learned_addr learned my addr 172.21.15.29:0/2000411174 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-21T11:41:38.291 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:38.287+0000 7fe3c1fa4700 1 -- 172.21.15.29:0/2000411174 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3b800b200 con 0x7fe3bc070430 2024-01-21T11:41:38.291 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:38.288+0000 7fe3c1fa4700 1 --2- 172.21.15.29:0/2000411174 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3bc070430 0x7fe3bc111050 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fe3b800e040 tx=0x7fe3b8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:41:38.291 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:38.288+0000 7fe3aa7fc700 1 -- 172.21.15.29:0/2000411174 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe3b800d9f0 con 0x7fe3bc070430 2024-01-21T11:41:38.291 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:38.288+0000 7fe3aa7fc700 1 -- 172.21.15.29:0/2000411174 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fe3b801a9f0 con 0x7fe3bc070430 2024-01-21T11:41:38.291 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:38.288+0000 7fe3c4208700 1 -- 172.21.15.29:0/2000411174 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3bc111740 con 0x7fe3bc070430 2024-01-21T11:41:38.291 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:38.288+0000 7fe3aa7fc700 1 -- 172.21.15.29:0/2000411174 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe3b801b8c0 con 0x7fe3bc070430 2024-01-21T11:41:38.291 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:38.288+0000 7fe3c4208700 1 -- 172.21.15.29:0/2000411174 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3bc111b50 con 0x7fe3bc070430 2024-01-21T11:41:38.291 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:38.289+0000 7fe3aa7fc700 1 -- 172.21.15.29:0/2000411174 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fe3b800f070 con 0x7fe3bc070430 2024-01-21T11:41:38.291 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:38.289+0000 7fe3c4208700 1 -- 172.21.15.29:0/2000411174 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3bc045ad0 con 0x7fe3bc070430 2024-01-21T11:41:38.291 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:38.289+0000 7fe3aa7fc700 1 --2- 172.21.15.29:0/2000411174 >> [v2:172.21.15.29:6800/1563185523,v1:172.21.15.29:6801/1563185523] conn(0x7fe3a402fdb0 0x7fe3a4032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:41:38.291 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:38.289+0000 7fe3aa7fc700 1 -- 172.21.15.29:0/2000411174 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fe3b80413a0 con 0x7fe3bc070430 2024-01-21T11:41:38.291 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:38.289+0000 7fe3c17a3700 1 --2- 172.21.15.29:0/2000411174 >> [v2:172.21.15.29:6800/1563185523,v1:172.21.15.29:6801/1563185523] conn(0x7fe3a402fdb0 0x7fe3a4032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:41:38.292 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:38.289+0000 7fe3c17a3700 1 --2- 172.21.15.29:0/2000411174 >> [v2:172.21.15.29:6800/1563185523,v1:172.21.15.29:6801/1563185523] conn(0x7fe3a402fdb0 0x7fe3a4032270 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fe3ac0096a0 tx=0x7fe3ac006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:41:38.293 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:38.293+0000 7fe3aa7fc700 1 -- 172.21.15.29:0/2000411174 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe3b8006a70 con 0x7fe3bc070430 2024-01-21T11:41:38.454 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:38.453+0000 7fe3c4208700 1 -- 172.21.15.29:0/2000411174 --> [v2:172.21.15.29:6800/1563185523,v1:172.21.15.29:6801/1563185523] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fe3bc112d20 con 0x7fe3a402fdb0 2024-01-21T11:41:38.454 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:38.454+0000 7fe3aa7fc700 1 -- 172.21.15.29:0/2000411174 <== mgr.14116 v2:172.21.15.29:6800/1563185523 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7fe3bc112d20 con 0x7fe3a402fdb0 2024-01-21T11:41:38.454 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCzYY7NOXmrvZvCsV3lwxHRFOYUUFRZNS1WnQdeVkaNeylw7qpYI0RNpM6hpzRd0EiOUSPtSpc5hofN+fnIk0VDvelykm5CUqaHLx+GlPMXMi9Euq7IfZUqf/BCjGlwZDb9bOPV1kDu6cNHZf98ys3fyyvJSFraDinff83Y2bza7ERRbOHDrCRfN5A74Uwk3oCxNbFOETe0hgeI3AUDk7OT8hpYy8hlj4ALsz5b6BO6k4eqrmgtQSMI8wOMx5dhplgNlSbkWA3BGn7IxO6KDH7gwcge4K+nLOE79+r74AnRfZC0kTEJQt+n4ovjkWFV36LTv3vX0u83FR4EcBqdokb3JprCgfeHO1fvUmofF3+IK0sEn7jDnf6zgCKwNaWWk261wbdGkFXcR/38VF8OV+9X4UP+y9a+j/s8BoERlmXpAVTG4HDBDwdi4KRjxXSwO010Bk719KOkLSzcYKwltySTo9Rca7Vj733blQpnuWl6GkeTSol++flMwBgnj3s8YS0= ceph-e4084074-b851-11ee-95b0-87774f69a715 2024-01-21T11:41:38.457 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:38.456+0000 7fe3c4208700 1 -- 172.21.15.29:0/2000411174 >> [v2:172.21.15.29:6800/1563185523,v1:172.21.15.29:6801/1563185523] conn(0x7fe3a402fdb0 msgr2=0x7fe3a4032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:41:38.457 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:38.456+0000 7fe3c4208700 1 --2- 172.21.15.29:0/2000411174 >> [v2:172.21.15.29:6800/1563185523,v1:172.21.15.29:6801/1563185523] conn(0x7fe3a402fdb0 0x7fe3a4032270 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fe3ac0096a0 tx=0x7fe3ac006b40).stop 2024-01-21T11:41:38.457 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:38.456+0000 7fe3c4208700 1 -- 172.21.15.29:0/2000411174 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3bc070430 msgr2=0x7fe3bc111050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:41:38.457 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:38.456+0000 7fe3c4208700 1 --2- 172.21.15.29:0/2000411174 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3bc070430 0x7fe3bc111050 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fe3b800e040 tx=0x7fe3b8006ee0).stop 2024-01-21T11:41:38.457 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:38.456+0000 7fe3c4208700 1 -- 172.21.15.29:0/2000411174 shutdown_connections 2024-01-21T11:41:38.457 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:38.456+0000 7fe3c4208700 1 --2- 172.21.15.29:0/2000411174 >> [v2:172.21.15.29:6800/1563185523,v1:172.21.15.29:6801/1563185523] conn(0x7fe3a402fdb0 0x7fe3a4032270 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:41:38.457 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:38.456+0000 7fe3c4208700 1 --2- 172.21.15.29:0/2000411174 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3bc070430 0x7fe3bc111050 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:41:38.457 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:38.456+0000 7fe3c4208700 1 -- 172.21.15.29:0/2000411174 >> 172.21.15.29:0/2000411174 conn(0x7fe3bc0f63c0 msgr2=0x7fe3bc0f7e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:41:38.457 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:38.456+0000 7fe3c4208700 1 -- 172.21.15.29:0/2000411174 shutdown_connections 2024-01-21T11:41:38.457 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:38.456+0000 7fe3c4208700 1 -- 172.21.15.29:0/2000411174 wait complete. 2024-01-21T11:41:38.737 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:38 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:41:37.605721+0000 mon.smithi029 (mon.0) 53 : audit [INF] from='mgr.14116 172.21.15.29:0/1084824939' entity='mgr.smithi029.blhxxk' 2024-01-21T11:41:38.737 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:38 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:41:37.607375+0000 mon.smithi029 (mon.0) 54 : audit [INF] from='mgr.14116 172.21.15.29:0/1084824939' entity='mgr.smithi029.blhxxk' 2024-01-21T11:41:38.737 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:38 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:41:38.455039+0000 mgr.smithi029.blhxxk (mgr.14116) 7 : audit [DBG] from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T11:41:38.794 INFO:teuthology.orchestra.run.smithi029.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-01-21T11:41:38.794 INFO:teuthology.orchestra.run.smithi029.stderr:Adding key to root@localhost authorized_keys... 2024-01-21T11:41:38.795 INFO:teuthology.orchestra.run.smithi029.stderr:Adding host smithi029... 2024-01-21T11:41:38.795 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e4084074-b851-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp77cpgrhz:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpea05xzin:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch host add smithi029 172.21.15.29 2024-01-21T11:41:39.146 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:39.145+0000 7f31e4d12700 1 Processor -- start 2024-01-21T11:41:39.146 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:39.145+0000 7f31e4d12700 1 -- start start 2024-01-21T11:41:39.146 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:39.145+0000 7f31e4d12700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31e00fd100 0x7f31e00fd520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T11:41:39.146 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:39.145+0000 7f31e4d12700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31e00fda10 con 0x7f31e00fd100 2024-01-21T11:41:39.146 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:39.145+0000 7f31df7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31e00fd100 0x7f31e00fd520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:41:39.146 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:39.145+0000 7f31df7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31e00fd100 0x7f31e00fd520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46640/0 (socket says 172.21.15.29:46640) 2024-01-21T11:41:39.146 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:39.146+0000 7f31df7fe700 1 -- 172.21.15.29:0/1657755413 learned_addr learned my addr 172.21.15.29:0/1657755413 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-21T11:41:39.146 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:39.146+0000 7f31df7fe700 1 -- 172.21.15.29:0/1657755413 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31e00fdb50 con 0x7f31e00fd100 2024-01-21T11:41:39.146 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:39.146+0000 7f31df7fe700 1 --2- 172.21.15.29:0/1657755413 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31e00fd100 0x7f31e00fd520 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f31d000b5f0 tx=0x7f31d0006ee0).ready entity=mon.0 client_cookie=bdcb4575e15045fb server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:41:39.147 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:39.146+0000 7f31de7fc700 1 -- 172.21.15.29:0/1657755413 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f31d000d600 con 0x7f31e00fd100 2024-01-21T11:41:39.149 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:39.146+0000 7f31de7fc700 1 -- 172.21.15.29:0/1657755413 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f31d000dbd0 con 0x7f31e00fd100 2024-01-21T11:41:39.149 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:39.146+0000 7f31de7fc700 1 -- 172.21.15.29:0/1657755413 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f31d0011c00 con 0x7f31e00fd100 2024-01-21T11:41:39.149 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:39.147+0000 7f31e4d12700 1 -- 172.21.15.29:0/1657755413 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31e00fd100 msgr2=0x7f31e00fd520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:41:39.149 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:39.147+0000 7f31e4d12700 1 --2- 172.21.15.29:0/1657755413 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31e00fd100 0x7f31e00fd520 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f31d000b5f0 tx=0x7f31d0006ee0).stop 2024-01-21T11:41:39.149 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:39.147+0000 7f31e4d12700 1 -- 172.21.15.29:0/1657755413 shutdown_connections 2024-01-21T11:41:39.149 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:39.147+0000 7f31e4d12700 1 --2- 172.21.15.29:0/1657755413 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31e00fd100 0x7f31e00fd520 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:41:39.149 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:39.147+0000 7f31e4d12700 1 -- 172.21.15.29:0/1657755413 >> 172.21.15.29:0/1657755413 conn(0x7f31e00f59a0 msgr2=0x7f31e00f7dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:41:39.149 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:39.147+0000 7f31e4d12700 1 -- 172.21.15.29:0/1657755413 shutdown_connections 2024-01-21T11:41:39.149 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:39.147+0000 7f31e4d12700 1 -- 172.21.15.29:0/1657755413 wait complete. 2024-01-21T11:41:39.149 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:39.147+0000 7f31e4d12700 1 Processor -- start 2024-01-21T11:41:39.149 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:39.148+0000 7f31e4d12700 1 -- start start 2024-01-21T11:41:39.149 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:39.148+0000 7f31e4d12700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31e00fd100 0x7f31e010a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:41:39.150 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:39.148+0000 7f31e4d12700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31e010ade0 con 0x7f31e00fd100 2024-01-21T11:41:39.150 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:39.148+0000 7f31df7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31e00fd100 0x7f31e010a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:41:39.150 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:39.148+0000 7f31df7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31e00fd100 0x7f31e010a8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46656/0 (socket says 172.21.15.29:46656) 2024-01-21T11:41:39.150 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:39.148+0000 7f31df7fe700 1 -- 172.21.15.29:0/3112164940 learned_addr learned my addr 172.21.15.29:0/3112164940 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-21T11:41:39.150 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:39.148+0000 7f31df7fe700 1 -- 172.21.15.29:0/3112164940 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31d000b200 con 0x7f31e00fd100 2024-01-21T11:41:39.150 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:39.148+0000 7f31df7fe700 1 --2- 172.21.15.29:0/3112164940 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31e00fd100 0x7f31e010a8f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f31d000b1d0 tx=0x7f31d0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:41:39.150 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:39.148+0000 7f31c7fff700 1 -- 172.21.15.29:0/3112164940 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f31d0011690 con 0x7f31e00fd100 2024-01-21T11:41:39.150 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:39.149+0000 7f31c7fff700 1 -- 172.21.15.29:0/3112164940 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f31d001b880 con 0x7f31e00fd100 2024-01-21T11:41:39.150 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:39.149+0000 7f31c7fff700 1 -- 172.21.15.29:0/3112164940 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f31d001a650 con 0x7f31e00fd100 2024-01-21T11:41:39.150 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:39.149+0000 7f31e4d12700 1 -- 172.21.15.29:0/3112164940 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31e010afe0 con 0x7f31e00fd100 2024-01-21T11:41:39.151 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:39.149+0000 7f31e4d12700 1 -- 172.21.15.29:0/3112164940 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31e010b3f0 con 0x7f31e00fd100 2024-01-21T11:41:39.151 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:39.150+0000 7f31c7fff700 1 -- 172.21.15.29:0/3112164940 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f31d000f070 con 0x7f31e00fd100 2024-01-21T11:41:39.151 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:39.150+0000 7f31c7fff700 1 --2- 172.21.15.29:0/3112164940 >> [v2:172.21.15.29:6800/1563185523,v1:172.21.15.29:6801/1563185523] conn(0x7f31cc02fdb0 0x7f31cc032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:41:39.151 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:39.150+0000 7f31c7fff700 1 -- 172.21.15.29:0/3112164940 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f31d0040510 con 0x7f31e00fd100 2024-01-21T11:41:39.151 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:39.150+0000 7f31deffd700 1 --2- 172.21.15.29:0/3112164940 >> [v2:172.21.15.29:6800/1563185523,v1:172.21.15.29:6801/1563185523] conn(0x7f31cc02fdb0 0x7f31cc032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:41:39.151 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:39.150+0000 7f31e4d12700 1 -- 172.21.15.29:0/3112164940 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31e0059100 con 0x7f31e00fd100 2024-01-21T11:41:39.151 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:39.150+0000 7f31deffd700 1 --2- 172.21.15.29:0/3112164940 >> [v2:172.21.15.29:6800/1563185523,v1:172.21.15.29:6801/1563185523] conn(0x7f31cc02fdb0 0x7f31cc032270 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f31d40096a0 tx=0x7f31d4006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:41:39.154 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:39.154+0000 7f31c7fff700 1 -- 172.21.15.29:0/3112164940 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f31d0016070 con 0x7f31e00fd100 2024-01-21T11:41:39.336 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:39.336+0000 7f31e4d12700 1 -- 172.21.15.29:0/3112164940 --> [v2:172.21.15.29:6800/1563185523,v1:172.21.15.29:6801/1563185523] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi029", "addr": "172.21.15.29", "target": ["mon-mgr", ""]}) v1 -- 0x7f31e0104430 con 0x7f31cc02fdb0 2024-01-21T11:41:39.668 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:39 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:41:39.337731+0000 mgr.smithi029.blhxxk (mgr.14116) 8 : audit [DBG] from='client.14134 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi029", "addr": "172.21.15.29", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T11:41:39.950 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: Added host 'smithi029' 2024-01-21T11:41:39.950 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:39.948+0000 7f31c7fff700 1 -- 172.21.15.29:0/3112164940 <== mgr.14116 v2:172.21.15.29:6800/1563185523 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7f31e0104430 con 0x7f31cc02fdb0 2024-01-21T11:41:39.951 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:39.951+0000 7f31e4d12700 1 -- 172.21.15.29:0/3112164940 >> [v2:172.21.15.29:6800/1563185523,v1:172.21.15.29:6801/1563185523] conn(0x7f31cc02fdb0 msgr2=0x7f31cc032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:41:39.951 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:39.951+0000 7f31e4d12700 1 --2- 172.21.15.29:0/3112164940 >> [v2:172.21.15.29:6800/1563185523,v1:172.21.15.29:6801/1563185523] conn(0x7f31cc02fdb0 0x7f31cc032270 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f31d40096a0 tx=0x7f31d4006b40).stop 2024-01-21T11:41:39.952 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:39.951+0000 7f31e4d12700 1 -- 172.21.15.29:0/3112164940 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31e00fd100 msgr2=0x7f31e010a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:41:39.952 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:39.951+0000 7f31e4d12700 1 --2- 172.21.15.29:0/3112164940 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31e00fd100 0x7f31e010a8f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f31d000b1d0 tx=0x7f31d0006ee0).stop 2024-01-21T11:41:39.952 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:39.951+0000 7f31e4d12700 1 -- 172.21.15.29:0/3112164940 shutdown_connections 2024-01-21T11:41:39.952 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:39.951+0000 7f31e4d12700 1 --2- 172.21.15.29:0/3112164940 >> [v2:172.21.15.29:6800/1563185523,v1:172.21.15.29:6801/1563185523] conn(0x7f31cc02fdb0 0x7f31cc032270 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:41:39.952 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:39.951+0000 7f31e4d12700 1 --2- 172.21.15.29:0/3112164940 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31e00fd100 0x7f31e010a8f0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:41:39.952 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:39.951+0000 7f31e4d12700 1 -- 172.21.15.29:0/3112164940 >> 172.21.15.29:0/3112164940 conn(0x7f31e00f59a0 msgr2=0x7f31e00ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:41:39.952 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:39.951+0000 7f31e4d12700 1 -- 172.21.15.29:0/3112164940 shutdown_connections 2024-01-21T11:41:39.952 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:39.951+0000 7f31e4d12700 1 -- 172.21.15.29:0/3112164940 wait complete. 2024-01-21T11:41:40.253 INFO:teuthology.orchestra.run.smithi029.stderr:Deploying mon service with default placement... 2024-01-21T11:41:40.253 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e4084074-b851-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp77cpgrhz:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpea05xzin:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mon 2024-01-21T11:41:40.610 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:40.608+0000 7f4aa22d9700 1 Processor -- start 2024-01-21T11:41:40.610 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:40.608+0000 7f4aa22d9700 1 -- start start 2024-01-21T11:41:40.610 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:40.608+0000 7f4aa22d9700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a9c0fbcb0 0x7f4a9c0fc0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T11:41:40.610 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:40.608+0000 7f4aa22d9700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a9c0fc5c0 con 0x7f4a9c0fbcb0 2024-01-21T11:41:40.610 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:40.609+0000 7f4a9bfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a9c0fbcb0 0x7f4a9c0fc0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:41:40.610 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:40.609+0000 7f4a9bfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a9c0fbcb0 0x7f4a9c0fc0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46672/0 (socket says 172.21.15.29:46672) 2024-01-21T11:41:40.610 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:40.609+0000 7f4a9bfff700 1 -- 172.21.15.29:0/2134961420 learned_addr learned my addr 172.21.15.29:0/2134961420 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-21T11:41:40.610 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:40.609+0000 7f4a9bfff700 1 -- 172.21.15.29:0/2134961420 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a9c0fc700 con 0x7f4a9c0fbcb0 2024-01-21T11:41:40.610 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:40.609+0000 7f4a9bfff700 1 --2- 172.21.15.29:0/2134961420 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a9c0fbcb0 0x7f4a9c0fc0d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f4a8400b5f0 tx=0x7f4a84006ee0).ready entity=mon.0 client_cookie=59950e952b8e1a34 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:41:40.610 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:40.609+0000 7f4a9affd700 1 -- 172.21.15.29:0/2134961420 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a8400d600 con 0x7f4a9c0fbcb0 2024-01-21T11:41:40.612 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:40.609+0000 7f4a9affd700 1 -- 172.21.15.29:0/2134961420 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f4a8400dbd0 con 0x7f4a9c0fbcb0 2024-01-21T11:41:40.612 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:40.609+0000 7f4a9affd700 1 -- 172.21.15.29:0/2134961420 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a84011c00 con 0x7f4a9c0fbcb0 2024-01-21T11:41:40.612 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:40.610+0000 7f4aa22d9700 1 -- 172.21.15.29:0/2134961420 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a9c0fbcb0 msgr2=0x7f4a9c0fc0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:41:40.613 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:40.610+0000 7f4aa22d9700 1 --2- 172.21.15.29:0/2134961420 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a9c0fbcb0 0x7f4a9c0fc0d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f4a8400b5f0 tx=0x7f4a84006ee0).stop 2024-01-21T11:41:40.613 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:40.610+0000 7f4aa22d9700 1 -- 172.21.15.29:0/2134961420 shutdown_connections 2024-01-21T11:41:40.613 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:40.610+0000 7f4aa22d9700 1 --2- 172.21.15.29:0/2134961420 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a9c0fbcb0 0x7f4a9c0fc0d0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:41:40.613 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:40.610+0000 7f4aa22d9700 1 -- 172.21.15.29:0/2134961420 >> 172.21.15.29:0/2134961420 conn(0x7f4a9c0f74b0 msgr2=0x7f4a9c0f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:41:40.613 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:40.610+0000 7f4aa22d9700 1 -- 172.21.15.29:0/2134961420 shutdown_connections 2024-01-21T11:41:40.613 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:40.610+0000 7f4aa22d9700 1 -- 172.21.15.29:0/2134961420 wait complete. 2024-01-21T11:41:40.613 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:40.610+0000 7f4aa22d9700 1 Processor -- start 2024-01-21T11:41:40.613 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:40.611+0000 7f4aa22d9700 1 -- start start 2024-01-21T11:41:40.613 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:40.611+0000 7f4aa22d9700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a9c0fbcb0 0x7f4a9c0725e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:41:40.613 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:40.611+0000 7f4aa22d9700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a9c070a60 con 0x7f4a9c0fbcb0 2024-01-21T11:41:40.613 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:40.611+0000 7f4a9bfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a9c0fbcb0 0x7f4a9c0725e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:41:40.613 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:40.611+0000 7f4a9bfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a9c0fbcb0 0x7f4a9c0725e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46676/0 (socket says 172.21.15.29:46676) 2024-01-21T11:41:40.613 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:40.611+0000 7f4a9bfff700 1 -- 172.21.15.29:0/261681649 learned_addr learned my addr 172.21.15.29:0/261681649 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-21T11:41:40.613 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:40.611+0000 7f4a9bfff700 1 -- 172.21.15.29:0/261681649 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a8400b200 con 0x7f4a9c0fbcb0 2024-01-21T11:41:40.614 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:40.611+0000 7f4a9bfff700 1 --2- 172.21.15.29:0/261681649 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a9c0fbcb0 0x7f4a9c0725e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f4a8400b1d0 tx=0x7f4a84006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:41:40.614 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:40.611+0000 7f4a98ff9700 1 -- 172.21.15.29:0/261681649 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a8401b410 con 0x7f4a9c0fbcb0 2024-01-21T11:41:40.614 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:40.611+0000 7f4aa22d9700 1 -- 172.21.15.29:0/261681649 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a9c070c00 con 0x7f4a9c0fbcb0 2024-01-21T11:41:40.614 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:40.611+0000 7f4a98ff9700 1 -- 172.21.15.29:0/261681649 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f4a8401b9e0 con 0x7f4a9c0fbcb0 2024-01-21T11:41:40.614 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:40.611+0000 7f4a98ff9700 1 -- 172.21.15.29:0/261681649 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a8401a840 con 0x7f4a9c0fbcb0 2024-01-21T11:41:40.614 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:40.612+0000 7f4aa22d9700 1 -- 172.21.15.29:0/261681649 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a9c071010 con 0x7f4a9c0fbcb0 2024-01-21T11:41:40.614 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:40.612+0000 7f4aa22d9700 1 -- 172.21.15.29:0/261681649 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a9c045ad0 con 0x7f4a9c0fbcb0 2024-01-21T11:41:40.615 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:40.613+0000 7f4a98ff9700 1 -- 172.21.15.29:0/261681649 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f4a8400f070 con 0x7f4a9c0fbcb0 2024-01-21T11:41:40.615 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:40.613+0000 7f4a98ff9700 1 --2- 172.21.15.29:0/261681649 >> [v2:172.21.15.29:6800/1563185523,v1:172.21.15.29:6801/1563185523] conn(0x7f4a7c02fd50 0x7f4a7c032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:41:40.615 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:40.613+0000 7f4a98ff9700 1 -- 172.21.15.29:0/261681649 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f4a84041d70 con 0x7f4a9c0fbcb0 2024-01-21T11:41:40.615 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:40.613+0000 7f4a9b7fe700 1 --2- 172.21.15.29:0/261681649 >> [v2:172.21.15.29:6800/1563185523,v1:172.21.15.29:6801/1563185523] conn(0x7f4a7c02fd50 0x7f4a7c032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:41:40.615 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:40.614+0000 7f4a9b7fe700 1 --2- 172.21.15.29:0/261681649 >> [v2:172.21.15.29:6800/1563185523,v1:172.21.15.29:6801/1563185523] conn(0x7f4a7c02fd50 0x7f4a7c032210 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f4a8c0096a0 tx=0x7f4a8c006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:41:40.617 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:40.617+0000 7f4a98ff9700 1 -- 172.21.15.29:0/261681649 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4a8402db70 con 0x7f4a9c0fbcb0 2024-01-21T11:41:40.803 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:40.803+0000 7f4aa22d9700 1 -- 172.21.15.29:0/261681649 --> [v2:172.21.15.29:6800/1563185523,v1:172.21.15.29:6801/1563185523] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7f4a9c071250 con 0x7f4a7c02fd50 2024-01-21T11:41:40.807 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:40.807+0000 7f4a98ff9700 1 -- 172.21.15.29:0/261681649 <== mgr.14116 v2:172.21.15.29:6800/1563185523 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f4a9c071250 con 0x7f4a7c02fd50 2024-01-21T11:41:40.807 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: Scheduled mon update... 2024-01-21T11:41:40.809 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:40.808+0000 7f4aa22d9700 1 -- 172.21.15.29:0/261681649 >> [v2:172.21.15.29:6800/1563185523,v1:172.21.15.29:6801/1563185523] conn(0x7f4a7c02fd50 msgr2=0x7f4a7c032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:41:40.809 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:40.808+0000 7f4aa22d9700 1 --2- 172.21.15.29:0/261681649 >> [v2:172.21.15.29:6800/1563185523,v1:172.21.15.29:6801/1563185523] conn(0x7f4a7c02fd50 0x7f4a7c032210 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f4a8c0096a0 tx=0x7f4a8c006b40).stop 2024-01-21T11:41:40.809 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:40.808+0000 7f4aa22d9700 1 -- 172.21.15.29:0/261681649 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a9c0fbcb0 msgr2=0x7f4a9c0725e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:41:40.809 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:40.808+0000 7f4aa22d9700 1 --2- 172.21.15.29:0/261681649 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a9c0fbcb0 0x7f4a9c0725e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f4a8400b1d0 tx=0x7f4a84006ee0).stop 2024-01-21T11:41:40.809 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:40.809+0000 7f4aa22d9700 1 -- 172.21.15.29:0/261681649 shutdown_connections 2024-01-21T11:41:40.809 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:40.809+0000 7f4aa22d9700 1 --2- 172.21.15.29:0/261681649 >> [v2:172.21.15.29:6800/1563185523,v1:172.21.15.29:6801/1563185523] conn(0x7f4a7c02fd50 0x7f4a7c032210 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:41:40.809 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:40.809+0000 7f4aa22d9700 1 --2- 172.21.15.29:0/261681649 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a9c0fbcb0 0x7f4a9c0725e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:41:40.809 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:40.809+0000 7f4aa22d9700 1 -- 172.21.15.29:0/261681649 >> 172.21.15.29:0/261681649 conn(0x7f4a9c0f74b0 msgr2=0x7f4a9c0f8f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:41:40.810 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:40.809+0000 7f4aa22d9700 1 -- 172.21.15.29:0/261681649 shutdown_connections 2024-01-21T11:41:40.810 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:40.809+0000 7f4aa22d9700 1 -- 172.21.15.29:0/261681649 wait complete. 2024-01-21T11:41:41.142 INFO:teuthology.orchestra.run.smithi029.stderr:Deploying mgr service with default placement... 2024-01-21T11:41:41.142 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e4084074-b851-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp77cpgrhz:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpea05xzin:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mgr 2024-01-21T11:41:41.168 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:40 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:41:39.949168+0000 mon.smithi029 (mon.0) 55 : audit [INF] from='mgr.14116 172.21.15.29:0/1084824939' entity='mgr.smithi029.blhxxk' 2024-01-21T11:41:41.168 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:40 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cephadm 2024-01-21T11:41:39.949474+0000 mgr.smithi029.blhxxk (mgr.14116) 9 : cephadm [INF] Added host smithi029 2024-01-21T11:41:41.168 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:40 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:41:39.949722+0000 mon.smithi029 (mon.0) 56 : audit [DBG] from='mgr.14116 172.21.15.29:0/1084824939' entity='mgr.smithi029.blhxxk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-21T11:41:41.168 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:40 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:41:40.807780+0000 mon.smithi029 (mon.0) 57 : audit [INF] from='mgr.14116 172.21.15.29:0/1084824939' entity='mgr.smithi029.blhxxk' 2024-01-21T11:41:41.473 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:41.472+0000 7f011df8f700 1 Processor -- start 2024-01-21T11:41:41.473 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:41.472+0000 7f011df8f700 1 -- start start 2024-01-21T11:41:41.473 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:41.472+0000 7f011df8f700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f01180fab80 0x7f01180fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T11:41:41.473 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:41.472+0000 7f011df8f700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01180fb490 con 0x7f01180fab80 2024-01-21T11:41:41.473 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:41.473+0000 7f01177fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f01180fab80 0x7f01180fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:41:41.473 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:41.473+0000 7f01177fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f01180fab80 0x7f01180fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46692/0 (socket says 172.21.15.29:46692) 2024-01-21T11:41:41.473 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:41.473+0000 7f01177fe700 1 -- 172.21.15.29:0/2483981358 learned_addr learned my addr 172.21.15.29:0/2483981358 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-21T11:41:41.476 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:41.473+0000 7f01177fe700 1 -- 172.21.15.29:0/2483981358 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01180fb5d0 con 0x7f01180fab80 2024-01-21T11:41:41.476 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:41.473+0000 7f01177fe700 1 --2- 172.21.15.29:0/2483981358 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f01180fab80 0x7f01180fafa0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f010000b5f0 tx=0x7f0100006ee0).ready entity=mon.0 client_cookie=7f8e8e55d20cf1f2 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:41:41.476 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:41.473+0000 7f01167fc700 1 -- 172.21.15.29:0/2483981358 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f010000d600 con 0x7f01180fab80 2024-01-21T11:41:41.476 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:41.473+0000 7f01167fc700 1 -- 172.21.15.29:0/2483981358 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f010000dbd0 con 0x7f01180fab80 2024-01-21T11:41:41.476 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:41.473+0000 7f01167fc700 1 -- 172.21.15.29:0/2483981358 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0100011c00 con 0x7f01180fab80 2024-01-21T11:41:41.476 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:41.474+0000 7f011df8f700 1 -- 172.21.15.29:0/2483981358 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f01180fab80 msgr2=0x7f01180fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:41:41.476 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:41.474+0000 7f011df8f700 1 --2- 172.21.15.29:0/2483981358 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f01180fab80 0x7f01180fafa0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f010000b5f0 tx=0x7f0100006ee0).stop 2024-01-21T11:41:41.476 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:41.474+0000 7f011df8f700 1 -- 172.21.15.29:0/2483981358 shutdown_connections 2024-01-21T11:41:41.476 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:41.474+0000 7f011df8f700 1 --2- 172.21.15.29:0/2483981358 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f01180fab80 0x7f01180fafa0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:41:41.477 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:41.474+0000 7f011df8f700 1 -- 172.21.15.29:0/2483981358 >> 172.21.15.29:0/2483981358 conn(0x7f01180f63c0 msgr2=0x7f01180f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:41:41.477 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:41.474+0000 7f011df8f700 1 -- 172.21.15.29:0/2483981358 shutdown_connections 2024-01-21T11:41:41.477 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:41.474+0000 7f011df8f700 1 -- 172.21.15.29:0/2483981358 wait complete. 2024-01-21T11:41:41.477 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:41.475+0000 7f011df8f700 1 Processor -- start 2024-01-21T11:41:41.477 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:41.475+0000 7f011df8f700 1 -- start start 2024-01-21T11:41:41.477 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:41.475+0000 7f011df8f700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f01180fab80 0x7f011810aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:41:41.477 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:41.475+0000 7f011df8f700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f011810af00 con 0x7f01180fab80 2024-01-21T11:41:41.477 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:41.475+0000 7f01177fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f01180fab80 0x7f011810aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:41:41.477 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:41.475+0000 7f01177fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f01180fab80 0x7f011810aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46708/0 (socket says 172.21.15.29:46708) 2024-01-21T11:41:41.477 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:41.475+0000 7f01177fe700 1 -- 172.21.15.29:0/152965717 learned_addr learned my addr 172.21.15.29:0/152965717 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-21T11:41:41.477 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:41.475+0000 7f01177fe700 1 -- 172.21.15.29:0/152965717 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f010000b200 con 0x7f01180fab80 2024-01-21T11:41:41.477 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:41.476+0000 7f01177fe700 1 --2- 172.21.15.29:0/152965717 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f01180fab80 0x7f011810aa10 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f010000b1d0 tx=0x7f0100006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:41:41.477 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:41.476+0000 7f010ffff700 1 -- 172.21.15.29:0/152965717 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f010001b410 con 0x7f01180fab80 2024-01-21T11:41:41.477 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:41.476+0000 7f010ffff700 1 -- 172.21.15.29:0/152965717 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f010001b9e0 con 0x7f01180fab80 2024-01-21T11:41:41.477 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:41.476+0000 7f010ffff700 1 -- 172.21.15.29:0/152965717 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f010001a840 con 0x7f01180fab80 2024-01-21T11:41:41.477 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:41.476+0000 7f011df8f700 1 -- 172.21.15.29:0/152965717 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f011810b100 con 0x7f01180fab80 2024-01-21T11:41:41.478 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:41.476+0000 7f011df8f700 1 -- 172.21.15.29:0/152965717 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f011810b510 con 0x7f01180fab80 2024-01-21T11:41:41.479 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:41.477+0000 7f010ffff700 1 -- 172.21.15.29:0/152965717 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f010000f070 con 0x7f01180fab80 2024-01-21T11:41:41.479 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:41.477+0000 7f011df8f700 1 -- 172.21.15.29:0/152965717 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0118045ad0 con 0x7f01180fab80 2024-01-21T11:41:41.479 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:41.477+0000 7f010ffff700 1 --2- 172.21.15.29:0/152965717 >> [v2:172.21.15.29:6800/1563185523,v1:172.21.15.29:6801/1563185523] conn(0x7f00f802fdb0 0x7f00f8032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:41:41.479 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:41.477+0000 7f010ffff700 1 -- 172.21.15.29:0/152965717 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f01000412b0 con 0x7f01180fab80 2024-01-21T11:41:41.479 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:41.477+0000 7f0116ffd700 1 --2- 172.21.15.29:0/152965717 >> [v2:172.21.15.29:6800/1563185523,v1:172.21.15.29:6801/1563185523] conn(0x7f00f802fdb0 0x7f00f8032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:41:41.479 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:41.477+0000 7f0116ffd700 1 --2- 172.21.15.29:0/152965717 >> [v2:172.21.15.29:6800/1563185523,v1:172.21.15.29:6801/1563185523] conn(0x7f00f802fdb0 0x7f00f8032270 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f01080096a0 tx=0x7f0108006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:41:41.481 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:41.481+0000 7f010ffff700 1 -- 172.21.15.29:0/152965717 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0100019cd0 con 0x7f01180fab80 2024-01-21T11:41:41.667 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:41.667+0000 7f011df8f700 1 -- 172.21.15.29:0/152965717 --> [v2:172.21.15.29:6800/1563185523,v1:172.21.15.29:6801/1563185523] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f011810bd80 con 0x7f00f802fdb0 2024-01-21T11:41:41.670 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:41.670+0000 7f010ffff700 1 -- 172.21.15.29:0/152965717 <== mgr.14116 v2:172.21.15.29:6800/1563185523 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f011810bd80 con 0x7f00f802fdb0 2024-01-21T11:41:41.670 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-01-21T11:41:41.673 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:41.672+0000 7f011df8f700 1 -- 172.21.15.29:0/152965717 >> [v2:172.21.15.29:6800/1563185523,v1:172.21.15.29:6801/1563185523] conn(0x7f00f802fdb0 msgr2=0x7f00f8032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:41:41.673 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:41.672+0000 7f011df8f700 1 --2- 172.21.15.29:0/152965717 >> [v2:172.21.15.29:6800/1563185523,v1:172.21.15.29:6801/1563185523] conn(0x7f00f802fdb0 0x7f00f8032270 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f01080096a0 tx=0x7f0108006b40).stop 2024-01-21T11:41:41.673 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:41.672+0000 7f011df8f700 1 -- 172.21.15.29:0/152965717 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f01180fab80 msgr2=0x7f011810aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:41:41.673 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:41.672+0000 7f011df8f700 1 --2- 172.21.15.29:0/152965717 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f01180fab80 0x7f011810aa10 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f010000b1d0 tx=0x7f0100006ee0).stop 2024-01-21T11:41:41.673 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:41.672+0000 7f011df8f700 1 -- 172.21.15.29:0/152965717 shutdown_connections 2024-01-21T11:41:41.673 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:41.672+0000 7f011df8f700 1 --2- 172.21.15.29:0/152965717 >> [v2:172.21.15.29:6800/1563185523,v1:172.21.15.29:6801/1563185523] conn(0x7f00f802fdb0 0x7f00f8032270 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:41:41.673 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:41.672+0000 7f011df8f700 1 --2- 172.21.15.29:0/152965717 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f01180fab80 0x7f011810aa10 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:41:41.673 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:41.672+0000 7f011df8f700 1 -- 172.21.15.29:0/152965717 >> 172.21.15.29:0/152965717 conn(0x7f01180f63c0 msgr2=0x7f0118070030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:41:41.673 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:41.672+0000 7f011df8f700 1 -- 172.21.15.29:0/152965717 shutdown_connections 2024-01-21T11:41:41.673 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:41.672+0000 7f011df8f700 1 -- 172.21.15.29:0/152965717 wait complete. 2024-01-21T11:41:42.026 INFO:teuthology.orchestra.run.smithi029.stderr:Deploying crash service with default placement... 2024-01-21T11:41:42.026 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e4084074-b851-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp77cpgrhz:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpea05xzin:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply crash 2024-01-21T11:41:42.353 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:42.350+0000 7fb3c1d28700 1 Processor -- start 2024-01-21T11:41:42.354 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:42.350+0000 7fb3c1d28700 1 -- start start 2024-01-21T11:41:42.354 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:42.350+0000 7fb3c1d28700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3bc0703d0 0x7fb3bc0fe940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T11:41:42.354 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:42.350+0000 7fb3c1d28700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3bc0fee30 con 0x7fb3bc0703d0 2024-01-21T11:41:42.354 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:42.350+0000 7fb3bb7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3bc0703d0 0x7fb3bc0fe940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:41:42.354 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:42.350+0000 7fb3bb7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3bc0703d0 0x7fb3bc0fe940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46722/0 (socket says 172.21.15.29:46722) 2024-01-21T11:41:42.354 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:42.350+0000 7fb3bb7fe700 1 -- 172.21.15.29:0/278418391 learned_addr learned my addr 172.21.15.29:0/278418391 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-21T11:41:42.354 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:42.350+0000 7fb3bb7fe700 1 -- 172.21.15.29:0/278418391 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3bc0fef70 con 0x7fb3bc0703d0 2024-01-21T11:41:42.354 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:42.350+0000 7fb3bb7fe700 1 --2- 172.21.15.29:0/278418391 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3bc0703d0 0x7fb3bc0fe940 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7fb3ac00b5f0 tx=0x7fb3ac006ee0).ready entity=mon.0 client_cookie=a6887bd67ad54784 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:41:42.354 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:42.351+0000 7fb3ba7fc700 1 -- 172.21.15.29:0/278418391 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb3ac00d600 con 0x7fb3bc0703d0 2024-01-21T11:41:42.354 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:42.351+0000 7fb3ba7fc700 1 -- 172.21.15.29:0/278418391 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fb3ac00dbd0 con 0x7fb3bc0703d0 2024-01-21T11:41:42.354 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:42.351+0000 7fb3ba7fc700 1 -- 172.21.15.29:0/278418391 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb3ac011c00 con 0x7fb3bc0703d0 2024-01-21T11:41:42.355 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:42.351+0000 7fb3c1d28700 1 -- 172.21.15.29:0/278418391 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3bc0703d0 msgr2=0x7fb3bc0fe940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:41:42.355 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:42.351+0000 7fb3c1d28700 1 --2- 172.21.15.29:0/278418391 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3bc0703d0 0x7fb3bc0fe940 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7fb3ac00b5f0 tx=0x7fb3ac006ee0).stop 2024-01-21T11:41:42.355 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:42.351+0000 7fb3c1d28700 1 -- 172.21.15.29:0/278418391 shutdown_connections 2024-01-21T11:41:42.355 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:42.351+0000 7fb3c1d28700 1 --2- 172.21.15.29:0/278418391 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3bc0703d0 0x7fb3bc0fe940 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:41:42.355 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:42.351+0000 7fb3c1d28700 1 -- 172.21.15.29:0/278418391 >> 172.21.15.29:0/278418391 conn(0x7fb3bc0f63c0 msgr2=0x7fb3bc0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:41:42.355 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:42.351+0000 7fb3c1d28700 1 -- 172.21.15.29:0/278418391 shutdown_connections 2024-01-21T11:41:42.355 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:42.351+0000 7fb3c1d28700 1 -- 172.21.15.29:0/278418391 wait complete. 2024-01-21T11:41:42.355 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:42.352+0000 7fb3c1d28700 1 Processor -- start 2024-01-21T11:41:42.355 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:42.352+0000 7fb3c1d28700 1 -- start start 2024-01-21T11:41:42.355 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:42.352+0000 7fb3c1d28700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3bc0703d0 0x7fb3bc0fc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:41:42.355 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:42.352+0000 7fb3c1d28700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3bc0fa840 con 0x7fb3bc0703d0 2024-01-21T11:41:42.355 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:42.352+0000 7fb3bb7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3bc0703d0 0x7fb3bc0fc3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:41:42.355 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:42.352+0000 7fb3bb7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3bc0703d0 0x7fb3bc0fc3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46730/0 (socket says 172.21.15.29:46730) 2024-01-21T11:41:42.356 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:42.352+0000 7fb3bb7fe700 1 -- 172.21.15.29:0/3275700907 learned_addr learned my addr 172.21.15.29:0/3275700907 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-21T11:41:42.356 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:42.353+0000 7fb3bb7fe700 1 -- 172.21.15.29:0/3275700907 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3ac00b200 con 0x7fb3bc0703d0 2024-01-21T11:41:42.356 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:42.353+0000 7fb3bb7fe700 1 --2- 172.21.15.29:0/3275700907 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3bc0703d0 0x7fb3bc0fc3c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7fb3ac00b190 tx=0x7fb3ac006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:41:42.356 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:42.353+0000 7fb3a3fff700 1 -- 172.21.15.29:0/3275700907 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb3ac01b410 con 0x7fb3bc0703d0 2024-01-21T11:41:42.356 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:42.353+0000 7fb3a3fff700 1 -- 172.21.15.29:0/3275700907 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fb3ac01b9e0 con 0x7fb3bc0703d0 2024-01-21T11:41:42.356 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:42.353+0000 7fb3a3fff700 1 -- 172.21.15.29:0/3275700907 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb3ac01a880 con 0x7fb3bc0703d0 2024-01-21T11:41:42.356 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:42.353+0000 7fb3c1d28700 1 -- 172.21.15.29:0/3275700907 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3bc0fa9e0 con 0x7fb3bc0703d0 2024-01-21T11:41:42.357 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:42.353+0000 7fb3c1d28700 1 -- 172.21.15.29:0/3275700907 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3bc0fadf0 con 0x7fb3bc0703d0 2024-01-21T11:41:42.357 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:42.354+0000 7fb3c1d28700 1 -- 172.21.15.29:0/3275700907 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3bc045ad0 con 0x7fb3bc0703d0 2024-01-21T11:41:42.357 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:42.354+0000 7fb3a3fff700 1 -- 172.21.15.29:0/3275700907 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fb3ac00f070 con 0x7fb3bc0703d0 2024-01-21T11:41:42.357 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:42.354+0000 7fb3a3fff700 1 --2- 172.21.15.29:0/3275700907 >> [v2:172.21.15.29:6800/1563185523,v1:172.21.15.29:6801/1563185523] conn(0x7fb3a802fd50 0x7fb3a8032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:41:42.357 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:42.354+0000 7fb3a3fff700 1 -- 172.21.15.29:0/3275700907 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fb3ac041290 con 0x7fb3bc0703d0 2024-01-21T11:41:42.357 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:42.355+0000 7fb3baffd700 1 --2- 172.21.15.29:0/3275700907 >> [v2:172.21.15.29:6800/1563185523,v1:172.21.15.29:6801/1563185523] conn(0x7fb3a802fd50 0x7fb3a8032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:41:42.357 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:42.355+0000 7fb3baffd700 1 --2- 172.21.15.29:0/3275700907 >> [v2:172.21.15.29:6800/1563185523,v1:172.21.15.29:6801/1563185523] conn(0x7fb3a802fd50 0x7fb3a8032210 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7fb3b00096a0 tx=0x7fb3b0006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:41:42.358 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:42.359+0000 7fb3a3fff700 1 -- 172.21.15.29:0/3275700907 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb3ac019e60 con 0x7fb3bc0703d0 2024-01-21T11:41:42.418 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:41 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:41:40.804942+0000 mgr.smithi029.blhxxk (mgr.14116) 10 : audit [DBG] from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T11:41:42.418 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:41 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cephadm 2024-01-21T11:41:40.805660+0000 mgr.smithi029.blhxxk (mgr.14116) 11 : cephadm [INF] Saving service mon spec with placement count:5 2024-01-21T11:41:42.418 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:41 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:41:41.671210+0000 mon.smithi029 (mon.0) 58 : audit [INF] from='mgr.14116 172.21.15.29:0/1084824939' entity='mgr.smithi029.blhxxk' 2024-01-21T11:41:42.545 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:42.545+0000 7fb3c1d28700 1 -- 172.21.15.29:0/3275700907 --> [v2:172.21.15.29:6800/1563185523,v1:172.21.15.29:6801/1563185523] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7fb3bc0fb030 con 0x7fb3a802fd50 2024-01-21T11:41:42.549 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:42.549+0000 7fb3a3fff700 1 -- 172.21.15.29:0/3275700907 <== mgr.14116 v2:172.21.15.29:6800/1563185523 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7fb3bc0fb030 con 0x7fb3a802fd50 2024-01-21T11:41:42.549 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: Scheduled crash update... 2024-01-21T11:41:42.551 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:42.550+0000 7fb3c1d28700 1 -- 172.21.15.29:0/3275700907 >> [v2:172.21.15.29:6800/1563185523,v1:172.21.15.29:6801/1563185523] conn(0x7fb3a802fd50 msgr2=0x7fb3a8032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:41:42.551 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:42.550+0000 7fb3c1d28700 1 --2- 172.21.15.29:0/3275700907 >> [v2:172.21.15.29:6800/1563185523,v1:172.21.15.29:6801/1563185523] conn(0x7fb3a802fd50 0x7fb3a8032210 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7fb3b00096a0 tx=0x7fb3b0006b40).stop 2024-01-21T11:41:42.551 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:42.550+0000 7fb3c1d28700 1 -- 172.21.15.29:0/3275700907 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3bc0703d0 msgr2=0x7fb3bc0fc3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:41:42.552 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:42.550+0000 7fb3c1d28700 1 --2- 172.21.15.29:0/3275700907 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3bc0703d0 0x7fb3bc0fc3c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7fb3ac00b190 tx=0x7fb3ac006ee0).stop 2024-01-21T11:41:42.552 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:42.550+0000 7fb3c1d28700 1 -- 172.21.15.29:0/3275700907 shutdown_connections 2024-01-21T11:41:42.552 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:42.551+0000 7fb3c1d28700 1 --2- 172.21.15.29:0/3275700907 >> [v2:172.21.15.29:6800/1563185523,v1:172.21.15.29:6801/1563185523] conn(0x7fb3a802fd50 0x7fb3a8032210 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:41:42.552 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:42.551+0000 7fb3c1d28700 1 --2- 172.21.15.29:0/3275700907 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3bc0703d0 0x7fb3bc0fc3c0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:41:42.552 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:42.551+0000 7fb3c1d28700 1 -- 172.21.15.29:0/3275700907 >> 172.21.15.29:0/3275700907 conn(0x7fb3bc0f63c0 msgr2=0x7fb3bc0f7ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:41:42.552 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:42.551+0000 7fb3c1d28700 1 -- 172.21.15.29:0/3275700907 shutdown_connections 2024-01-21T11:41:42.552 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:42.551+0000 7fb3c1d28700 1 -- 172.21.15.29:0/3275700907 wait complete. 2024-01-21T11:41:42.893 INFO:teuthology.orchestra.run.smithi029.stderr:Enabling mgr prometheus module... 2024-01-21T11:41:42.893 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e4084074-b851-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp77cpgrhz:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpea05xzin:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable prometheus 2024-01-21T11:41:43.168 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:42 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:41:41.668532+0000 mgr.smithi029.blhxxk (mgr.14116) 12 : audit [DBG] from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T11:41:43.168 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:42 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cephadm 2024-01-21T11:41:41.669065+0000 mgr.smithi029.blhxxk (mgr.14116) 13 : cephadm [INF] Saving service mgr spec with placement count:2 2024-01-21T11:41:43.168 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:42 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:41:42.549789+0000 mon.smithi029 (mon.0) 59 : audit [INF] from='mgr.14116 172.21.15.29:0/1084824939' entity='mgr.smithi029.blhxxk' 2024-01-21T11:41:43.239 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:43.239+0000 7f3889549700 1 Processor -- start 2024-01-21T11:41:43.239 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:43.239+0000 7f3889549700 1 -- start start 2024-01-21T11:41:43.242 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:43.239+0000 7f3889549700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3884070460 0x7f38840fe930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T11:41:43.242 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:43.239+0000 7f3889549700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38840fee20 con 0x7f3884070460 2024-01-21T11:41:43.242 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:43.239+0000 7f3882ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3884070460 0x7f38840fe930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:41:43.242 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:43.239+0000 7f3882ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3884070460 0x7f38840fe930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33080/0 (socket says 172.21.15.29:33080) 2024-01-21T11:41:43.242 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:43.239+0000 7f3882ffd700 1 -- 172.21.15.29:0/4218229107 learned_addr learned my addr 172.21.15.29:0/4218229107 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-21T11:41:43.242 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:43.240+0000 7f3882ffd700 1 -- 172.21.15.29:0/4218229107 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38840fef60 con 0x7f3884070460 2024-01-21T11:41:43.242 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:43.240+0000 7f3882ffd700 1 --2- 172.21.15.29:0/4218229107 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3884070460 0x7f38840fe930 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f387800b5f0 tx=0x7f3878006ee0).ready entity=mon.0 client_cookie=3353df8641a12854 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:41:43.242 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:43.240+0000 7f3881ffb700 1 -- 172.21.15.29:0/4218229107 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f387800d600 con 0x7f3884070460 2024-01-21T11:41:43.242 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:43.240+0000 7f3881ffb700 1 -- 172.21.15.29:0/4218229107 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f387800dbd0 con 0x7f3884070460 2024-01-21T11:41:43.242 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:43.240+0000 7f3881ffb700 1 -- 172.21.15.29:0/4218229107 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3878011c00 con 0x7f3884070460 2024-01-21T11:41:43.242 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:43.241+0000 7f3889549700 1 -- 172.21.15.29:0/4218229107 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3884070460 msgr2=0x7f38840fe930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:41:43.242 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:43.241+0000 7f3889549700 1 --2- 172.21.15.29:0/4218229107 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3884070460 0x7f38840fe930 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f387800b5f0 tx=0x7f3878006ee0).stop 2024-01-21T11:41:43.243 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:43.241+0000 7f3889549700 1 -- 172.21.15.29:0/4218229107 shutdown_connections 2024-01-21T11:41:43.243 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:43.241+0000 7f3889549700 1 --2- 172.21.15.29:0/4218229107 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3884070460 0x7f38840fe930 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:41:43.243 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:43.241+0000 7f3889549700 1 -- 172.21.15.29:0/4218229107 >> 172.21.15.29:0/4218229107 conn(0x7f38840f63b0 msgr2=0x7f38840f87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:41:43.243 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:43.241+0000 7f3889549700 1 -- 172.21.15.29:0/4218229107 shutdown_connections 2024-01-21T11:41:43.243 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:43.241+0000 7f3889549700 1 -- 172.21.15.29:0/4218229107 wait complete. 2024-01-21T11:41:43.243 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:43.242+0000 7f3889549700 1 Processor -- start 2024-01-21T11:41:43.243 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:43.242+0000 7f3889549700 1 -- start start 2024-01-21T11:41:43.243 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:43.242+0000 7f3889549700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3884070460 0x7f38840fc3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:41:43.243 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:43.242+0000 7f3889549700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38840fa830 con 0x7f3884070460 2024-01-21T11:41:43.243 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:43.242+0000 7f3882ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3884070460 0x7f38840fc3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:41:43.245 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:43.242+0000 7f3882ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3884070460 0x7f38840fc3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33096/0 (socket says 172.21.15.29:33096) 2024-01-21T11:41:43.245 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:43.242+0000 7f3882ffd700 1 -- 172.21.15.29:0/2476311633 learned_addr learned my addr 172.21.15.29:0/2476311633 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-21T11:41:43.245 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:43.242+0000 7f3882ffd700 1 -- 172.21.15.29:0/2476311633 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f387800b200 con 0x7f3884070460 2024-01-21T11:41:43.245 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:43.242+0000 7f3882ffd700 1 --2- 172.21.15.29:0/2476311633 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3884070460 0x7f38840fc3b0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f387800b190 tx=0x7f3878006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:41:43.245 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:43.242+0000 7f386b7fe700 1 -- 172.21.15.29:0/2476311633 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f387801b410 con 0x7f3884070460 2024-01-21T11:41:43.245 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:43.243+0000 7f3889549700 1 -- 172.21.15.29:0/2476311633 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38840fa9d0 con 0x7f3884070460 2024-01-21T11:41:43.245 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:43.243+0000 7f386b7fe700 1 -- 172.21.15.29:0/2476311633 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f387801b9e0 con 0x7f3884070460 2024-01-21T11:41:43.245 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:43.243+0000 7f386b7fe700 1 -- 172.21.15.29:0/2476311633 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f387801a880 con 0x7f3884070460 2024-01-21T11:41:43.245 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:43.243+0000 7f3889549700 1 -- 172.21.15.29:0/2476311633 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38840fade0 con 0x7f3884070460 2024-01-21T11:41:43.245 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:43.243+0000 7f386b7fe700 1 -- 172.21.15.29:0/2476311633 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f387800f070 con 0x7f3884070460 2024-01-21T11:41:43.245 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:43.244+0000 7f3889549700 1 -- 172.21.15.29:0/2476311633 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3884045ad0 con 0x7f3884070460 2024-01-21T11:41:43.246 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:43.244+0000 7f386b7fe700 1 --2- 172.21.15.29:0/2476311633 >> [v2:172.21.15.29:6800/1563185523,v1:172.21.15.29:6801/1563185523] conn(0x7f387002fdb0 0x7f3870032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:41:43.246 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:43.244+0000 7f386b7fe700 1 -- 172.21.15.29:0/2476311633 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f3878041170 con 0x7f3884070460 2024-01-21T11:41:43.246 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:43.244+0000 7f38827fc700 1 --2- 172.21.15.29:0/2476311633 >> [v2:172.21.15.29:6800/1563185523,v1:172.21.15.29:6801/1563185523] conn(0x7f387002fdb0 0x7f3870032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:41:43.246 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:43.244+0000 7f38827fc700 1 --2- 172.21.15.29:0/2476311633 >> [v2:172.21.15.29:6800/1563185523,v1:172.21.15.29:6801/1563185523] conn(0x7f387002fdb0 0x7f3870032270 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f38740096a0 tx=0x7f3874006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:41:43.248 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:43.248+0000 7f386b7fe700 1 -- 172.21.15.29:0/2476311633 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3878019950 con 0x7f3884070460 2024-01-21T11:41:43.443 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:43.443+0000 7f3889549700 1 -- 172.21.15.29:0/2476311633 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "prometheus"} v 0) v1 -- 0x7f38840ffbe0 con 0x7f3884070460 2024-01-21T11:41:43.953 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:43.953+0000 7f386b7fe700 1 -- 172.21.15.29:0/2476311633 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 9) v1 ==== 36134+0+0 (secure 0 0 0) 0x7f387801b570 con 0x7f3884070460 2024-01-21T11:41:43.954 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:43.954+0000 7f386b7fe700 1 -- 172.21.15.29:0/2476311633 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "prometheus"}]=0 v9) v1 ==== 89+0+0 (secure 0 0 0) 0x7f38780224f0 con 0x7f3884070460 2024-01-21T11:41:43.958 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:43.957+0000 7f3889549700 1 -- 172.21.15.29:0/2476311633 >> [v2:172.21.15.29:6800/1563185523,v1:172.21.15.29:6801/1563185523] conn(0x7f387002fdb0 msgr2=0x7f3870032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:41:43.958 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:43.957+0000 7f3889549700 1 --2- 172.21.15.29:0/2476311633 >> [v2:172.21.15.29:6800/1563185523,v1:172.21.15.29:6801/1563185523] conn(0x7f387002fdb0 0x7f3870032270 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f38740096a0 tx=0x7f3874006b40).stop 2024-01-21T11:41:43.958 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:43.957+0000 7f3889549700 1 -- 172.21.15.29:0/2476311633 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3884070460 msgr2=0x7f38840fc3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:41:43.958 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:43.957+0000 7f3889549700 1 --2- 172.21.15.29:0/2476311633 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3884070460 0x7f38840fc3b0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f387800b190 tx=0x7f3878006ee0).stop 2024-01-21T11:41:43.959 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:43.957+0000 7f3889549700 1 -- 172.21.15.29:0/2476311633 shutdown_connections 2024-01-21T11:41:43.959 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:43.957+0000 7f3889549700 1 --2- 172.21.15.29:0/2476311633 >> [v2:172.21.15.29:6800/1563185523,v1:172.21.15.29:6801/1563185523] conn(0x7f387002fdb0 0x7f3870032270 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:41:43.959 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:43.957+0000 7f3889549700 1 --2- 172.21.15.29:0/2476311633 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3884070460 0x7f38840fc3b0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:41:43.959 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:43.957+0000 7f3889549700 1 -- 172.21.15.29:0/2476311633 >> 172.21.15.29:0/2476311633 conn(0x7f38840f63b0 msgr2=0x7f38840f7e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:41:43.959 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:43.958+0000 7f3889549700 1 -- 172.21.15.29:0/2476311633 shutdown_connections 2024-01-21T11:41:43.959 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:43.958+0000 7f3889549700 1 -- 172.21.15.29:0/2476311633 wait complete. 2024-01-21T11:41:44.275 INFO:teuthology.orchestra.run.smithi029.stderr:Deploying prometheus service with default placement... 2024-01-21T11:41:44.275 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e4084074-b851-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp77cpgrhz:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpea05xzin:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply prometheus 2024-01-21T11:41:44.279 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:43 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:41:42.546902+0000 mgr.smithi029.blhxxk (mgr.14116) 14 : audit [DBG] from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T11:41:44.279 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:43 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cephadm 2024-01-21T11:41:42.547545+0000 mgr.smithi029.blhxxk (mgr.14116) 15 : cephadm [INF] Saving service crash spec with placement * 2024-01-21T11:41:44.279 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:43 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:41:42.730608+0000 mgr.smithi029.blhxxk (mgr.14116) 16 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:41:44.279 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:43 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:41:43.444741+0000 mon.smithi029 (mon.0) 60 : audit [INF] from='client.? 172.21.15.29:0/2476311633' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-01-21T11:41:44.627 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:44.626+0000 7fc028a99700 1 Processor -- start 2024-01-21T11:41:44.628 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:44.626+0000 7fc028a99700 1 -- start start 2024-01-21T11:41:44.628 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:44.626+0000 7fc028a99700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0240703d0 0x7fc024100d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T11:41:44.628 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:44.626+0000 7fc028a99700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0240fc730 con 0x7fc0240703d0 2024-01-21T11:41:44.628 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:44.626+0000 7fc02259c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0240703d0 0x7fc024100d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:41:44.628 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:44.626+0000 7fc02259c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0240703d0 0x7fc024100d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33128/0 (socket says 172.21.15.29:33128) 2024-01-21T11:41:44.628 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:44.626+0000 7fc02259c700 1 -- 172.21.15.29:0/1480464725 learned_addr learned my addr 172.21.15.29:0/1480464725 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-21T11:41:44.628 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:44.627+0000 7fc02259c700 1 -- 172.21.15.29:0/1480464725 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc024101230 con 0x7fc0240703d0 2024-01-21T11:41:44.628 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:44.627+0000 7fc02259c700 1 --2- 172.21.15.29:0/1480464725 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0240703d0 0x7fc024100d40 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7fc01400b5f0 tx=0x7fc014006ee0).ready entity=mon.0 client_cookie=a8b03fdd73a388c0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:41:44.628 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:44.627+0000 7fc02159a700 1 -- 172.21.15.29:0/1480464725 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc01400d600 con 0x7fc0240703d0 2024-01-21T11:41:44.628 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:44.627+0000 7fc02159a700 1 -- 172.21.15.29:0/1480464725 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fc01400dbd0 con 0x7fc0240703d0 2024-01-21T11:41:44.628 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:44.627+0000 7fc02159a700 1 -- 172.21.15.29:0/1480464725 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc014011c00 con 0x7fc0240703d0 2024-01-21T11:41:44.632 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:44.628+0000 7fc028a99700 1 -- 172.21.15.29:0/1480464725 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0240703d0 msgr2=0x7fc024100d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:41:44.632 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:44.628+0000 7fc028a99700 1 --2- 172.21.15.29:0/1480464725 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0240703d0 0x7fc024100d40 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7fc01400b5f0 tx=0x7fc014006ee0).stop 2024-01-21T11:41:44.632 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:44.628+0000 7fc028a99700 1 -- 172.21.15.29:0/1480464725 shutdown_connections 2024-01-21T11:41:44.632 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:44.628+0000 7fc028a99700 1 --2- 172.21.15.29:0/1480464725 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0240703d0 0x7fc024100d40 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:41:44.632 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:44.628+0000 7fc028a99700 1 -- 172.21.15.29:0/1480464725 >> 172.21.15.29:0/1480464725 conn(0x7fc0240f63c0 msgr2=0x7fc0240f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:41:44.632 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:44.628+0000 7fc028a99700 1 -- 172.21.15.29:0/1480464725 shutdown_connections 2024-01-21T11:41:44.632 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:44.628+0000 7fc028a99700 1 -- 172.21.15.29:0/1480464725 wait complete. 2024-01-21T11:41:44.632 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:44.629+0000 7fc028a99700 1 Processor -- start 2024-01-21T11:41:44.632 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:44.629+0000 7fc028a99700 1 -- start start 2024-01-21T11:41:44.632 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:44.629+0000 7fc028a99700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0240703d0 0x7fc02410cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:41:44.632 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:44.629+0000 7fc028a99700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc02410d100 con 0x7fc0240703d0 2024-01-21T11:41:44.632 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:44.629+0000 7fc02259c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0240703d0 0x7fc02410cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:41:44.632 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:44.629+0000 7fc02259c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0240703d0 0x7fc02410cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33136/0 (socket says 172.21.15.29:33136) 2024-01-21T11:41:44.633 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:44.629+0000 7fc02259c700 1 -- 172.21.15.29:0/2047693856 learned_addr learned my addr 172.21.15.29:0/2047693856 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-21T11:41:44.633 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:44.629+0000 7fc02259c700 1 -- 172.21.15.29:0/2047693856 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc01400b200 con 0x7fc0240703d0 2024-01-21T11:41:44.633 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:44.630+0000 7fc02259c700 1 --2- 172.21.15.29:0/2047693856 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0240703d0 0x7fc02410cc10 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7fc01400e040 tx=0x7fc014006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:41:44.633 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:44.630+0000 7fc012ffd700 1 -- 172.21.15.29:0/2047693856 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc01400d9f0 con 0x7fc0240703d0 2024-01-21T11:41:44.633 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:44.630+0000 7fc012ffd700 1 -- 172.21.15.29:0/2047693856 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fc01401a9f0 con 0x7fc0240703d0 2024-01-21T11:41:44.633 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:44.630+0000 7fc028a99700 1 -- 172.21.15.29:0/2047693856 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc02410d300 con 0x7fc0240703d0 2024-01-21T11:41:44.633 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:44.630+0000 7fc012ffd700 1 -- 172.21.15.29:0/2047693856 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc01401b8c0 con 0x7fc0240703d0 2024-01-21T11:41:44.633 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:44.630+0000 7fc028a99700 1 -- 172.21.15.29:0/2047693856 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc02410d710 con 0x7fc0240703d0 2024-01-21T11:41:44.633 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:44.631+0000 7fc012ffd700 1 -- 172.21.15.29:0/2047693856 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 9) v1 ==== 36134+0+0 (secure 0 0 0) 0x7fc01400f070 con 0x7fc0240703d0 2024-01-21T11:41:44.633 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:44.631+0000 7fc012ffd700 1 --2- 172.21.15.29:0/2047693856 >> [v2:172.21.15.29:6800/1563185523,v1:172.21.15.29:6801/1563185523] conn(0x7fc00802fe00 0x7fc0080322c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:41:44.633 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:44.631+0000 7fc012ffd700 1 -- 172.21.15.29:0/2047693856 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fc014041380 con 0x7fc0240703d0 2024-01-21T11:41:44.633 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:44.631+0000 7fc021d9b700 1 -- 172.21.15.29:0/2047693856 >> [v2:172.21.15.29:6800/1563185523,v1:172.21.15.29:6801/1563185523] conn(0x7fc00802fe00 msgr2=0x7fc0080322c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/1563185523 2024-01-21T11:41:44.633 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:44.631+0000 7fc021d9b700 1 --2- 172.21.15.29:0/2047693856 >> [v2:172.21.15.29:6800/1563185523,v1:172.21.15.29:6801/1563185523] conn(0x7fc00802fe00 0x7fc0080322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-01-21T11:41:44.633 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:44.631+0000 7fc028a99700 1 -- 172.21.15.29:0/2047693856 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc024059100 con 0x7fc0240703d0 2024-01-21T11:41:44.635 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:44.636+0000 7fc012ffd700 1 -- 172.21.15.29:0/2047693856 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc014016070 con 0x7fc0240703d0 2024-01-21T11:41:44.823 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:44.823+0000 7fc028a99700 1 -- 172.21.15.29:0/2047693856 --> [v2:172.21.15.29:6800/1563185523,v1:172.21.15.29:6801/1563185523] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7fc024106a60 con 0x7fc00802fe00 2024-01-21T11:41:44.832 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:44.832+0000 7fc021d9b700 1 -- 172.21.15.29:0/2047693856 >> [v2:172.21.15.29:6800/1563185523,v1:172.21.15.29:6801/1563185523] conn(0x7fc00802fe00 msgr2=0x7fc0080322c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/1563185523 2024-01-21T11:41:44.832 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:44.832+0000 7fc021d9b700 1 --2- 172.21.15.29:0/2047693856 >> [v2:172.21.15.29:6800/1563185523,v1:172.21.15.29:6801/1563185523] conn(0x7fc00802fe00 0x7fc0080322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-01-21T11:41:45.233 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:45.233+0000 7fc021d9b700 1 -- 172.21.15.29:0/2047693856 >> [v2:172.21.15.29:6800/1563185523,v1:172.21.15.29:6801/1563185523] conn(0x7fc00802fe00 msgr2=0x7fc0080322c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/1563185523 2024-01-21T11:41:45.234 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:45.233+0000 7fc021d9b700 1 --2- 172.21.15.29:0/2047693856 >> [v2:172.21.15.29:6800/1563185523,v1:172.21.15.29:6801/1563185523] conn(0x7fc00802fe00 0x7fc0080322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2024-01-21T11:41:45.418 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:44 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:41:43.955401+0000 mon.smithi029 (mon.0) 61 : audit [INF] from='client.? 172.21.15.29:0/2476311633' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-01-21T11:41:45.418 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:44 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:41:43.955479+0000 mon.smithi029 (mon.0) 62 : cluster [DBG] mgrmap e9: smithi029.blhxxk(active, since 21s) 2024-01-21T11:41:46.034 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:46.034+0000 7fc021d9b700 1 -- 172.21.15.29:0/2047693856 >> [v2:172.21.15.29:6800/1563185523,v1:172.21.15.29:6801/1563185523] conn(0x7fc00802fe00 msgr2=0x7fc0080322c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/1563185523 2024-01-21T11:41:46.034 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:46.034+0000 7fc021d9b700 1 --2- 172.21.15.29:0/2047693856 >> [v2:172.21.15.29:6800/1563185523,v1:172.21.15.29:6801/1563185523] conn(0x7fc00802fe00 0x7fc0080322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2024-01-21T11:41:47.635 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:47.635+0000 7fc021d9b700 1 -- 172.21.15.29:0/2047693856 >> [v2:172.21.15.29:6800/1563185523,v1:172.21.15.29:6801/1563185523] conn(0x7fc00802fe00 msgr2=0x7fc0080322c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/1563185523 2024-01-21T11:41:47.635 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:47.635+0000 7fc021d9b700 1 --2- 172.21.15.29:0/2047693856 >> [v2:172.21.15.29:6800/1563185523,v1:172.21.15.29:6801/1563185523] conn(0x7fc00802fe00 0x7fc0080322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2024-01-21T11:41:49.212 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:49.210+0000 7fc012ffd700 1 -- 172.21.15.29:0/2047693856 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 10) v1 ==== 35984+0+0 (secure 0 0 0) 0x7fc0140257e0 con 0x7fc0240703d0 2024-01-21T11:41:49.212 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:49.210+0000 7fc012ffd700 1 -- 172.21.15.29:0/2047693856 >> [v2:172.21.15.29:6800/1563185523,v1:172.21.15.29:6801/1563185523] conn(0x7fc00802fe00 msgr2=0x7fc0080322c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-21T11:41:49.212 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:49.210+0000 7fc012ffd700 1 --2- 172.21.15.29:0/2047693856 >> [v2:172.21.15.29:6800/1563185523,v1:172.21.15.29:6801/1563185523] conn(0x7fc00802fe00 0x7fc0080322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-21T11:41:50.213 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:50.212+0000 7fc012ffd700 1 -- 172.21.15.29:0/2047693856 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mgrmap(e 11) v1 ==== 36054+0+0 (secure 0 0 0) 0x7fc014024ca0 con 0x7fc0240703d0 2024-01-21T11:41:50.213 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:50.212+0000 7fc012ffd700 1 --2- 172.21.15.29:0/2047693856 >> [v2:172.21.15.29:6800/3957292961,v1:172.21.15.29:6801/3957292961] conn(0x7fc008033c20 0x7fc008036010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:41:50.213 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:50.212+0000 7fc012ffd700 1 -- 172.21.15.29:0/2047693856 --> [v2:172.21.15.29:6800/3957292961,v1:172.21.15.29:6801/3957292961] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7fc024106a60 con 0x7fc008033c20 2024-01-21T11:41:50.213 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:50.212+0000 7fc021d9b700 1 --2- 172.21.15.29:0/2047693856 >> [v2:172.21.15.29:6800/3957292961,v1:172.21.15.29:6801/3957292961] conn(0x7fc008033c20 0x7fc008036010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:41:50.213 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:50.213+0000 7fc021d9b700 1 --2- 172.21.15.29:0/2047693856 >> [v2:172.21.15.29:6800/3957292961,v1:172.21.15.29:6801/3957292961] conn(0x7fc008033c20 0x7fc008036010 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fc018006fd0 tx=0x7fc018006d30).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:41:50.217 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:50.217+0000 7fc012ffd700 1 -- 172.21.15.29:0/2047693856 <== mgr.14144 v2:172.21.15.29:6800/3957292961 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7fc024106a60 con 0x7fc008033c20 2024-01-21T11:41:50.218 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: Scheduled prometheus update... 2024-01-21T11:41:50.220 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:50.219+0000 7fc028a99700 1 -- 172.21.15.29:0/2047693856 >> [v2:172.21.15.29:6800/3957292961,v1:172.21.15.29:6801/3957292961] conn(0x7fc008033c20 msgr2=0x7fc008036010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:41:50.221 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:50.219+0000 7fc028a99700 1 --2- 172.21.15.29:0/2047693856 >> [v2:172.21.15.29:6800/3957292961,v1:172.21.15.29:6801/3957292961] conn(0x7fc008033c20 0x7fc008036010 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fc018006fd0 tx=0x7fc018006d30).stop 2024-01-21T11:41:50.221 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:50.219+0000 7fc028a99700 1 -- 172.21.15.29:0/2047693856 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0240703d0 msgr2=0x7fc02410cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:41:50.221 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:50.219+0000 7fc028a99700 1 --2- 172.21.15.29:0/2047693856 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0240703d0 0x7fc02410cc10 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7fc01400e040 tx=0x7fc014006ee0).stop 2024-01-21T11:41:50.221 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:50.219+0000 7fc028a99700 1 -- 172.21.15.29:0/2047693856 shutdown_connections 2024-01-21T11:41:50.221 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:50.219+0000 7fc028a99700 1 --2- 172.21.15.29:0/2047693856 >> [v2:172.21.15.29:6800/3957292961,v1:172.21.15.29:6801/3957292961] conn(0x7fc008033c20 0x7fc008036010 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:41:50.221 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:50.219+0000 7fc028a99700 1 --2- 172.21.15.29:0/2047693856 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0240703d0 0x7fc02410cc10 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:41:50.221 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:50.220+0000 7fc028a99700 1 --2- 172.21.15.29:0/2047693856 >> [v2:172.21.15.29:6800/1563185523,v1:172.21.15.29:6801/1563185523] conn(0x7fc00802fe00 0x7fc0080322c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-21T11:41:50.221 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:50.220+0000 7fc028a99700 1 -- 172.21.15.29:0/2047693856 >> 172.21.15.29:0/2047693856 conn(0x7fc0240f63c0 msgr2=0x7fc0240f7ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:41:50.221 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:50.220+0000 7fc028a99700 1 -- 172.21.15.29:0/2047693856 shutdown_connections 2024-01-21T11:41:50.221 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:50.220+0000 7fc028a99700 1 -- 172.21.15.29:0/2047693856 wait complete. 2024-01-21T11:41:50.418 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:50 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:41:49.159605+0000 mon.smithi029 (mon.0) 63 : cluster [INF] Active manager daemon smithi029.blhxxk restarted 2024-01-21T11:41:50.419 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:50 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:41:49.160013+0000 mon.smithi029 (mon.0) 64 : cluster [INF] Activating manager daemon smithi029.blhxxk 2024-01-21T11:41:50.419 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:50 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:41:49.161549+0000 mon.smithi029 (mon.0) 65 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2024-01-21T11:41:50.419 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:50 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:41:49.212439+0000 mon.smithi029 (mon.0) 66 : cluster [DBG] mgrmap e10: smithi029.blhxxk(active, starting, since 0.0525103s) 2024-01-21T11:41:50.419 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:50 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:41:49.213602+0000 mon.smithi029 (mon.0) 67 : audit [DBG] from='mgr.14144 172.21.15.29:0/3161775375' entity='mgr.smithi029.blhxxk' cmd=[{"prefix": "mon metadata", "id": "smithi029"}]: dispatch 2024-01-21T11:41:50.419 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:50 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:41:49.214155+0000 mon.smithi029 (mon.0) 68 : audit [DBG] from='mgr.14144 172.21.15.29:0/3161775375' entity='mgr.smithi029.blhxxk' cmd=[{"prefix": "mgr metadata", "who": "smithi029.blhxxk", "id": "smithi029.blhxxk"}]: dispatch 2024-01-21T11:41:50.419 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:50 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:41:49.214965+0000 mon.smithi029 (mon.0) 69 : audit [DBG] from='mgr.14144 172.21.15.29:0/3161775375' entity='mgr.smithi029.blhxxk' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-21T11:41:50.419 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:50 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:41:49.215128+0000 mon.smithi029 (mon.0) 70 : audit [DBG] from='mgr.14144 172.21.15.29:0/3161775375' entity='mgr.smithi029.blhxxk' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-21T11:41:50.419 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:50 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:41:49.215289+0000 mon.smithi029 (mon.0) 71 : audit [DBG] from='mgr.14144 172.21.15.29:0/3161775375' entity='mgr.smithi029.blhxxk' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-21T11:41:50.419 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:50 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:41:49.217164+0000 mon.smithi029 (mon.0) 72 : cluster [INF] Manager daemon smithi029.blhxxk is now available 2024-01-21T11:41:50.419 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:50 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:41:49.320822+0000 mon.smithi029 (mon.0) 73 : audit [DBG] from='mgr.14144 172.21.15.29:0/3161775375' entity='mgr.smithi029.blhxxk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-21T11:41:50.419 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:50 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:41:49.322325+0000 mon.smithi029 (mon.0) 74 : audit [DBG] from='mgr.14144 172.21.15.29:0/3161775375' entity='mgr.smithi029.blhxxk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-21T11:41:50.419 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:50 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:41:49.337095+0000 mon.smithi029 (mon.0) 75 : audit [INF] from='mgr.14144 172.21.15.29:0/3161775375' entity='mgr.smithi029.blhxxk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi029.blhxxk/mirror_snapshot_schedule"}]: dispatch 2024-01-21T11:41:50.420 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:50 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:41:49.339778+0000 mon.smithi029 (mon.0) 76 : audit [INF] from='mgr.14144 172.21.15.29:0/3161775375' entity='mgr.smithi029.blhxxk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi029.blhxxk/trash_purge_schedule"}]: dispatch 2024-01-21T11:41:50.592 INFO:teuthology.orchestra.run.smithi029.stderr:Deploying grafana service with default placement... 2024-01-21T11:41:50.592 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e4084074-b851-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp77cpgrhz:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpea05xzin:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply grafana 2024-01-21T11:41:50.945 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:50.941+0000 7f33f6ea5700 1 Processor -- start 2024-01-21T11:41:50.945 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:50.941+0000 7f33f6ea5700 1 -- start start 2024-01-21T11:41:50.945 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:50.941+0000 7f33f6ea5700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33f00704a0 0x7f33f00fe750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T11:41:50.945 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:50.941+0000 7f33f6ea5700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33f00fec40 con 0x7f33f00704a0 2024-01-21T11:41:50.945 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:50.941+0000 7f33f5ea3700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33f00704a0 0x7f33f00fe750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:41:50.945 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:50.941+0000 7f33f5ea3700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33f00704a0 0x7f33f00fe750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33180/0 (socket says 172.21.15.29:33180) 2024-01-21T11:41:50.945 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:50.941+0000 7f33f5ea3700 1 -- 172.21.15.29:0/375425369 learned_addr learned my addr 172.21.15.29:0/375425369 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-21T11:41:50.945 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:50.942+0000 7f33f5ea3700 1 -- 172.21.15.29:0/375425369 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33f00fed80 con 0x7f33f00704a0 2024-01-21T11:41:50.945 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:50.942+0000 7f33f5ea3700 1 --2- 172.21.15.29:0/375425369 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33f00704a0 0x7f33f00fe750 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f33ec00b5f0 tx=0x7f33ec006ee0).ready entity=mon.0 client_cookie=9c78c5d7082b736 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:41:50.945 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:50.942+0000 7f33f4ea1700 1 -- 172.21.15.29:0/375425369 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f33ec00d600 con 0x7f33f00704a0 2024-01-21T11:41:50.945 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:50.942+0000 7f33f4ea1700 1 -- 172.21.15.29:0/375425369 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f33ec00dbd0 con 0x7f33f00704a0 2024-01-21T11:41:50.945 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:50.942+0000 7f33f4ea1700 1 -- 172.21.15.29:0/375425369 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f33ec011c00 con 0x7f33f00704a0 2024-01-21T11:41:50.945 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:50.942+0000 7f33f6ea5700 1 -- 172.21.15.29:0/375425369 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33f00704a0 msgr2=0x7f33f00fe750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:41:50.946 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:50.942+0000 7f33f6ea5700 1 --2- 172.21.15.29:0/375425369 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33f00704a0 0x7f33f00fe750 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f33ec00b5f0 tx=0x7f33ec006ee0).stop 2024-01-21T11:41:50.946 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:50.943+0000 7f33f6ea5700 1 -- 172.21.15.29:0/375425369 shutdown_connections 2024-01-21T11:41:50.946 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:50.943+0000 7f33f6ea5700 1 --2- 172.21.15.29:0/375425369 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33f00704a0 0x7f33f00fe750 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:41:50.946 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:50.943+0000 7f33f6ea5700 1 -- 172.21.15.29:0/375425369 >> 172.21.15.29:0/375425369 conn(0x7f33f00f6250 msgr2=0x7f33f00f8670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:41:50.946 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:50.943+0000 7f33f6ea5700 1 -- 172.21.15.29:0/375425369 shutdown_connections 2024-01-21T11:41:50.946 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:50.943+0000 7f33f6ea5700 1 -- 172.21.15.29:0/375425369 wait complete. 2024-01-21T11:41:50.946 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:50.943+0000 7f33f6ea5700 1 Processor -- start 2024-01-21T11:41:50.946 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:50.943+0000 7f33f6ea5700 1 -- start start 2024-01-21T11:41:50.946 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:50.943+0000 7f33f6ea5700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33f00704a0 0x7f33f010a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:41:50.946 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:50.943+0000 7f33f6ea5700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33f010ada0 con 0x7f33f00704a0 2024-01-21T11:41:50.946 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:50.944+0000 7f33f5ea3700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33f00704a0 0x7f33f010a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:41:50.946 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:50.944+0000 7f33f5ea3700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33f00704a0 0x7f33f010a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33188/0 (socket says 172.21.15.29:33188) 2024-01-21T11:41:50.946 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:50.944+0000 7f33f5ea3700 1 -- 172.21.15.29:0/745995056 learned_addr learned my addr 172.21.15.29:0/745995056 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-21T11:41:50.946 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:50.944+0000 7f33f5ea3700 1 -- 172.21.15.29:0/745995056 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33ec00b200 con 0x7f33f00704a0 2024-01-21T11:41:50.946 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:50.944+0000 7f33f5ea3700 1 --2- 172.21.15.29:0/745995056 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33f00704a0 0x7f33f010a8b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f33ec00b1d0 tx=0x7f33ec006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:41:50.946 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:50.944+0000 7f33de7fc700 1 -- 172.21.15.29:0/745995056 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f33ec011a20 con 0x7f33f00704a0 2024-01-21T11:41:50.947 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:50.944+0000 7f33f6ea5700 1 -- 172.21.15.29:0/745995056 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33f010afa0 con 0x7f33f00704a0 2024-01-21T11:41:50.947 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:50.944+0000 7f33de7fc700 1 -- 172.21.15.29:0/745995056 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f33ec01b880 con 0x7f33f00704a0 2024-01-21T11:41:50.947 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:50.944+0000 7f33de7fc700 1 -- 172.21.15.29:0/745995056 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f33ec01a700 con 0x7f33f00704a0 2024-01-21T11:41:50.947 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:50.944+0000 7f33f6ea5700 1 -- 172.21.15.29:0/745995056 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33f010b3b0 con 0x7f33f00704a0 2024-01-21T11:41:50.947 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:50.945+0000 7f33de7fc700 1 -- 172.21.15.29:0/745995056 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 11) v1 ==== 36054+0+0 (secure 0 0 0) 0x7f33ec00f070 con 0x7f33f00704a0 2024-01-21T11:41:50.947 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:50.945+0000 7f33de7fc700 1 --2- 172.21.15.29:0/745995056 >> [v2:172.21.15.29:6800/3957292961,v1:172.21.15.29:6801/3957292961] conn(0x7f33d802fd70 0x7f33d8032230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:41:50.948 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:50.945+0000 7f33de7fc700 1 -- 172.21.15.29:0/745995056 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f33ec041030 con 0x7f33f00704a0 2024-01-21T11:41:50.948 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:50.945+0000 7f33f56a2700 1 --2- 172.21.15.29:0/745995056 >> [v2:172.21.15.29:6800/3957292961,v1:172.21.15.29:6801/3957292961] conn(0x7f33d802fd70 0x7f33d8032230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:41:50.948 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:50.946+0000 7f33f56a2700 1 --2- 172.21.15.29:0/745995056 >> [v2:172.21.15.29:6800/3957292961,v1:172.21.15.29:6801/3957292961] conn(0x7f33d802fd70 0x7f33d8032230 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f33e00096a0 tx=0x7f33e0006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:41:50.948 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:50.946+0000 7f33f6ea5700 1 -- 172.21.15.29:0/745995056 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33f0059100 con 0x7f33f00704a0 2024-01-21T11:41:50.950 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:50.950+0000 7f33de7fc700 1 -- 172.21.15.29:0/745995056 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f33ec016080 con 0x7f33f00704a0 2024-01-21T11:41:51.136 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:51.135+0000 7f33f6ea5700 1 -- 172.21.15.29:0/745995056 --> [v2:172.21.15.29:6800/3957292961,v1:172.21.15.29:6801/3957292961] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7f33f0104760 con 0x7f33d802fd70 2024-01-21T11:41:51.139 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:51.139+0000 7f33de7fc700 1 -- 172.21.15.29:0/745995056 <== mgr.14144 v2:172.21.15.29:6800/3957292961 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f33f0104760 con 0x7f33d802fd70 2024-01-21T11:41:51.139 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: Scheduled grafana update... 2024-01-21T11:41:51.142 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:51.141+0000 7f33f6ea5700 1 -- 172.21.15.29:0/745995056 >> [v2:172.21.15.29:6800/3957292961,v1:172.21.15.29:6801/3957292961] conn(0x7f33d802fd70 msgr2=0x7f33d8032230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:41:51.142 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:51.141+0000 7f33f6ea5700 1 --2- 172.21.15.29:0/745995056 >> [v2:172.21.15.29:6800/3957292961,v1:172.21.15.29:6801/3957292961] conn(0x7f33d802fd70 0x7f33d8032230 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f33e00096a0 tx=0x7f33e0006b40).stop 2024-01-21T11:41:51.142 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:51.141+0000 7f33f6ea5700 1 -- 172.21.15.29:0/745995056 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33f00704a0 msgr2=0x7f33f010a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:41:51.142 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:51.141+0000 7f33f6ea5700 1 --2- 172.21.15.29:0/745995056 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33f00704a0 0x7f33f010a8b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f33ec00b1d0 tx=0x7f33ec006ee0).stop 2024-01-21T11:41:51.142 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:51.141+0000 7f33f6ea5700 1 -- 172.21.15.29:0/745995056 shutdown_connections 2024-01-21T11:41:51.142 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:51.141+0000 7f33f6ea5700 1 --2- 172.21.15.29:0/745995056 >> [v2:172.21.15.29:6800/3957292961,v1:172.21.15.29:6801/3957292961] conn(0x7f33d802fd70 0x7f33d8032230 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:41:51.142 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:51.141+0000 7f33f6ea5700 1 --2- 172.21.15.29:0/745995056 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33f00704a0 0x7f33f010a8b0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:41:51.142 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:51.141+0000 7f33f6ea5700 1 -- 172.21.15.29:0/745995056 >> 172.21.15.29:0/745995056 conn(0x7f33f00f6250 msgr2=0x7f33f00f7d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:41:51.142 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:51.141+0000 7f33f6ea5700 1 -- 172.21.15.29:0/745995056 shutdown_connections 2024-01-21T11:41:51.142 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:51.141+0000 7f33f6ea5700 1 -- 172.21.15.29:0/745995056 wait complete. 2024-01-21T11:41:51.419 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:51 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:41:50.214863+0000 mon.smithi029 (mon.0) 77 : cluster [DBG] mgrmap e11: smithi029.blhxxk(active, since 1.05493s) 2024-01-21T11:41:51.419 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:51 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cephadm 2024-01-21T11:41:50.216035+0000 mgr.smithi029.blhxxk (mgr.14144) 2 : cephadm [INF] Saving service prometheus spec with placement count:1 2024-01-21T11:41:51.419 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:51 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:41:50.218175+0000 mon.smithi029 (mon.0) 78 : audit [INF] from='mgr.14144 172.21.15.29:0/3161775375' entity='mgr.smithi029.blhxxk' 2024-01-21T11:41:51.419 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:51 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:41:51.140070+0000 mon.smithi029 (mon.0) 79 : audit [INF] from='mgr.14144 172.21.15.29:0/3161775375' entity='mgr.smithi029.blhxxk' 2024-01-21T11:41:51.483 INFO:teuthology.orchestra.run.smithi029.stderr:Deploying node-exporter service with default placement... 2024-01-21T11:41:51.483 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e4084074-b851-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp77cpgrhz:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpea05xzin:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply node-exporter 2024-01-21T11:41:51.848 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:51.845+0000 7f674206d700 1 Processor -- start 2024-01-21T11:41:51.848 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:51.845+0000 7f674206d700 1 -- start start 2024-01-21T11:41:51.848 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:51.846+0000 7f674206d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f673c0703b0 0x7f673c100d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T11:41:51.848 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:51.846+0000 7f674206d700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f673c0fc750 con 0x7f673c0703b0 2024-01-21T11:41:51.848 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:51.846+0000 7f673b7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f673c0703b0 0x7f673c100d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:41:51.848 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:51.846+0000 7f673b7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f673c0703b0 0x7f673c100d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33200/0 (socket says 172.21.15.29:33200) 2024-01-21T11:41:51.848 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:51.846+0000 7f673b7fe700 1 -- 172.21.15.29:0/3117770301 learned_addr learned my addr 172.21.15.29:0/3117770301 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-21T11:41:51.848 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:51.846+0000 7f673b7fe700 1 -- 172.21.15.29:0/3117770301 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f673c101250 con 0x7f673c0703b0 2024-01-21T11:41:51.848 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:51.846+0000 7f673b7fe700 1 --2- 172.21.15.29:0/3117770301 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f673c0703b0 0x7f673c100d60 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f672c00b5f0 tx=0x7f672c006ee0).ready entity=mon.0 client_cookie=e08ed2997f112559 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:41:51.848 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:51.846+0000 7f673a7fc700 1 -- 172.21.15.29:0/3117770301 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f672c00d600 con 0x7f673c0703b0 2024-01-21T11:41:51.848 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:51.846+0000 7f673a7fc700 1 -- 172.21.15.29:0/3117770301 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f672c00dbd0 con 0x7f673c0703b0 2024-01-21T11:41:51.848 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:51.846+0000 7f673a7fc700 1 -- 172.21.15.29:0/3117770301 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f672c011c00 con 0x7f673c0703b0 2024-01-21T11:41:51.849 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:51.847+0000 7f674206d700 1 -- 172.21.15.29:0/3117770301 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f673c0703b0 msgr2=0x7f673c100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:41:51.849 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:51.847+0000 7f674206d700 1 --2- 172.21.15.29:0/3117770301 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f673c0703b0 0x7f673c100d60 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f672c00b5f0 tx=0x7f672c006ee0).stop 2024-01-21T11:41:51.850 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:51.847+0000 7f674206d700 1 -- 172.21.15.29:0/3117770301 shutdown_connections 2024-01-21T11:41:51.850 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:51.847+0000 7f674206d700 1 --2- 172.21.15.29:0/3117770301 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f673c0703b0 0x7f673c100d60 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:41:51.850 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:51.847+0000 7f674206d700 1 -- 172.21.15.29:0/3117770301 >> 172.21.15.29:0/3117770301 conn(0x7f673c0f63c0 msgr2=0x7f673c0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:41:51.850 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:51.847+0000 7f674206d700 1 -- 172.21.15.29:0/3117770301 shutdown_connections 2024-01-21T11:41:51.850 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:51.847+0000 7f674206d700 1 -- 172.21.15.29:0/3117770301 wait complete. 2024-01-21T11:41:51.850 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:51.848+0000 7f674206d700 1 Processor -- start 2024-01-21T11:41:51.850 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:51.848+0000 7f674206d700 1 -- start start 2024-01-21T11:41:51.850 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:51.848+0000 7f674206d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f673c0703b0 0x7f673c10ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:41:51.850 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:51.848+0000 7f674206d700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f673c10d1e0 con 0x7f673c0703b0 2024-01-21T11:41:51.851 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:51.848+0000 7f673b7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f673c0703b0 0x7f673c10ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:41:51.851 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:51.848+0000 7f673b7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f673c0703b0 0x7f673c10ccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33212/0 (socket says 172.21.15.29:33212) 2024-01-21T11:41:51.851 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:51.848+0000 7f673b7fe700 1 -- 172.21.15.29:0/1181963758 learned_addr learned my addr 172.21.15.29:0/1181963758 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-21T11:41:51.852 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:51.849+0000 7f673b7fe700 1 -- 172.21.15.29:0/1181963758 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f672c00b200 con 0x7f673c0703b0 2024-01-21T11:41:51.852 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:51.849+0000 7f673b7fe700 1 --2- 172.21.15.29:0/1181963758 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f673c0703b0 0x7f673c10ccf0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f672c00e040 tx=0x7f672c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:41:51.852 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:51.849+0000 7f6733fff700 1 -- 172.21.15.29:0/1181963758 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f672c00d9f0 con 0x7f673c0703b0 2024-01-21T11:41:51.852 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:51.849+0000 7f6733fff700 1 -- 172.21.15.29:0/1181963758 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f672c01a9f0 con 0x7f673c0703b0 2024-01-21T11:41:51.852 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:51.849+0000 7f6733fff700 1 -- 172.21.15.29:0/1181963758 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f672c01b8c0 con 0x7f673c0703b0 2024-01-21T11:41:51.852 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:51.849+0000 7f674206d700 1 -- 172.21.15.29:0/1181963758 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f673c10d3e0 con 0x7f673c0703b0 2024-01-21T11:41:51.852 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:51.849+0000 7f674206d700 1 -- 172.21.15.29:0/1181963758 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f673c10d7f0 con 0x7f673c0703b0 2024-01-21T11:41:51.852 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:51.850+0000 7f6733fff700 1 -- 172.21.15.29:0/1181963758 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 11) v1 ==== 36054+0+0 (secure 0 0 0) 0x7f672c00f070 con 0x7f673c0703b0 2024-01-21T11:41:51.852 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:51.850+0000 7f6733fff700 1 --2- 172.21.15.29:0/1181963758 >> [v2:172.21.15.29:6800/3957292961,v1:172.21.15.29:6801/3957292961] conn(0x7f671c02fd70 0x7f671c032230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:41:51.852 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:51.850+0000 7f6733fff700 1 -- 172.21.15.29:0/1181963758 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f672c0412a0 con 0x7f673c0703b0 2024-01-21T11:41:51.852 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:51.850+0000 7f673affd700 1 --2- 172.21.15.29:0/1181963758 >> [v2:172.21.15.29:6800/3957292961,v1:172.21.15.29:6801/3957292961] conn(0x7f671c02fd70 0x7f671c032230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:41:51.852 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:51.850+0000 7f673affd700 1 --2- 172.21.15.29:0/1181963758 >> [v2:172.21.15.29:6800/3957292961,v1:172.21.15.29:6801/3957292961] conn(0x7f671c02fd70 0x7f671c032230 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f67240096a0 tx=0x7f6724006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:41:51.852 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:51.850+0000 7f674206d700 1 -- 172.21.15.29:0/1181963758 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f673c101db0 con 0x7f673c0703b0 2024-01-21T11:41:51.855 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:51.855+0000 7f6733fff700 1 -- 172.21.15.29:0/1181963758 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f672c016070 con 0x7f673c0703b0 2024-01-21T11:41:52.040 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:52.040+0000 7f674206d700 1 -- 172.21.15.29:0/1181963758 --> [v2:172.21.15.29:6800/3957292961,v1:172.21.15.29:6801/3957292961] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f673c101f70 con 0x7f671c02fd70 2024-01-21T11:41:52.044 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:52.044+0000 7f6733fff700 1 -- 172.21.15.29:0/1181963758 <== mgr.14144 v2:172.21.15.29:6800/3957292961 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f673c101f70 con 0x7f671c02fd70 2024-01-21T11:41:52.044 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: Scheduled node-exporter update... 2024-01-21T11:41:52.046 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:52.045+0000 7f674206d700 1 -- 172.21.15.29:0/1181963758 >> [v2:172.21.15.29:6800/3957292961,v1:172.21.15.29:6801/3957292961] conn(0x7f671c02fd70 msgr2=0x7f671c032230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:41:52.046 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:52.045+0000 7f674206d700 1 --2- 172.21.15.29:0/1181963758 >> [v2:172.21.15.29:6800/3957292961,v1:172.21.15.29:6801/3957292961] conn(0x7f671c02fd70 0x7f671c032230 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f67240096a0 tx=0x7f6724006b40).stop 2024-01-21T11:41:52.046 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:52.045+0000 7f674206d700 1 -- 172.21.15.29:0/1181963758 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f673c0703b0 msgr2=0x7f673c10ccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:41:52.046 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:52.045+0000 7f674206d700 1 --2- 172.21.15.29:0/1181963758 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f673c0703b0 0x7f673c10ccf0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f672c00e040 tx=0x7f672c006ee0).stop 2024-01-21T11:41:52.047 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:52.045+0000 7f674206d700 1 -- 172.21.15.29:0/1181963758 shutdown_connections 2024-01-21T11:41:52.047 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:52.045+0000 7f674206d700 1 --2- 172.21.15.29:0/1181963758 >> [v2:172.21.15.29:6800/3957292961,v1:172.21.15.29:6801/3957292961] conn(0x7f671c02fd70 0x7f671c032230 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:41:52.047 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:52.045+0000 7f674206d700 1 --2- 172.21.15.29:0/1181963758 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f673c0703b0 0x7f673c10ccf0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:41:52.047 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:52.045+0000 7f674206d700 1 -- 172.21.15.29:0/1181963758 >> 172.21.15.29:0/1181963758 conn(0x7f673c0f63c0 msgr2=0x7f673c0f7f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:41:52.047 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:52.046+0000 7f674206d700 1 -- 172.21.15.29:0/1181963758 shutdown_connections 2024-01-21T11:41:52.047 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:52.046+0000 7f674206d700 1 -- 172.21.15.29:0/1181963758 wait complete. 2024-01-21T11:41:52.327 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:52 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:41:51.137408+0000 mgr.smithi029.blhxxk (mgr.14144) 3 : audit [DBG] from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T11:41:52.327 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:52 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cephadm 2024-01-21T11:41:51.137926+0000 mgr.smithi029.blhxxk (mgr.14144) 4 : cephadm [INF] Saving service grafana spec with placement count:1 2024-01-21T11:41:52.327 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:52 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:41:52.044599+0000 mon.smithi029 (mon.0) 80 : audit [INF] from='mgr.14144 172.21.15.29:0/3161775375' entity='mgr.smithi029.blhxxk' 2024-01-21T11:41:52.327 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:52 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:41:52.141480+0000 mon.smithi029 (mon.0) 81 : cluster [DBG] mgrmap e12: smithi029.blhxxk(active, since 2s) 2024-01-21T11:41:52.400 INFO:teuthology.orchestra.run.smithi029.stderr:Deploying alertmanager service with default placement... 2024-01-21T11:41:52.400 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e4084074-b851-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp77cpgrhz:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpea05xzin:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply alertmanager 2024-01-21T11:41:52.751 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:52.750+0000 7efdd1657700 1 Processor -- start 2024-01-21T11:41:52.751 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:52.750+0000 7efdd1657700 1 -- start start 2024-01-21T11:41:52.751 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:52.750+0000 7efdd1657700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efdcc0fa9d0 0x7efdcc0fadf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T11:41:52.751 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:52.750+0000 7efdd1657700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdcc0fb2e0 con 0x7efdcc0fa9d0 2024-01-21T11:41:52.751 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:52.750+0000 7efdcbfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efdcc0fa9d0 0x7efdcc0fadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:41:52.751 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:52.750+0000 7efdcbfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efdcc0fa9d0 0x7efdcc0fadf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59860/0 (socket says 172.21.15.29:59860) 2024-01-21T11:41:52.752 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:52.750+0000 7efdcbfff700 1 -- 172.21.15.29:0/4294201836 learned_addr learned my addr 172.21.15.29:0/4294201836 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-21T11:41:52.752 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:52.751+0000 7efdcbfff700 1 -- 172.21.15.29:0/4294201836 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdcc0fb420 con 0x7efdcc0fa9d0 2024-01-21T11:41:52.752 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:52.751+0000 7efdcbfff700 1 --2- 172.21.15.29:0/4294201836 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efdcc0fa9d0 0x7efdcc0fadf0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7efdbc00b5f0 tx=0x7efdbc006ee0).ready entity=mon.0 client_cookie=57dd22380e6fece9 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:41:52.752 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:52.751+0000 7efdcaffd700 1 -- 172.21.15.29:0/4294201836 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efdbc00d600 con 0x7efdcc0fa9d0 2024-01-21T11:41:52.754 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:52.751+0000 7efdcaffd700 1 -- 172.21.15.29:0/4294201836 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7efdbc00dbd0 con 0x7efdcc0fa9d0 2024-01-21T11:41:52.754 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:52.751+0000 7efdcaffd700 1 -- 172.21.15.29:0/4294201836 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efdbc011c00 con 0x7efdcc0fa9d0 2024-01-21T11:41:52.754 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:52.752+0000 7efdd1657700 1 -- 172.21.15.29:0/4294201836 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efdcc0fa9d0 msgr2=0x7efdcc0fadf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:41:52.754 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:52.752+0000 7efdd1657700 1 --2- 172.21.15.29:0/4294201836 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efdcc0fa9d0 0x7efdcc0fadf0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7efdbc00b5f0 tx=0x7efdbc006ee0).stop 2024-01-21T11:41:52.754 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:52.752+0000 7efdd1657700 1 -- 172.21.15.29:0/4294201836 shutdown_connections 2024-01-21T11:41:52.754 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:52.752+0000 7efdd1657700 1 --2- 172.21.15.29:0/4294201836 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efdcc0fa9d0 0x7efdcc0fadf0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:41:52.755 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:52.752+0000 7efdd1657700 1 -- 172.21.15.29:0/4294201836 >> 172.21.15.29:0/4294201836 conn(0x7efdcc0f6210 msgr2=0x7efdcc0f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:41:52.755 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:52.752+0000 7efdd1657700 1 -- 172.21.15.29:0/4294201836 shutdown_connections 2024-01-21T11:41:52.755 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:52.752+0000 7efdd1657700 1 -- 172.21.15.29:0/4294201836 wait complete. 2024-01-21T11:41:52.755 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:52.752+0000 7efdd1657700 1 Processor -- start 2024-01-21T11:41:52.755 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:52.753+0000 7efdd1657700 1 -- start start 2024-01-21T11:41:52.755 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:52.753+0000 7efdd1657700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efdcc0fa9d0 0x7efdcc109700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:41:52.755 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:52.753+0000 7efdd1657700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdcc109bf0 con 0x7efdcc0fa9d0 2024-01-21T11:41:52.755 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:52.753+0000 7efdcbfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efdcc0fa9d0 0x7efdcc109700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:41:52.755 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:52.753+0000 7efdcbfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efdcc0fa9d0 0x7efdcc109700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59874/0 (socket says 172.21.15.29:59874) 2024-01-21T11:41:52.755 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:52.753+0000 7efdcbfff700 1 -- 172.21.15.29:0/138462013 learned_addr learned my addr 172.21.15.29:0/138462013 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-21T11:41:52.755 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:52.753+0000 7efdcbfff700 1 -- 172.21.15.29:0/138462013 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdbc00b200 con 0x7efdcc0fa9d0 2024-01-21T11:41:52.755 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:52.753+0000 7efdcbfff700 1 --2- 172.21.15.29:0/138462013 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efdcc0fa9d0 0x7efdcc109700 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7efdbc00b1d0 tx=0x7efdbc006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:41:52.755 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:52.753+0000 7efdc8ff9700 1 -- 172.21.15.29:0/138462013 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efdbc01b410 con 0x7efdcc0fa9d0 2024-01-21T11:41:52.755 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:52.753+0000 7efdc8ff9700 1 -- 172.21.15.29:0/138462013 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7efdbc01b9e0 con 0x7efdcc0fa9d0 2024-01-21T11:41:52.755 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:52.753+0000 7efdc8ff9700 1 -- 172.21.15.29:0/138462013 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efdbc01a840 con 0x7efdcc0fa9d0 2024-01-21T11:41:52.756 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:52.754+0000 7efdd1657700 1 -- 172.21.15.29:0/138462013 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdcc109df0 con 0x7efdcc0fa9d0 2024-01-21T11:41:52.756 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:52.754+0000 7efdd1657700 1 -- 172.21.15.29:0/138462013 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdcc10a200 con 0x7efdcc0fa9d0 2024-01-21T11:41:52.756 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:52.754+0000 7efdc8ff9700 1 -- 172.21.15.29:0/138462013 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7efdbc00f070 con 0x7efdcc0fa9d0 2024-01-21T11:41:52.756 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:52.755+0000 7efdd1657700 1 -- 172.21.15.29:0/138462013 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdcc045ad0 con 0x7efdcc0fa9d0 2024-01-21T11:41:52.756 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:52.755+0000 7efdc8ff9700 1 --2- 172.21.15.29:0/138462013 >> [v2:172.21.15.29:6800/3957292961,v1:172.21.15.29:6801/3957292961] conn(0x7efdb802feb0 0x7efdb8032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:41:52.756 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:52.755+0000 7efdc8ff9700 1 -- 172.21.15.29:0/138462013 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7efdbc041350 con 0x7efdcc0fa9d0 2024-01-21T11:41:52.756 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:52.755+0000 7efdcb7fe700 1 --2- 172.21.15.29:0/138462013 >> [v2:172.21.15.29:6800/3957292961,v1:172.21.15.29:6801/3957292961] conn(0x7efdb802feb0 0x7efdb8032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:41:52.757 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:52.755+0000 7efdcb7fe700 1 --2- 172.21.15.29:0/138462013 >> [v2:172.21.15.29:6800/3957292961,v1:172.21.15.29:6801/3957292961] conn(0x7efdb802feb0 0x7efdb8032370 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7efdc00096a0 tx=0x7efdc0006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:41:52.759 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:52.759+0000 7efdc8ff9700 1 -- 172.21.15.29:0/138462013 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efdbc0224f0 con 0x7efdcc0fa9d0 2024-01-21T11:41:52.945 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:52.945+0000 7efdd1657700 1 -- 172.21.15.29:0/138462013 --> [v2:172.21.15.29:6800/3957292961,v1:172.21.15.29:6801/3957292961] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7efdcc10aae0 con 0x7efdb802feb0 2024-01-21T11:41:52.949 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:52.949+0000 7efdc8ff9700 1 -- 172.21.15.29:0/138462013 <== mgr.14144 v2:172.21.15.29:6800/3957292961 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7efdcc10aae0 con 0x7efdb802feb0 2024-01-21T11:41:52.949 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: Scheduled alertmanager update... 2024-01-21T11:41:52.951 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:52.951+0000 7efdd1657700 1 -- 172.21.15.29:0/138462013 >> [v2:172.21.15.29:6800/3957292961,v1:172.21.15.29:6801/3957292961] conn(0x7efdb802feb0 msgr2=0x7efdb8032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:41:52.952 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:52.951+0000 7efdd1657700 1 --2- 172.21.15.29:0/138462013 >> [v2:172.21.15.29:6800/3957292961,v1:172.21.15.29:6801/3957292961] conn(0x7efdb802feb0 0x7efdb8032370 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7efdc00096a0 tx=0x7efdc0006b40).stop 2024-01-21T11:41:52.952 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:52.951+0000 7efdd1657700 1 -- 172.21.15.29:0/138462013 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efdcc0fa9d0 msgr2=0x7efdcc109700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:41:52.952 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:52.951+0000 7efdd1657700 1 --2- 172.21.15.29:0/138462013 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efdcc0fa9d0 0x7efdcc109700 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7efdbc00b1d0 tx=0x7efdbc006ee0).stop 2024-01-21T11:41:52.952 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:52.951+0000 7efdd1657700 1 -- 172.21.15.29:0/138462013 shutdown_connections 2024-01-21T11:41:52.952 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:52.951+0000 7efdd1657700 1 --2- 172.21.15.29:0/138462013 >> [v2:172.21.15.29:6800/3957292961,v1:172.21.15.29:6801/3957292961] conn(0x7efdb802feb0 0x7efdb8032370 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:41:52.952 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:52.951+0000 7efdd1657700 1 --2- 172.21.15.29:0/138462013 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efdcc0fa9d0 0x7efdcc109700 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:41:52.952 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:52.951+0000 7efdd1657700 1 -- 172.21.15.29:0/138462013 >> 172.21.15.29:0/138462013 conn(0x7efdcc0f6210 msgr2=0x7efdcc0f7cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:41:52.952 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:52.951+0000 7efdd1657700 1 -- 172.21.15.29:0/138462013 shutdown_connections 2024-01-21T11:41:52.952 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:52.951+0000 7efdd1657700 1 -- 172.21.15.29:0/138462013 wait complete. 2024-01-21T11:41:53.291 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e4084074-b851-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp77cpgrhz:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpea05xzin:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/cephadm/container_init True --force 2024-01-21T11:41:53.645 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:53.644+0000 7fb722e63700 1 Processor -- start 2024-01-21T11:41:53.645 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:53.644+0000 7fb722e63700 1 -- start start 2024-01-21T11:41:53.646 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:53.644+0000 7fb722e63700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb71c070430 0x7fb71c0fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T11:41:53.646 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:53.644+0000 7fb722e63700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb71c0fedd0 con 0x7fb71c070430 2024-01-21T11:41:53.646 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:53.644+0000 7fb720bff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb71c070430 0x7fb71c0fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:41:53.646 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:53.644+0000 7fb720bff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb71c070430 0x7fb71c0fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59888/0 (socket says 172.21.15.29:59888) 2024-01-21T11:41:53.646 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:53.644+0000 7fb720bff700 1 -- 172.21.15.29:0/2663019069 learned_addr learned my addr 172.21.15.29:0/2663019069 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-21T11:41:53.646 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:53.644+0000 7fb720bff700 1 -- 172.21.15.29:0/2663019069 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb71c0fef10 con 0x7fb71c070430 2024-01-21T11:41:53.646 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:53.645+0000 7fb720bff700 1 --2- 172.21.15.29:0/2663019069 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb71c070430 0x7fb71c0fe8e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fb71000b5f0 tx=0x7fb710006ee0).ready entity=mon.0 client_cookie=1e3c2c0c87efc36d server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:41:53.646 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:53.645+0000 7fb71b7fe700 1 -- 172.21.15.29:0/2663019069 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb71000d600 con 0x7fb71c070430 2024-01-21T11:41:53.646 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:53.645+0000 7fb71b7fe700 1 -- 172.21.15.29:0/2663019069 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fb71000dbd0 con 0x7fb71c070430 2024-01-21T11:41:53.646 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:53.645+0000 7fb71b7fe700 1 -- 172.21.15.29:0/2663019069 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb710011c00 con 0x7fb71c070430 2024-01-21T11:41:53.646 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:53.645+0000 7fb722e63700 1 -- 172.21.15.29:0/2663019069 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb71c070430 msgr2=0x7fb71c0fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:41:53.646 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:53.645+0000 7fb722e63700 1 --2- 172.21.15.29:0/2663019069 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb71c070430 0x7fb71c0fe8e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fb71000b5f0 tx=0x7fb710006ee0).stop 2024-01-21T11:41:53.648 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:53.645+0000 7fb722e63700 1 -- 172.21.15.29:0/2663019069 shutdown_connections 2024-01-21T11:41:53.648 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:53.645+0000 7fb722e63700 1 --2- 172.21.15.29:0/2663019069 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb71c070430 0x7fb71c0fe8e0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:41:53.648 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:53.645+0000 7fb722e63700 1 -- 172.21.15.29:0/2663019069 >> 172.21.15.29:0/2663019069 conn(0x7fb71c0f63c0 msgr2=0x7fb71c0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:41:53.648 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:53.646+0000 7fb722e63700 1 -- 172.21.15.29:0/2663019069 shutdown_connections 2024-01-21T11:41:53.648 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:53.646+0000 7fb722e63700 1 -- 172.21.15.29:0/2663019069 wait complete. 2024-01-21T11:41:53.648 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:53.646+0000 7fb722e63700 1 Processor -- start 2024-01-21T11:41:53.648 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:53.646+0000 7fb722e63700 1 -- start start 2024-01-21T11:41:53.648 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:53.647+0000 7fb722e63700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb71c070430 0x7fb71c10aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:41:53.648 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:53.647+0000 7fb722e63700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb71c10af20 con 0x7fb71c070430 2024-01-21T11:41:53.648 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:53.647+0000 7fb720bff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb71c070430 0x7fb71c10aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:41:53.648 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:53.647+0000 7fb720bff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb71c070430 0x7fb71c10aa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59896/0 (socket says 172.21.15.29:59896) 2024-01-21T11:41:53.649 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:53.647+0000 7fb720bff700 1 -- 172.21.15.29:0/2334763982 learned_addr learned my addr 172.21.15.29:0/2334763982 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-21T11:41:53.649 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:53.647+0000 7fb720bff700 1 -- 172.21.15.29:0/2334763982 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb71000b200 con 0x7fb71c070430 2024-01-21T11:41:53.649 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:53.647+0000 7fb720bff700 1 --2- 172.21.15.29:0/2334763982 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb71c070430 0x7fb71c10aa30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fb71000b190 tx=0x7fb710006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:41:53.649 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:53.647+0000 7fb7197fa700 1 -- 172.21.15.29:0/2334763982 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb71001b410 con 0x7fb71c070430 2024-01-21T11:41:53.649 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:53.647+0000 7fb722e63700 1 -- 172.21.15.29:0/2334763982 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb71c10b120 con 0x7fb71c070430 2024-01-21T11:41:53.649 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:53.647+0000 7fb7197fa700 1 -- 172.21.15.29:0/2334763982 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fb71001b9e0 con 0x7fb71c070430 2024-01-21T11:41:53.649 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:53.647+0000 7fb7197fa700 1 -- 172.21.15.29:0/2334763982 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb71001a880 con 0x7fb71c070430 2024-01-21T11:41:53.649 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:53.648+0000 7fb722e63700 1 -- 172.21.15.29:0/2334763982 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb71c10b530 con 0x7fb71c070430 2024-01-21T11:41:53.650 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:53.648+0000 7fb7197fa700 1 -- 172.21.15.29:0/2334763982 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7fb71000f070 con 0x7fb71c070430 2024-01-21T11:41:53.650 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:53.648+0000 7fb7197fa700 1 --2- 172.21.15.29:0/2334763982 >> [v2:172.21.15.29:6800/3957292961,v1:172.21.15.29:6801/3957292961] conn(0x7fb70002fe60 0x7fb700032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:41:53.650 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:53.648+0000 7fb7197fa700 1 -- 172.21.15.29:0/2334763982 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fb7100413d0 con 0x7fb71c070430 2024-01-21T11:41:53.650 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:53.648+0000 7fb71bfff700 1 --2- 172.21.15.29:0/2334763982 >> [v2:172.21.15.29:6800/3957292961,v1:172.21.15.29:6801/3957292961] conn(0x7fb70002fe60 0x7fb700032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:41:53.650 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:53.649+0000 7fb71bfff700 1 --2- 172.21.15.29:0/2334763982 >> [v2:172.21.15.29:6800/3957292961,v1:172.21.15.29:6801/3957292961] conn(0x7fb70002fe60 0x7fb700032320 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fb7080096a0 tx=0x7fb708006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:41:53.650 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:53.649+0000 7fb722e63700 1 -- 172.21.15.29:0/2334763982 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb71c059100 con 0x7fb71c070430 2024-01-21T11:41:53.653 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:53.653+0000 7fb7197fa700 1 -- 172.21.15.29:0/2334763982 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb710016070 con 0x7fb71c070430 2024-01-21T11:41:53.811 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:53.811+0000 7fb722e63700 1 -- 172.21.15.29:0/2334763982 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7fb71c104820 con 0x7fb71c070430 2024-01-21T11:41:53.813 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:53.814+0000 7fb7197fa700 1 -- 172.21.15.29:0/2334763982 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v8)=0 v8) v1 ==== 142+0+0 (secure 0 0 0) 0x7fb710010360 con 0x7fb71c070430 2024-01-21T11:41:53.816 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:53.815+0000 7fb722e63700 1 -- 172.21.15.29:0/2334763982 >> [v2:172.21.15.29:6800/3957292961,v1:172.21.15.29:6801/3957292961] conn(0x7fb70002fe60 msgr2=0x7fb700032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:41:53.816 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:53.815+0000 7fb722e63700 1 --2- 172.21.15.29:0/2334763982 >> [v2:172.21.15.29:6800/3957292961,v1:172.21.15.29:6801/3957292961] conn(0x7fb70002fe60 0x7fb700032320 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fb7080096a0 tx=0x7fb708006b40).stop 2024-01-21T11:41:53.816 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:53.815+0000 7fb722e63700 1 -- 172.21.15.29:0/2334763982 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb71c070430 msgr2=0x7fb71c10aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:41:53.816 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:53.815+0000 7fb722e63700 1 --2- 172.21.15.29:0/2334763982 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb71c070430 0x7fb71c10aa30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fb71000b190 tx=0x7fb710006ee0).stop 2024-01-21T11:41:53.817 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:53.816+0000 7fb722e63700 1 -- 172.21.15.29:0/2334763982 shutdown_connections 2024-01-21T11:41:53.817 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:53.816+0000 7fb722e63700 1 --2- 172.21.15.29:0/2334763982 >> [v2:172.21.15.29:6800/3957292961,v1:172.21.15.29:6801/3957292961] conn(0x7fb70002fe60 0x7fb700032320 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:41:53.817 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:53.816+0000 7fb722e63700 1 --2- 172.21.15.29:0/2334763982 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb71c070430 0x7fb71c10aa30 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:41:53.817 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:53.816+0000 7fb722e63700 1 -- 172.21.15.29:0/2334763982 >> 172.21.15.29:0/2334763982 conn(0x7fb71c0f63c0 msgr2=0x7fb71c0f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:41:53.817 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:53.816+0000 7fb722e63700 1 -- 172.21.15.29:0/2334763982 shutdown_connections 2024-01-21T11:41:53.817 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:53.816+0000 7fb722e63700 1 -- 172.21.15.29:0/2334763982 wait complete. 2024-01-21T11:41:54.102 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:53 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:41:52.041905+0000 mgr.smithi029.blhxxk (mgr.14144) 5 : audit [DBG] from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T11:41:54.102 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:53 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cephadm 2024-01-21T11:41:52.042528+0000 mgr.smithi029.blhxxk (mgr.14144) 6 : cephadm [INF] Saving service node-exporter spec with placement * 2024-01-21T11:41:54.102 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:53 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:41:52.947399+0000 mgr.smithi029.blhxxk (mgr.14144) 7 : audit [DBG] from='client.14156 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T11:41:54.102 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:53 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cephadm 2024-01-21T11:41:52.947932+0000 mgr.smithi029.blhxxk (mgr.14144) 8 : cephadm [INF] Saving service alertmanager spec with placement count:1 2024-01-21T11:41:54.102 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:53 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:41:52.949997+0000 mon.smithi029 (mon.0) 82 : audit [INF] from='mgr.14144 172.21.15.29:0/3161775375' entity='mgr.smithi029.blhxxk' 2024-01-21T11:41:54.102 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:53 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:41:53.814735+0000 mon.smithi029 (mon.0) 83 : audit [INF] from='client.? 172.21.15.29:0/2334763982' entity='client.admin' 2024-01-21T11:41:54.175 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e4084074-b851-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp77cpgrhz:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpea05xzin:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/dashboard/ssl_server_port 8443 2024-01-21T11:41:54.501 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:54.497+0000 7f0b80dfd700 1 Processor -- start 2024-01-21T11:41:54.501 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:54.497+0000 7f0b80dfd700 1 -- start start 2024-01-21T11:41:54.501 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:54.497+0000 7f0b80dfd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b7c06fbb0 0x7f0b7c100ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T11:41:54.501 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:54.497+0000 7f0b80dfd700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b7c0fc590 con 0x7f0b7c06fbb0 2024-01-21T11:41:54.501 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:54.497+0000 7f0b7b7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b7c06fbb0 0x7f0b7c100ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:41:54.502 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:54.497+0000 7f0b7b7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b7c06fbb0 0x7f0b7c100ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59904/0 (socket says 172.21.15.29:59904) 2024-01-21T11:41:54.502 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:54.497+0000 7f0b7b7fe700 1 -- 172.21.15.29:0/3831175808 learned_addr learned my addr 172.21.15.29:0/3831175808 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-21T11:41:54.502 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:54.498+0000 7f0b7b7fe700 1 -- 172.21.15.29:0/3831175808 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b7c101090 con 0x7f0b7c06fbb0 2024-01-21T11:41:54.502 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:54.498+0000 7f0b7b7fe700 1 --2- 172.21.15.29:0/3831175808 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b7c06fbb0 0x7f0b7c100ba0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f0b6c00b5f0 tx=0x7f0b6c006ee0).ready entity=mon.0 client_cookie=9c37dc783df9d01e server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:41:54.502 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:54.498+0000 7f0b7a7fc700 1 -- 172.21.15.29:0/3831175808 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b6c00d600 con 0x7f0b7c06fbb0 2024-01-21T11:41:54.502 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:54.498+0000 7f0b7a7fc700 1 -- 172.21.15.29:0/3831175808 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f0b6c00dbd0 con 0x7f0b7c06fbb0 2024-01-21T11:41:54.502 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:54.498+0000 7f0b7a7fc700 1 -- 172.21.15.29:0/3831175808 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b6c011c00 con 0x7f0b7c06fbb0 2024-01-21T11:41:54.502 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:54.499+0000 7f0b80dfd700 1 -- 172.21.15.29:0/3831175808 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b7c06fbb0 msgr2=0x7f0b7c100ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:41:54.502 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:54.499+0000 7f0b80dfd700 1 --2- 172.21.15.29:0/3831175808 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b7c06fbb0 0x7f0b7c100ba0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f0b6c00b5f0 tx=0x7f0b6c006ee0).stop 2024-01-21T11:41:54.502 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:54.499+0000 7f0b80dfd700 1 -- 172.21.15.29:0/3831175808 shutdown_connections 2024-01-21T11:41:54.502 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:54.499+0000 7f0b80dfd700 1 --2- 172.21.15.29:0/3831175808 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b7c06fbb0 0x7f0b7c100ba0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:41:54.502 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:54.499+0000 7f0b80dfd700 1 -- 172.21.15.29:0/3831175808 >> 172.21.15.29:0/3831175808 conn(0x7f0b7c0f6220 msgr2=0x7f0b7c0f8640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:41:54.503 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:54.499+0000 7f0b80dfd700 1 -- 172.21.15.29:0/3831175808 shutdown_connections 2024-01-21T11:41:54.503 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:54.499+0000 7f0b80dfd700 1 -- 172.21.15.29:0/3831175808 wait complete. 2024-01-21T11:41:54.503 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:54.499+0000 7f0b80dfd700 1 Processor -- start 2024-01-21T11:41:54.503 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:54.499+0000 7f0b80dfd700 1 -- start start 2024-01-21T11:41:54.503 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:54.500+0000 7f0b80dfd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b7c06fbb0 0x7f0b7c10ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:41:54.503 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:54.500+0000 7f0b80dfd700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b7c10cf70 con 0x7f0b7c06fbb0 2024-01-21T11:41:54.503 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:54.500+0000 7f0b7b7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b7c06fbb0 0x7f0b7c10ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:41:54.503 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:54.500+0000 7f0b7b7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b7c06fbb0 0x7f0b7c10ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59920/0 (socket says 172.21.15.29:59920) 2024-01-21T11:41:54.503 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:54.500+0000 7f0b7b7fe700 1 -- 172.21.15.29:0/3746959930 learned_addr learned my addr 172.21.15.29:0/3746959930 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-21T11:41:54.503 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:54.500+0000 7f0b7b7fe700 1 -- 172.21.15.29:0/3746959930 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b6c00b200 con 0x7f0b7c06fbb0 2024-01-21T11:41:54.503 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:54.500+0000 7f0b7b7fe700 1 --2- 172.21.15.29:0/3746959930 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b7c06fbb0 0x7f0b7c10ca80 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f0b6c00b1d0 tx=0x7f0b6c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:41:54.503 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:54.500+0000 7f0b73fff700 1 -- 172.21.15.29:0/3746959930 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b6c011a20 con 0x7f0b7c06fbb0 2024-01-21T11:41:54.503 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:54.500+0000 7f0b73fff700 1 -- 172.21.15.29:0/3746959930 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f0b6c01b880 con 0x7f0b7c06fbb0 2024-01-21T11:41:54.503 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:54.500+0000 7f0b80dfd700 1 -- 172.21.15.29:0/3746959930 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b7c10d170 con 0x7f0b7c06fbb0 2024-01-21T11:41:54.504 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:54.500+0000 7f0b73fff700 1 -- 172.21.15.29:0/3746959930 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b6c01a700 con 0x7f0b7c06fbb0 2024-01-21T11:41:54.504 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:54.500+0000 7f0b80dfd700 1 -- 172.21.15.29:0/3746959930 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b7c10d580 con 0x7f0b7c06fbb0 2024-01-21T11:41:54.504 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:54.501+0000 7f0b73fff700 1 -- 172.21.15.29:0/3746959930 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f0b6c00f070 con 0x7f0b7c06fbb0 2024-01-21T11:41:54.504 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:54.501+0000 7f0b73fff700 1 --2- 172.21.15.29:0/3746959930 >> [v2:172.21.15.29:6800/3957292961,v1:172.21.15.29:6801/3957292961] conn(0x7f0b5c02fe60 0x7f0b5c032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:41:54.504 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:54.501+0000 7f0b73fff700 1 -- 172.21.15.29:0/3746959930 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f0b6c041240 con 0x7f0b7c06fbb0 2024-01-21T11:41:54.505 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:54.501+0000 7f0b80dfd700 1 -- 172.21.15.29:0/3746959930 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b7c045ad0 con 0x7f0b7c06fbb0 2024-01-21T11:41:54.505 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:54.501+0000 7f0b7affd700 1 --2- 172.21.15.29:0/3746959930 >> [v2:172.21.15.29:6800/3957292961,v1:172.21.15.29:6801/3957292961] conn(0x7f0b5c02fe60 0x7f0b5c032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:41:54.505 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:54.502+0000 7f0b7affd700 1 --2- 172.21.15.29:0/3746959930 >> [v2:172.21.15.29:6800/3957292961,v1:172.21.15.29:6801/3957292961] conn(0x7f0b5c02fe60 0x7f0b5c032320 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f0b640096a0 tx=0x7f0b64006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:41:54.505 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:54.506+0000 7f0b73fff700 1 -- 172.21.15.29:0/3746959930 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0b6c016080 con 0x7f0b7c06fbb0 2024-01-21T11:41:54.661 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:54.661+0000 7f0b80dfd700 1 -- 172.21.15.29:0/3746959930 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f0b7c101bf0 con 0x7f0b7c06fbb0 2024-01-21T11:41:54.664 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:54.664+0000 7f0b73fff700 1 -- 172.21.15.29:0/3746959930 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v9)=0 v9) v1 ==== 130+0+0 (secure 0 0 0) 0x7f0b6c019950 con 0x7f0b7c06fbb0 2024-01-21T11:41:54.667 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:54.666+0000 7f0b80dfd700 1 -- 172.21.15.29:0/3746959930 >> [v2:172.21.15.29:6800/3957292961,v1:172.21.15.29:6801/3957292961] conn(0x7f0b5c02fe60 msgr2=0x7f0b5c032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:41:54.667 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:54.666+0000 7f0b80dfd700 1 --2- 172.21.15.29:0/3746959930 >> [v2:172.21.15.29:6800/3957292961,v1:172.21.15.29:6801/3957292961] conn(0x7f0b5c02fe60 0x7f0b5c032320 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f0b640096a0 tx=0x7f0b64006b40).stop 2024-01-21T11:41:54.667 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:54.666+0000 7f0b80dfd700 1 -- 172.21.15.29:0/3746959930 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b7c06fbb0 msgr2=0x7f0b7c10ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:41:54.667 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:54.666+0000 7f0b80dfd700 1 --2- 172.21.15.29:0/3746959930 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b7c06fbb0 0x7f0b7c10ca80 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f0b6c00b1d0 tx=0x7f0b6c006ee0).stop 2024-01-21T11:41:54.668 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:54.666+0000 7f0b80dfd700 1 -- 172.21.15.29:0/3746959930 shutdown_connections 2024-01-21T11:41:54.668 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:54.666+0000 7f0b80dfd700 1 --2- 172.21.15.29:0/3746959930 >> [v2:172.21.15.29:6800/3957292961,v1:172.21.15.29:6801/3957292961] conn(0x7f0b5c02fe60 0x7f0b5c032320 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:41:54.668 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:54.666+0000 7f0b80dfd700 1 --2- 172.21.15.29:0/3746959930 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b7c06fbb0 0x7f0b7c10ca80 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:41:54.668 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:54.666+0000 7f0b80dfd700 1 -- 172.21.15.29:0/3746959930 >> 172.21.15.29:0/3746959930 conn(0x7f0b7c0f6220 msgr2=0x7f0b7c0f7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:41:54.668 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:54.666+0000 7f0b80dfd700 1 -- 172.21.15.29:0/3746959930 shutdown_connections 2024-01-21T11:41:54.668 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:54.667+0000 7f0b80dfd700 1 -- 172.21.15.29:0/3746959930 wait complete. 2024-01-21T11:41:55.008 INFO:teuthology.orchestra.run.smithi029.stderr:Enabling the dashboard module... 2024-01-21T11:41:55.008 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e4084074-b851-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp77cpgrhz:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpea05xzin:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable dashboard 2024-01-21T11:41:55.358 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:55.357+0000 7f48d9be1700 1 Processor -- start 2024-01-21T11:41:55.359 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:55.357+0000 7f48d9be1700 1 -- start start 2024-01-21T11:41:55.359 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:55.357+0000 7f48d9be1700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48d40fab80 0x7f48d40fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T11:41:55.359 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:55.357+0000 7f48d9be1700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48d40fb490 con 0x7f48d40fab80 2024-01-21T11:41:55.359 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:55.358+0000 7f48d37fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48d40fab80 0x7f48d40fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:41:55.359 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:55.358+0000 7f48d37fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48d40fab80 0x7f48d40fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59932/0 (socket says 172.21.15.29:59932) 2024-01-21T11:41:55.359 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:55.358+0000 7f48d37fe700 1 -- 172.21.15.29:0/1333822076 learned_addr learned my addr 172.21.15.29:0/1333822076 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-21T11:41:55.359 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:55.358+0000 7f48d37fe700 1 -- 172.21.15.29:0/1333822076 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48d40fb5d0 con 0x7f48d40fab80 2024-01-21T11:41:55.359 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:55.358+0000 7f48d37fe700 1 --2- 172.21.15.29:0/1333822076 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48d40fab80 0x7f48d40fafa0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f48c800b5f0 tx=0x7f48c8006ee0).ready entity=mon.0 client_cookie=2a79b92d10f23ce0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:41:55.361 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:55.358+0000 7f48d27fc700 1 -- 172.21.15.29:0/1333822076 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f48c800d600 con 0x7f48d40fab80 2024-01-21T11:41:55.361 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:55.358+0000 7f48d27fc700 1 -- 172.21.15.29:0/1333822076 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f48c800dbd0 con 0x7f48d40fab80 2024-01-21T11:41:55.361 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:55.358+0000 7f48d27fc700 1 -- 172.21.15.29:0/1333822076 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f48c8011c00 con 0x7f48d40fab80 2024-01-21T11:41:55.361 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:55.359+0000 7f48d9be1700 1 -- 172.21.15.29:0/1333822076 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48d40fab80 msgr2=0x7f48d40fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:41:55.361 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:55.359+0000 7f48d9be1700 1 --2- 172.21.15.29:0/1333822076 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48d40fab80 0x7f48d40fafa0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f48c800b5f0 tx=0x7f48c8006ee0).stop 2024-01-21T11:41:55.361 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:55.359+0000 7f48d9be1700 1 -- 172.21.15.29:0/1333822076 shutdown_connections 2024-01-21T11:41:55.361 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:55.359+0000 7f48d9be1700 1 --2- 172.21.15.29:0/1333822076 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48d40fab80 0x7f48d40fafa0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:41:55.361 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:55.359+0000 7f48d9be1700 1 -- 172.21.15.29:0/1333822076 >> 172.21.15.29:0/1333822076 conn(0x7f48d40f63c0 msgr2=0x7f48d40f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:41:55.361 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:55.359+0000 7f48d9be1700 1 -- 172.21.15.29:0/1333822076 shutdown_connections 2024-01-21T11:41:55.362 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:55.359+0000 7f48d9be1700 1 -- 172.21.15.29:0/1333822076 wait complete. 2024-01-21T11:41:55.362 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:55.360+0000 7f48d9be1700 1 Processor -- start 2024-01-21T11:41:55.362 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:55.360+0000 7f48d9be1700 1 -- start start 2024-01-21T11:41:55.362 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:55.360+0000 7f48d9be1700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48d40fab80 0x7f48d4108830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:41:55.362 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:55.360+0000 7f48d9be1700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48d4108d20 con 0x7f48d40fab80 2024-01-21T11:41:55.362 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:55.360+0000 7f48d37fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48d40fab80 0x7f48d4108830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:41:55.362 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:55.360+0000 7f48d37fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48d40fab80 0x7f48d4108830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59948/0 (socket says 172.21.15.29:59948) 2024-01-21T11:41:55.362 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:55.360+0000 7f48d37fe700 1 -- 172.21.15.29:0/472472456 learned_addr learned my addr 172.21.15.29:0/472472456 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-21T11:41:55.362 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:55.360+0000 7f48d37fe700 1 -- 172.21.15.29:0/472472456 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48c800b200 con 0x7f48d40fab80 2024-01-21T11:41:55.362 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:55.361+0000 7f48d37fe700 1 --2- 172.21.15.29:0/472472456 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48d40fab80 0x7f48d4108830 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f48c800e040 tx=0x7f48c8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:41:55.362 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:55.361+0000 7f48bbfff700 1 -- 172.21.15.29:0/472472456 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f48c801b410 con 0x7f48d40fab80 2024-01-21T11:41:55.362 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:55.361+0000 7f48bbfff700 1 -- 172.21.15.29:0/472472456 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f48c801b9e0 con 0x7f48d40fab80 2024-01-21T11:41:55.362 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:55.361+0000 7f48d9be1700 1 -- 172.21.15.29:0/472472456 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48d4108f20 con 0x7f48d40fab80 2024-01-21T11:41:55.362 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:55.361+0000 7f48bbfff700 1 -- 172.21.15.29:0/472472456 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f48c801a820 con 0x7f48d40fab80 2024-01-21T11:41:55.362 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:55.361+0000 7f48d9be1700 1 -- 172.21.15.29:0/472472456 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48d4109330 con 0x7f48d40fab80 2024-01-21T11:41:55.363 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:55.362+0000 7f48bbfff700 1 -- 172.21.15.29:0/472472456 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f48c800f070 con 0x7f48d40fab80 2024-01-21T11:41:55.363 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:55.362+0000 7f48bbfff700 1 --2- 172.21.15.29:0/472472456 >> [v2:172.21.15.29:6800/3957292961,v1:172.21.15.29:6801/3957292961] conn(0x7f48c00320a0 0x7f48c0034560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:41:55.363 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:55.362+0000 7f48d9be1700 1 -- 172.21.15.29:0/472472456 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48d4045ad0 con 0x7f48d40fab80 2024-01-21T11:41:55.363 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:55.362+0000 7f48bbfff700 1 -- 172.21.15.29:0/472472456 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f48c8041510 con 0x7f48d40fab80 2024-01-21T11:41:55.364 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:55.362+0000 7f48d2ffd700 1 --2- 172.21.15.29:0/472472456 >> [v2:172.21.15.29:6800/3957292961,v1:172.21.15.29:6801/3957292961] conn(0x7f48c00320a0 0x7f48c0034560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:41:55.364 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:55.363+0000 7f48d2ffd700 1 --2- 172.21.15.29:0/472472456 >> [v2:172.21.15.29:6800/3957292961,v1:172.21.15.29:6801/3957292961] conn(0x7f48c00320a0 0x7f48c0034560 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f48c40096a0 tx=0x7f48c4006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:41:55.366 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:55.366+0000 7f48bbfff700 1 -- 172.21.15.29:0/472472456 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f48c8025460 con 0x7f48d40fab80 2024-01-21T11:41:55.563 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:55.563+0000 7f48d9be1700 1 -- 172.21.15.29:0/472472456 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f48d40fc890 con 0x7f48d40fab80 2024-01-21T11:41:55.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:55 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:41:54.665289+0000 mon.smithi029 (mon.0) 84 : audit [INF] from='client.? 172.21.15.29:0/3746959930' entity='client.admin' 2024-01-21T11:41:55.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:55 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:41:55.564473+0000 mon.smithi029 (mon.0) 85 : audit [INF] from='client.? 172.21.15.29:0/472472456' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-01-21T11:41:56.497 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:56.496+0000 7f48bbfff700 1 -- 172.21.15.29:0/472472456 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f48c80257e0 con 0x7f48d40fab80 2024-01-21T11:41:56.498 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:56.498+0000 7f48bbfff700 1 -- 172.21.15.29:0/472472456 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v13) v1 ==== 88+0+0 (secure 0 0 0) 0x7f48c80237d0 con 0x7f48d40fab80 2024-01-21T11:41:56.503 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:56.502+0000 7f48d9be1700 1 -- 172.21.15.29:0/472472456 >> [v2:172.21.15.29:6800/3957292961,v1:172.21.15.29:6801/3957292961] conn(0x7f48c00320a0 msgr2=0x7f48c0034560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:41:56.503 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:56.502+0000 7f48d9be1700 1 --2- 172.21.15.29:0/472472456 >> [v2:172.21.15.29:6800/3957292961,v1:172.21.15.29:6801/3957292961] conn(0x7f48c00320a0 0x7f48c0034560 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f48c40096a0 tx=0x7f48c4006b40).stop 2024-01-21T11:41:56.504 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:56.502+0000 7f48d9be1700 1 -- 172.21.15.29:0/472472456 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48d40fab80 msgr2=0x7f48d4108830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:41:56.504 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:56.502+0000 7f48d9be1700 1 --2- 172.21.15.29:0/472472456 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48d40fab80 0x7f48d4108830 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f48c800e040 tx=0x7f48c8006ee0).stop 2024-01-21T11:41:56.504 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:56.502+0000 7f48d9be1700 1 -- 172.21.15.29:0/472472456 shutdown_connections 2024-01-21T11:41:56.504 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:56.502+0000 7f48d9be1700 1 --2- 172.21.15.29:0/472472456 >> [v2:172.21.15.29:6800/3957292961,v1:172.21.15.29:6801/3957292961] conn(0x7f48c00320a0 0x7f48c0034560 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:41:56.504 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:56.502+0000 7f48d9be1700 1 --2- 172.21.15.29:0/472472456 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48d40fab80 0x7f48d4108830 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:41:56.504 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:56.502+0000 7f48d9be1700 1 -- 172.21.15.29:0/472472456 >> 172.21.15.29:0/472472456 conn(0x7f48d40f63c0 msgr2=0x7f48d40ff910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:41:56.504 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:56.503+0000 7f48d9be1700 1 -- 172.21.15.29:0/472472456 shutdown_connections 2024-01-21T11:41:56.504 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:56.503+0000 7f48d9be1700 1 -- 172.21.15.29:0/472472456 wait complete. 2024-01-21T11:41:57.025 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e4084074-b851-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp77cpgrhz:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpea05xzin:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2024-01-21T11:41:57.400 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:57.399+0000 7f086a0b5700 1 Processor -- start 2024-01-21T11:41:57.401 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:57.399+0000 7f086a0b5700 1 -- start start 2024-01-21T11:41:57.401 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:57.399+0000 7f086a0b5700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f08640fbca0 0x7f08640fc0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T11:41:57.401 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:57.399+0000 7f086a0b5700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08640fc5b0 con 0x7f08640fbca0 2024-01-21T11:41:57.401 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:57.400+0000 7f08637fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f08640fbca0 0x7f08640fc0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:41:57.401 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:57.400+0000 7f08637fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f08640fbca0 0x7f08640fc0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59978/0 (socket says 172.21.15.29:59978) 2024-01-21T11:41:57.401 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:57.400+0000 7f08637fe700 1 -- 172.21.15.29:0/3957289217 learned_addr learned my addr 172.21.15.29:0/3957289217 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-21T11:41:57.401 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:57.400+0000 7f08637fe700 1 -- 172.21.15.29:0/3957289217 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08640fc6f0 con 0x7f08640fbca0 2024-01-21T11:41:57.404 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:57.400+0000 7f08637fe700 1 --2- 172.21.15.29:0/3957289217 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f08640fbca0 0x7f08640fc0c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f085400b5f0 tx=0x7f0854006ee0).ready entity=mon.0 client_cookie=302b38488a4c132a server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:41:57.404 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:57.400+0000 7f08627fc700 1 -- 172.21.15.29:0/3957289217 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f085400d600 con 0x7f08640fbca0 2024-01-21T11:41:57.404 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:57.400+0000 7f08627fc700 1 -- 172.21.15.29:0/3957289217 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f085400dbd0 con 0x7f08640fbca0 2024-01-21T11:41:57.404 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:57.400+0000 7f08627fc700 1 -- 172.21.15.29:0/3957289217 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0854011c00 con 0x7f08640fbca0 2024-01-21T11:41:57.404 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:57.401+0000 7f086a0b5700 1 -- 172.21.15.29:0/3957289217 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f08640fbca0 msgr2=0x7f08640fc0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:41:57.405 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:57.401+0000 7f086a0b5700 1 --2- 172.21.15.29:0/3957289217 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f08640fbca0 0x7f08640fc0c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f085400b5f0 tx=0x7f0854006ee0).stop 2024-01-21T11:41:57.405 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:57.401+0000 7f086a0b5700 1 -- 172.21.15.29:0/3957289217 shutdown_connections 2024-01-21T11:41:57.405 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:57.401+0000 7f086a0b5700 1 --2- 172.21.15.29:0/3957289217 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f08640fbca0 0x7f08640fc0c0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:41:57.405 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:57.401+0000 7f086a0b5700 1 -- 172.21.15.29:0/3957289217 >> 172.21.15.29:0/3957289217 conn(0x7f08640f74c0 msgr2=0x7f08640f98e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:41:57.405 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:57.401+0000 7f086a0b5700 1 -- 172.21.15.29:0/3957289217 shutdown_connections 2024-01-21T11:41:57.405 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:57.401+0000 7f086a0b5700 1 -- 172.21.15.29:0/3957289217 wait complete. 2024-01-21T11:41:57.405 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:57.402+0000 7f086a0b5700 1 Processor -- start 2024-01-21T11:41:57.405 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:57.402+0000 7f086a0b5700 1 -- start start 2024-01-21T11:41:57.405 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:57.403+0000 7f086a0b5700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f08640fbca0 0x7f086410aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:41:57.405 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:57.403+0000 7f086a0b5700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f086410aef0 con 0x7f08640fbca0 2024-01-21T11:41:57.405 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:57.403+0000 7f08637fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f08640fbca0 0x7f086410aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:41:57.405 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:57.403+0000 7f08637fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f08640fbca0 0x7f086410aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59984/0 (socket says 172.21.15.29:59984) 2024-01-21T11:41:57.405 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:57.403+0000 7f08637fe700 1 -- 172.21.15.29:0/1113593157 learned_addr learned my addr 172.21.15.29:0/1113593157 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-21T11:41:57.405 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:57.403+0000 7f08637fe700 1 -- 172.21.15.29:0/1113593157 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f085400b200 con 0x7f08640fbca0 2024-01-21T11:41:57.406 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:57.403+0000 7f08637fe700 1 --2- 172.21.15.29:0/1113593157 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f08640fbca0 0x7f086410aa00 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f085400b1d0 tx=0x7f0854006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:41:57.406 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:57.403+0000 7f084bfff700 1 -- 172.21.15.29:0/1113593157 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f085401b410 con 0x7f08640fbca0 2024-01-21T11:41:57.406 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:57.404+0000 7f086a0b5700 1 -- 172.21.15.29:0/1113593157 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f086410b0f0 con 0x7f08640fbca0 2024-01-21T11:41:57.406 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:57.404+0000 7f084bfff700 1 -- 172.21.15.29:0/1113593157 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f085401b9e0 con 0x7f08640fbca0 2024-01-21T11:41:57.406 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:57.404+0000 7f084bfff700 1 -- 172.21.15.29:0/1113593157 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f085401a840 con 0x7f08640fbca0 2024-01-21T11:41:57.406 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:57.404+0000 7f086a0b5700 1 -- 172.21.15.29:0/1113593157 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f086410b500 con 0x7f08640fbca0 2024-01-21T11:41:57.407 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:57.405+0000 7f086a0b5700 1 -- 172.21.15.29:0/1113593157 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0864045ad0 con 0x7f08640fbca0 2024-01-21T11:41:57.407 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:57.405+0000 7f084bfff700 1 -- 172.21.15.29:0/1113593157 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f085400f070 con 0x7f08640fbca0 2024-01-21T11:41:57.407 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:57.405+0000 7f084bfff700 1 --2- 172.21.15.29:0/1113593157 >> [v2:172.21.15.29:6800/3957292961,v1:172.21.15.29:6801/3957292961] conn(0x7f085002feb0 0x7f0850032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:41:57.407 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:57.405+0000 7f084bfff700 1 -- 172.21.15.29:0/1113593157 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f08540414a0 con 0x7f08640fbca0 2024-01-21T11:41:57.407 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:57.405+0000 7f0862ffd700 1 -- 172.21.15.29:0/1113593157 >> [v2:172.21.15.29:6800/3957292961,v1:172.21.15.29:6801/3957292961] conn(0x7f085002feb0 msgr2=0x7f0850032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/3957292961 2024-01-21T11:41:57.407 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:57.405+0000 7f0862ffd700 1 --2- 172.21.15.29:0/1113593157 >> [v2:172.21.15.29:6800/3957292961,v1:172.21.15.29:6801/3957292961] conn(0x7f085002feb0 0x7f0850032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-01-21T11:41:57.409 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:57.409+0000 7f084bfff700 1 -- 172.21.15.29:0/1113593157 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0854019c90 con 0x7f08640fbca0 2024-01-21T11:41:57.607 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:57.606+0000 7f0862ffd700 1 -- 172.21.15.29:0/1113593157 >> [v2:172.21.15.29:6800/3957292961,v1:172.21.15.29:6801/3957292961] conn(0x7f085002feb0 msgr2=0x7f0850032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/3957292961 2024-01-21T11:41:57.607 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:57.606+0000 7f0862ffd700 1 --2- 172.21.15.29:0/1113593157 >> [v2:172.21.15.29:6800/3957292961,v1:172.21.15.29:6801/3957292961] conn(0x7f085002feb0 0x7f0850032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-01-21T11:41:57.614 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:57.614+0000 7f086a0b5700 1 -- 172.21.15.29:0/1113593157 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f086410c5b0 con 0x7f08640fbca0 2024-01-21T11:41:57.615 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:57.614+0000 7f084bfff700 1 -- 172.21.15.29:0/1113593157 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v13) v1 ==== 56+0+104 (secure 0 0 0) 0x7f0854016070 con 0x7f08640fbca0 2024-01-21T11:41:57.615 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: { 2024-01-21T11:41:57.615 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "epoch": 13, 2024-01-21T11:41:57.615 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "available": true, 2024-01-21T11:41:57.615 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "active_name": "smithi029.blhxxk", 2024-01-21T11:41:57.615 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_standby": 0 2024-01-21T11:41:57.615 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: } 2024-01-21T11:41:57.618 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:57.617+0000 7f086a0b5700 1 -- 172.21.15.29:0/1113593157 >> [v2:172.21.15.29:6800/3957292961,v1:172.21.15.29:6801/3957292961] conn(0x7f085002feb0 msgr2=0x7f0850032370 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-21T11:41:57.618 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:57.617+0000 7f086a0b5700 1 --2- 172.21.15.29:0/1113593157 >> [v2:172.21.15.29:6800/3957292961,v1:172.21.15.29:6801/3957292961] conn(0x7f085002feb0 0x7f0850032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-21T11:41:57.618 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:57.617+0000 7f086a0b5700 1 -- 172.21.15.29:0/1113593157 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f08640fbca0 msgr2=0x7f086410aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:41:57.618 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:57.617+0000 7f086a0b5700 1 --2- 172.21.15.29:0/1113593157 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f08640fbca0 0x7f086410aa00 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f085400b1d0 tx=0x7f0854006ee0).stop 2024-01-21T11:41:57.618 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:57.617+0000 7f086a0b5700 1 -- 172.21.15.29:0/1113593157 shutdown_connections 2024-01-21T11:41:57.618 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:57.617+0000 7f086a0b5700 1 --2- 172.21.15.29:0/1113593157 >> [v2:172.21.15.29:6800/3957292961,v1:172.21.15.29:6801/3957292961] conn(0x7f085002feb0 0x7f0850032370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-21T11:41:57.619 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:57.617+0000 7f086a0b5700 1 --2- 172.21.15.29:0/1113593157 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f08640fbca0 0x7f086410aa00 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:41:57.619 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:57.617+0000 7f086a0b5700 1 -- 172.21.15.29:0/1113593157 >> 172.21.15.29:0/1113593157 conn(0x7f08640f74c0 msgr2=0x7f08640f8f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:41:57.619 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:57.618+0000 7f086a0b5700 1 -- 172.21.15.29:0/1113593157 shutdown_connections 2024-01-21T11:41:57.619 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:57.618+0000 7f086a0b5700 1 -- 172.21.15.29:0/1113593157 wait complete. 2024-01-21T11:41:57.798 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:57 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:41:56.499456+0000 mon.smithi029 (mon.0) 86 : audit [INF] from='client.? 172.21.15.29:0/472472456' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-01-21T11:41:57.798 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:57 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:41:56.499528+0000 mon.smithi029 (mon.0) 87 : cluster [DBG] mgrmap e13: smithi029.blhxxk(active, since 7s) 2024-01-21T11:41:57.983 INFO:teuthology.orchestra.run.smithi029.stderr:Waiting for the mgr to restart... 2024-01-21T11:41:57.983 INFO:teuthology.orchestra.run.smithi029.stderr:Waiting for mgr epoch 13... 2024-01-21T11:41:57.983 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e4084074-b851-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp77cpgrhz:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpea05xzin:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2024-01-21T11:41:58.506 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:58.504+0000 7fbbb47aa700 1 Processor -- start 2024-01-21T11:41:58.506 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:58.504+0000 7fbbb47aa700 1 -- start start 2024-01-21T11:41:58.506 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:58.504+0000 7fbbb47aa700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbac0703d0 0x7fbbac0fe940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T11:41:58.506 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:58.504+0000 7fbbb47aa700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbac0fee30 con 0x7fbbac0703d0 2024-01-21T11:41:58.506 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:58.504+0000 7fbbb2546700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbac0703d0 0x7fbbac0fe940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:41:58.506 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:58.504+0000 7fbbb2546700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbac0703d0 0x7fbbac0fe940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60000/0 (socket says 172.21.15.29:60000) 2024-01-21T11:41:58.506 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:58.504+0000 7fbbb2546700 1 -- 172.21.15.29:0/222437076 learned_addr learned my addr 172.21.15.29:0/222437076 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-21T11:41:58.506 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:58.504+0000 7fbbb2546700 1 -- 172.21.15.29:0/222437076 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbac0fef70 con 0x7fbbac0703d0 2024-01-21T11:41:58.506 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:58.505+0000 7fbbb2546700 1 --2- 172.21.15.29:0/222437076 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbac0703d0 0x7fbbac0fe940 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7fbba800b5f0 tx=0x7fbba8006ee0).ready entity=mon.0 client_cookie=c759e9f1a2b79148 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:41:58.506 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:58.505+0000 7fbbb1544700 1 -- 172.21.15.29:0/222437076 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbba800d600 con 0x7fbbac0703d0 2024-01-21T11:41:58.506 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:58.505+0000 7fbbb1544700 1 -- 172.21.15.29:0/222437076 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fbba800dbd0 con 0x7fbbac0703d0 2024-01-21T11:41:58.506 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:58.505+0000 7fbbb1544700 1 -- 172.21.15.29:0/222437076 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbba8011c00 con 0x7fbbac0703d0 2024-01-21T11:41:58.507 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:58.505+0000 7fbbb47aa700 1 -- 172.21.15.29:0/222437076 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbac0703d0 msgr2=0x7fbbac0fe940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:41:58.507 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:58.505+0000 7fbbb47aa700 1 --2- 172.21.15.29:0/222437076 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbac0703d0 0x7fbbac0fe940 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7fbba800b5f0 tx=0x7fbba8006ee0).stop 2024-01-21T11:41:58.507 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:58.506+0000 7fbbb47aa700 1 -- 172.21.15.29:0/222437076 shutdown_connections 2024-01-21T11:41:58.510 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:58.506+0000 7fbbb47aa700 1 --2- 172.21.15.29:0/222437076 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbac0703d0 0x7fbbac0fe940 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:41:58.510 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:58.506+0000 7fbbb47aa700 1 -- 172.21.15.29:0/222437076 >> 172.21.15.29:0/222437076 conn(0x7fbbac0f63c0 msgr2=0x7fbbac0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:41:58.510 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:58.506+0000 7fbbb47aa700 1 -- 172.21.15.29:0/222437076 shutdown_connections 2024-01-21T11:41:58.510 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:58.506+0000 7fbbb47aa700 1 -- 172.21.15.29:0/222437076 wait complete. 2024-01-21T11:41:58.510 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:58.506+0000 7fbbb47aa700 1 Processor -- start 2024-01-21T11:41:58.510 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:58.506+0000 7fbbb47aa700 1 -- start start 2024-01-21T11:41:58.510 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:58.507+0000 7fbbb47aa700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbac0703d0 0x7fbbac10aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:41:58.510 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:58.507+0000 7fbbb47aa700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbac10af20 con 0x7fbbac0703d0 2024-01-21T11:41:58.510 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:58.507+0000 7fbbb2546700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbac0703d0 0x7fbbac10aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:41:58.511 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:58.507+0000 7fbbb2546700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbac0703d0 0x7fbbac10aa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60008/0 (socket says 172.21.15.29:60008) 2024-01-21T11:41:58.511 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:58.507+0000 7fbbb2546700 1 -- 172.21.15.29:0/4226012561 learned_addr learned my addr 172.21.15.29:0/4226012561 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-21T11:41:58.511 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:58.507+0000 7fbbb2546700 1 -- 172.21.15.29:0/4226012561 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbba800b200 con 0x7fbbac0703d0 2024-01-21T11:41:58.511 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:58.507+0000 7fbbb2546700 1 --2- 172.21.15.29:0/4226012561 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbac0703d0 0x7fbbac10aa30 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7fbba800e040 tx=0x7fbba8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:41:58.511 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:58.507+0000 7fbb9affd700 1 -- 172.21.15.29:0/4226012561 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbba800d9f0 con 0x7fbbac0703d0 2024-01-21T11:41:58.511 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:58.507+0000 7fbb9affd700 1 -- 172.21.15.29:0/4226012561 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fbba801a9f0 con 0x7fbbac0703d0 2024-01-21T11:41:58.511 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:58.507+0000 7fbbb47aa700 1 -- 172.21.15.29:0/4226012561 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbac10b120 con 0x7fbbac0703d0 2024-01-21T11:41:58.511 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:58.507+0000 7fbb9affd700 1 -- 172.21.15.29:0/4226012561 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbba801b8c0 con 0x7fbbac0703d0 2024-01-21T11:41:58.511 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:58.507+0000 7fbbb47aa700 1 -- 172.21.15.29:0/4226012561 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbac10b530 con 0x7fbbac0703d0 2024-01-21T11:41:58.511 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:58.508+0000 7fbb9affd700 1 -- 172.21.15.29:0/4226012561 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7fbba800f070 con 0x7fbbac0703d0 2024-01-21T11:41:58.511 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:58.508+0000 7fbb9affd700 1 --2- 172.21.15.29:0/4226012561 >> [v2:172.21.15.29:6800/3957292961,v1:172.21.15.29:6801/3957292961] conn(0x7fbb9402feb0 0x7fbb94032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:41:58.511 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:58.509+0000 7fbbb47aa700 1 -- 172.21.15.29:0/4226012561 --> [v2:172.21.15.29:6800/3957292961,v1:172.21.15.29:6801/3957292961] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbbac059100 con 0x7fbb9402feb0 2024-01-21T11:41:58.511 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:58.509+0000 7fbb9affd700 1 -- 172.21.15.29:0/4226012561 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fbba80414d0 con 0x7fbbac0703d0 2024-01-21T11:41:58.511 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:58.509+0000 7fbbb1d45700 1 -- 172.21.15.29:0/4226012561 >> [v2:172.21.15.29:6800/3957292961,v1:172.21.15.29:6801/3957292961] conn(0x7fbb9402feb0 msgr2=0x7fbb94032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/3957292961 2024-01-21T11:41:58.511 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:58.509+0000 7fbbb1d45700 1 --2- 172.21.15.29:0/4226012561 >> [v2:172.21.15.29:6800/3957292961,v1:172.21.15.29:6801/3957292961] conn(0x7fbb9402feb0 0x7fbb94032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-01-21T11:41:58.710 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:58.710+0000 7fbbb1d45700 1 -- 172.21.15.29:0/4226012561 >> [v2:172.21.15.29:6800/3957292961,v1:172.21.15.29:6801/3957292961] conn(0x7fbb9402feb0 msgr2=0x7fbb94032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/3957292961 2024-01-21T11:41:58.710 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:58.710+0000 7fbbb1d45700 1 --2- 172.21.15.29:0/4226012561 >> [v2:172.21.15.29:6800/3957292961,v1:172.21.15.29:6801/3957292961] conn(0x7fbb9402feb0 0x7fbb94032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-01-21T11:41:58.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:41:58 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:41:57.615119+0000 mon.smithi029 (mon.0) 88 : audit [DBG] from='client.? 172.21.15.29:0/1113593157' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-01-21T11:41:59.111 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:59.111+0000 7fbbb1d45700 1 -- 172.21.15.29:0/4226012561 >> [v2:172.21.15.29:6800/3957292961,v1:172.21.15.29:6801/3957292961] conn(0x7fbb9402feb0 msgr2=0x7fbb94032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/3957292961 2024-01-21T11:41:59.111 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:59.111+0000 7fbbb1d45700 1 --2- 172.21.15.29:0/4226012561 >> [v2:172.21.15.29:6800/3957292961,v1:172.21.15.29:6801/3957292961] conn(0x7fbb9402feb0 0x7fbb94032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2024-01-21T11:41:59.912 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:59.911+0000 7fbbb1d45700 1 -- 172.21.15.29:0/4226012561 >> [v2:172.21.15.29:6800/3957292961,v1:172.21.15.29:6801/3957292961] conn(0x7fbb9402feb0 msgr2=0x7fbb94032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/3957292961 2024-01-21T11:41:59.912 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:41:59.911+0000 7fbbb1d45700 1 --2- 172.21.15.29:0/4226012561 >> [v2:172.21.15.29:6800/3957292961,v1:172.21.15.29:6801/3957292961] conn(0x7fbb9402feb0 0x7fbb94032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2024-01-21T11:42:01.513 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:01.512+0000 7fbbb1d45700 1 -- 172.21.15.29:0/4226012561 >> [v2:172.21.15.29:6800/3957292961,v1:172.21.15.29:6801/3957292961] conn(0x7fbb9402feb0 msgr2=0x7fbb94032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/3957292961 2024-01-21T11:42:01.513 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:01.512+0000 7fbbb1d45700 1 --2- 172.21.15.29:0/4226012561 >> [v2:172.21.15.29:6800/3957292961,v1:172.21.15.29:6801/3957292961] conn(0x7fbb9402feb0 0x7fbb94032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2024-01-21T11:42:01.798 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:01.797+0000 7fbb9affd700 1 -- 172.21.15.29:0/4226012561 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mgrmap(e 14) v1 ==== 35997+0+0 (secure 0 0 0) 0x7fbba8022d80 con 0x7fbbac0703d0 2024-01-21T11:42:01.798 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:01.797+0000 7fbb9affd700 1 -- 172.21.15.29:0/4226012561 >> [v2:172.21.15.29:6800/3957292961,v1:172.21.15.29:6801/3957292961] conn(0x7fbb9402feb0 msgr2=0x7fbb94032370 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-21T11:42:01.798 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:01.797+0000 7fbb9affd700 1 --2- 172.21.15.29:0/4226012561 >> [v2:172.21.15.29:6800/3957292961,v1:172.21.15.29:6801/3957292961] conn(0x7fbb9402feb0 0x7fbb94032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-21T11:42:02.800 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:02.798+0000 7fbb9affd700 1 -- 172.21.15.29:0/4226012561 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 15) v1 ==== 36067+0+0 (secure 0 0 0) 0x7fbba8023d70 con 0x7fbbac0703d0 2024-01-21T11:42:02.801 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:02.798+0000 7fbb9affd700 1 --2- 172.21.15.29:0/4226012561 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fbb94033990 0x7fbb94035d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:42:02.801 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:02.798+0000 7fbb9affd700 1 -- 172.21.15.29:0/4226012561 --> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbbac059100 con 0x7fbb94033990 2024-01-21T11:42:02.801 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:02.799+0000 7fbbb1d45700 1 --2- 172.21.15.29:0/4226012561 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fbb94033990 0x7fbb94035d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:42:02.801 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:02.799+0000 7fbbb1d45700 1 --2- 172.21.15.29:0/4226012561 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fbb94033990 0x7fbb94035d80 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fbb9c006fd0 tx=0x7fbb9c006cf0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:42:02.801 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:02.800+0000 7fbb9affd700 1 -- 172.21.15.29:0/4226012561 <== mgr.14164 v2:172.21.15.29:6800/3399348778 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7fbbac059100 con 0x7fbb94033990 2024-01-21T11:42:02.807 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:02.806+0000 7fbbb47aa700 1 -- 172.21.15.29:0/4226012561 --> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fbbac045ad0 con 0x7fbb94033990 2024-01-21T11:42:02.808 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:02.807+0000 7fbb9affd700 1 -- 172.21.15.29:0/4226012561 <== mgr.14164 v2:172.21.15.29:6800/3399348778 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7fbbac045ad0 con 0x7fbb94033990 2024-01-21T11:42:02.809 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: { 2024-01-21T11:42:02.809 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "mgrmap_epoch": 15, 2024-01-21T11:42:02.809 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "initialized": true 2024-01-21T11:42:02.809 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: } 2024-01-21T11:42:02.809 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:02.807+0000 7fbbb47aa700 1 -- 172.21.15.29:0/4226012561 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fbb94033990 msgr2=0x7fbb94035d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:42:02.810 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:02.807+0000 7fbbb47aa700 1 --2- 172.21.15.29:0/4226012561 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fbb94033990 0x7fbb94035d80 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fbb9c006fd0 tx=0x7fbb9c006cf0).stop 2024-01-21T11:42:02.810 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:02.807+0000 7fbbb47aa700 1 -- 172.21.15.29:0/4226012561 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbac0703d0 msgr2=0x7fbbac10aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:42:02.810 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:02.807+0000 7fbbb47aa700 1 --2- 172.21.15.29:0/4226012561 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbac0703d0 0x7fbbac10aa30 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7fbba800e040 tx=0x7fbba8006ee0).stop 2024-01-21T11:42:02.810 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:02.807+0000 7fbbb47aa700 1 -- 172.21.15.29:0/4226012561 shutdown_connections 2024-01-21T11:42:02.810 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:02.807+0000 7fbbb47aa700 1 --2- 172.21.15.29:0/4226012561 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fbb94033990 0x7fbb94035d80 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:42:02.810 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:02.807+0000 7fbbb47aa700 1 --2- 172.21.15.29:0/4226012561 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbac0703d0 0x7fbbac10aa30 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:42:02.810 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:02.807+0000 7fbbb47aa700 1 --2- 172.21.15.29:0/4226012561 >> [v2:172.21.15.29:6800/3957292961,v1:172.21.15.29:6801/3957292961] conn(0x7fbb9402feb0 0x7fbb94032370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-21T11:42:02.810 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:02.807+0000 7fbbb47aa700 1 -- 172.21.15.29:0/4226012561 >> 172.21.15.29:0/4226012561 conn(0x7fbbac0f63c0 msgr2=0x7fbbac0f7f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:42:02.810 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:02.807+0000 7fbbb47aa700 1 -- 172.21.15.29:0/4226012561 shutdown_connections 2024-01-21T11:42:02.810 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:02.807+0000 7fbbb47aa700 1 -- 172.21.15.29:0/4226012561 wait complete. 2024-01-21T11:42:03.102 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:42:02 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:42:01.746860+0000 mon.smithi029 (mon.0) 89 : cluster [INF] Active manager daemon smithi029.blhxxk restarted 2024-01-21T11:42:03.102 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:42:02 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:42:01.747354+0000 mon.smithi029 (mon.0) 90 : cluster [INF] Activating manager daemon smithi029.blhxxk 2024-01-21T11:42:03.102 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:42:02 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:42:01.749006+0000 mon.smithi029 (mon.0) 91 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-01-21T11:42:03.102 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:42:02 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:42:01.799817+0000 mon.smithi029 (mon.0) 92 : cluster [DBG] mgrmap e14: smithi029.blhxxk(active, starting, since 0.0525807s) 2024-01-21T11:42:03.102 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:42:02 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:42:01.800771+0000 mon.smithi029 (mon.0) 93 : audit [DBG] from='mgr.14164 172.21.15.29:0/3252518221' entity='mgr.smithi029.blhxxk' cmd=[{"prefix": "mon metadata", "id": "smithi029"}]: dispatch 2024-01-21T11:42:03.102 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:42:02 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:42:01.801287+0000 mon.smithi029 (mon.0) 94 : audit [DBG] from='mgr.14164 172.21.15.29:0/3252518221' entity='mgr.smithi029.blhxxk' cmd=[{"prefix": "mgr metadata", "who": "smithi029.blhxxk", "id": "smithi029.blhxxk"}]: dispatch 2024-01-21T11:42:03.102 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:42:02 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:42:01.801967+0000 mon.smithi029 (mon.0) 95 : audit [DBG] from='mgr.14164 172.21.15.29:0/3252518221' entity='mgr.smithi029.blhxxk' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-21T11:42:03.102 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:42:02 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:42:01.802124+0000 mon.smithi029 (mon.0) 96 : audit [DBG] from='mgr.14164 172.21.15.29:0/3252518221' entity='mgr.smithi029.blhxxk' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-21T11:42:03.102 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:42:02 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:42:01.802259+0000 mon.smithi029 (mon.0) 97 : audit [DBG] from='mgr.14164 172.21.15.29:0/3252518221' entity='mgr.smithi029.blhxxk' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-21T11:42:03.102 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:42:02 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:42:01.804180+0000 mon.smithi029 (mon.0) 98 : cluster [INF] Manager daemon smithi029.blhxxk is now available 2024-01-21T11:42:03.102 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:42:02 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:42:01.908731+0000 mon.smithi029 (mon.0) 99 : audit [DBG] from='mgr.14164 172.21.15.29:0/3252518221' entity='mgr.smithi029.blhxxk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-21T11:42:03.102 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:42:02 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:42:01.909885+0000 mon.smithi029 (mon.0) 100 : audit [DBG] from='mgr.14164 172.21.15.29:0/3252518221' entity='mgr.smithi029.blhxxk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-21T11:42:03.102 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:42:02 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:42:01.935380+0000 mon.smithi029 (mon.0) 101 : audit [INF] from='mgr.14164 172.21.15.29:0/3252518221' entity='mgr.smithi029.blhxxk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi029.blhxxk/mirror_snapshot_schedule"}]: dispatch 2024-01-21T11:42:03.103 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:42:02 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:42:01.938387+0000 mon.smithi029 (mon.0) 102 : audit [INF] from='mgr.14164 172.21.15.29:0/3252518221' entity='mgr.smithi029.blhxxk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi029.blhxxk/trash_purge_schedule"}]: dispatch 2024-01-21T11:42:03.103 INFO:teuthology.orchestra.run.smithi029.stderr:mgr epoch 13 is available 2024-01-21T11:42:03.103 INFO:teuthology.orchestra.run.smithi029.stderr:Generating a dashboard self-signed certificate... 2024-01-21T11:42:03.103 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e4084074-b851-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp77cpgrhz:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpea05xzin:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 dashboard create-self-signed-cert 2024-01-21T11:42:03.654 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:03.652+0000 7f0bd2aa9700 1 Processor -- start 2024-01-21T11:42:03.656 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:03.652+0000 7f0bd2aa9700 1 -- start start 2024-01-21T11:42:03.656 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:03.652+0000 7f0bd2aa9700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0bcc0fab80 0x7f0bcc0fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T11:42:03.656 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:03.652+0000 7f0bd2aa9700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bcc0fb490 con 0x7f0bcc0fab80 2024-01-21T11:42:03.656 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:03.652+0000 7f0bd0845700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0bcc0fab80 0x7f0bcc0fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:42:03.656 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:03.652+0000 7f0bd0845700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0bcc0fab80 0x7f0bcc0fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60010/0 (socket says 172.21.15.29:60010) 2024-01-21T11:42:03.656 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:03.652+0000 7f0bd0845700 1 -- 172.21.15.29:0/1848952385 learned_addr learned my addr 172.21.15.29:0/1848952385 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-21T11:42:03.656 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:03.653+0000 7f0bd0845700 1 -- 172.21.15.29:0/1848952385 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bcc0fb5d0 con 0x7f0bcc0fab80 2024-01-21T11:42:03.657 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:03.653+0000 7f0bd0845700 1 --2- 172.21.15.29:0/1848952385 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0bcc0fab80 0x7f0bcc0fafa0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f0bc000b5f0 tx=0x7f0bc0006ee0).ready entity=mon.0 client_cookie=2e56dc4ad7d6c183 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:42:03.657 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:03.653+0000 7f0bcb7fe700 1 -- 172.21.15.29:0/1848952385 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0bc000d600 con 0x7f0bcc0fab80 2024-01-21T11:42:03.657 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:03.653+0000 7f0bcb7fe700 1 -- 172.21.15.29:0/1848952385 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f0bc000dbd0 con 0x7f0bcc0fab80 2024-01-21T11:42:03.657 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:03.653+0000 7f0bcb7fe700 1 -- 172.21.15.29:0/1848952385 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0bc0011c00 con 0x7f0bcc0fab80 2024-01-21T11:42:03.657 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:03.653+0000 7f0bd2aa9700 1 -- 172.21.15.29:0/1848952385 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0bcc0fab80 msgr2=0x7f0bcc0fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:42:03.657 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:03.653+0000 7f0bd2aa9700 1 --2- 172.21.15.29:0/1848952385 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0bcc0fab80 0x7f0bcc0fafa0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f0bc000b5f0 tx=0x7f0bc0006ee0).stop 2024-01-21T11:42:03.657 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:03.653+0000 7f0bd2aa9700 1 -- 172.21.15.29:0/1848952385 shutdown_connections 2024-01-21T11:42:03.657 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:03.653+0000 7f0bd2aa9700 1 --2- 172.21.15.29:0/1848952385 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0bcc0fab80 0x7f0bcc0fafa0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:42:03.657 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:03.653+0000 7f0bd2aa9700 1 -- 172.21.15.29:0/1848952385 >> 172.21.15.29:0/1848952385 conn(0x7f0bcc0f63c0 msgr2=0x7f0bcc0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:42:03.657 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:03.653+0000 7f0bd2aa9700 1 -- 172.21.15.29:0/1848952385 shutdown_connections 2024-01-21T11:42:03.657 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:03.654+0000 7f0bd2aa9700 1 -- 172.21.15.29:0/1848952385 wait complete. 2024-01-21T11:42:03.657 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:03.654+0000 7f0bd2aa9700 1 Processor -- start 2024-01-21T11:42:03.657 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:03.654+0000 7f0bd2aa9700 1 -- start start 2024-01-21T11:42:03.657 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:03.655+0000 7f0bd2aa9700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0bcc0fab80 0x7f0bcc1087d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:42:03.657 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:03.655+0000 7f0bd2aa9700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bcc108cc0 con 0x7f0bcc0fab80 2024-01-21T11:42:03.658 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:03.655+0000 7f0bd0845700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0bcc0fab80 0x7f0bcc1087d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:42:03.658 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:03.655+0000 7f0bd0845700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0bcc0fab80 0x7f0bcc1087d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60024/0 (socket says 172.21.15.29:60024) 2024-01-21T11:42:03.659 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:03.655+0000 7f0bd0845700 1 -- 172.21.15.29:0/142782075 learned_addr learned my addr 172.21.15.29:0/142782075 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-21T11:42:03.659 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:03.655+0000 7f0bd0845700 1 -- 172.21.15.29:0/142782075 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bc000b200 con 0x7f0bcc0fab80 2024-01-21T11:42:03.659 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:03.655+0000 7f0bd0845700 1 --2- 172.21.15.29:0/142782075 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0bcc0fab80 0x7f0bcc1087d0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f0bc000b1d0 tx=0x7f0bc0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:42:03.659 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:03.655+0000 7f0bc97fa700 1 -- 172.21.15.29:0/142782075 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0bc001b410 con 0x7f0bcc0fab80 2024-01-21T11:42:03.659 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:03.655+0000 7f0bd2aa9700 1 -- 172.21.15.29:0/142782075 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bcc108ec0 con 0x7f0bcc0fab80 2024-01-21T11:42:03.659 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:03.655+0000 7f0bc97fa700 1 -- 172.21.15.29:0/142782075 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f0bc001b9e0 con 0x7f0bcc0fab80 2024-01-21T11:42:03.659 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:03.655+0000 7f0bc97fa700 1 -- 172.21.15.29:0/142782075 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0bc001a840 con 0x7f0bcc0fab80 2024-01-21T11:42:03.659 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:03.655+0000 7f0bd2aa9700 1 -- 172.21.15.29:0/142782075 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bcc1092d0 con 0x7f0bcc0fab80 2024-01-21T11:42:03.659 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:03.656+0000 7f0bd2aa9700 1 -- 172.21.15.29:0/142782075 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bcc045ad0 con 0x7f0bcc0fab80 2024-01-21T11:42:03.659 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:03.657+0000 7f0bc97fa700 1 -- 172.21.15.29:0/142782075 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 15) v1 ==== 36067+0+0 (secure 0 0 0) 0x7f0bc000f070 con 0x7f0bcc0fab80 2024-01-21T11:42:03.659 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:03.657+0000 7f0bc97fa700 1 --2- 172.21.15.29:0/142782075 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f0bb0031f90 0x7f0bb0034450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:42:03.659 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:03.657+0000 7f0bc97fa700 1 -- 172.21.15.29:0/142782075 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0bc0041e50 con 0x7f0bcc0fab80 2024-01-21T11:42:03.659 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:03.657+0000 7f0bcbfff700 1 --2- 172.21.15.29:0/142782075 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f0bb0031f90 0x7f0bb0034450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:42:03.659 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:03.657+0000 7f0bcbfff700 1 --2- 172.21.15.29:0/142782075 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f0bb0031f90 0x7f0bb0034450 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f0bb80096a0 tx=0x7f0bb8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:42:03.662 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:03.661+0000 7f0bc97fa700 1 -- 172.21.15.29:0/142782075 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0bc002db70 con 0x7f0bcc0fab80 2024-01-21T11:42:03.803 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:03.801+0000 7f0bc97fa700 1 -- 172.21.15.29:0/142782075 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f0bc001bc10 con 0x7f0bcc0fab80 2024-01-21T11:42:03.824 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:03.823+0000 7f0bd2aa9700 1 -- 172.21.15.29:0/142782075 --> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f0bcc109bb0 con 0x7f0bb0031f90 2024-01-21T11:42:03.970 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:03.969+0000 7f0bc97fa700 1 -- 172.21.15.29:0/142782075 <== mgr.14164 v2:172.21.15.29:6800/3399348778 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f0bcc109bb0 con 0x7f0bb0031f90 2024-01-21T11:42:03.970 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: Self-signed certificate created 2024-01-21T11:42:03.972 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:03.971+0000 7f0bd2aa9700 1 -- 172.21.15.29:0/142782075 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f0bb0031f90 msgr2=0x7f0bb0034450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:42:03.973 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:03.971+0000 7f0bd2aa9700 1 --2- 172.21.15.29:0/142782075 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f0bb0031f90 0x7f0bb0034450 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f0bb80096a0 tx=0x7f0bb8006b40).stop 2024-01-21T11:42:03.973 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:03.971+0000 7f0bd2aa9700 1 -- 172.21.15.29:0/142782075 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0bcc0fab80 msgr2=0x7f0bcc1087d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:42:03.973 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:03.971+0000 7f0bd2aa9700 1 --2- 172.21.15.29:0/142782075 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0bcc0fab80 0x7f0bcc1087d0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f0bc000b1d0 tx=0x7f0bc0006ee0).stop 2024-01-21T11:42:03.973 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:03.971+0000 7f0bd2aa9700 1 -- 172.21.15.29:0/142782075 shutdown_connections 2024-01-21T11:42:03.973 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:03.971+0000 7f0bd2aa9700 1 --2- 172.21.15.29:0/142782075 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f0bb0031f90 0x7f0bb0034450 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:42:03.973 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:03.971+0000 7f0bd2aa9700 1 --2- 172.21.15.29:0/142782075 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0bcc0fab80 0x7f0bcc1087d0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:42:03.973 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:03.971+0000 7f0bd2aa9700 1 -- 172.21.15.29:0/142782075 >> 172.21.15.29:0/142782075 conn(0x7f0bcc0f63c0 msgr2=0x7f0bcc0ff910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:42:03.973 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:03.971+0000 7f0bd2aa9700 1 -- 172.21.15.29:0/142782075 shutdown_connections 2024-01-21T11:42:03.973 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:03.971+0000 7f0bd2aa9700 1 -- 172.21.15.29:0/142782075 wait complete. 2024-01-21T11:42:04.130 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:42:03 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:42:02.801728+0000 mgr.smithi029.blhxxk (mgr.14164) 1 : audit [DBG] from='client.14168 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-01-21T11:42:04.130 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:42:03 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:42:02.802431+0000 mon.smithi029 (mon.0) 103 : cluster [DBG] mgrmap e15: smithi029.blhxxk(active, since 1.05519s) 2024-01-21T11:42:04.130 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:42:03 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:42:02.808915+0000 mgr.smithi029.blhxxk (mgr.14164) 2 : audit [DBG] from='client.14168 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-01-21T11:42:04.323 INFO:teuthology.orchestra.run.smithi029.stderr:Creating initial admin user... 2024-01-21T11:42:04.323 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e4084074-b851-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp77cpgrhz:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpea05xzin:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpdxraxyyb:/tmp/dashboard.pw:z docker.io/ceph/ceph:v16.2.4 dashboard ac-user-create admin -i /tmp/dashboard.pw administrator --force-password --pwd-update-required 2024-01-21T11:42:04.656 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:04.654+0000 7f9d1d3bd700 1 Processor -- start 2024-01-21T11:42:04.657 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:04.654+0000 7f9d1d3bd700 1 -- start start 2024-01-21T11:42:04.657 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:04.654+0000 7f9d1d3bd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d18072490 0x7f9d18070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T11:42:04.657 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:04.654+0000 7f9d1d3bd700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d180728b0 con 0x7f9d18072490 2024-01-21T11:42:04.657 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:04.655+0000 7f9d16ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d18072490 0x7f9d18070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:42:04.657 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:04.655+0000 7f9d16ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d18072490 0x7f9d18070a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60030/0 (socket says 172.21.15.29:60030) 2024-01-21T11:42:04.657 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:04.655+0000 7f9d16ffd700 1 -- 172.21.15.29:0/477519599 learned_addr learned my addr 172.21.15.29:0/477519599 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-21T11:42:04.657 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:04.655+0000 7f9d16ffd700 1 -- 172.21.15.29:0/477519599 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d18070f20 con 0x7f9d18072490 2024-01-21T11:42:04.657 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:04.655+0000 7f9d16ffd700 1 --2- 172.21.15.29:0/477519599 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d18072490 0x7f9d18070a30 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f9d0000b5f0 tx=0x7f9d00006ee0).ready entity=mon.0 client_cookie=48d5daf67cdec010 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:42:04.657 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:04.655+0000 7f9d15ffb700 1 -- 172.21.15.29:0/477519599 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d0000d600 con 0x7f9d18072490 2024-01-21T11:42:04.657 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:04.655+0000 7f9d15ffb700 1 -- 172.21.15.29:0/477519599 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f9d0000dbd0 con 0x7f9d18072490 2024-01-21T11:42:04.660 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:04.655+0000 7f9d15ffb700 1 -- 172.21.15.29:0/477519599 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d00011c00 con 0x7f9d18072490 2024-01-21T11:42:04.660 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:04.656+0000 7f9d1d3bd700 1 -- 172.21.15.29:0/477519599 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d18072490 msgr2=0x7f9d18070a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:42:04.660 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:04.656+0000 7f9d1d3bd700 1 --2- 172.21.15.29:0/477519599 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d18072490 0x7f9d18070a30 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f9d0000b5f0 tx=0x7f9d00006ee0).stop 2024-01-21T11:42:04.660 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:04.656+0000 7f9d1d3bd700 1 -- 172.21.15.29:0/477519599 shutdown_connections 2024-01-21T11:42:04.660 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:04.656+0000 7f9d1d3bd700 1 --2- 172.21.15.29:0/477519599 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d18072490 0x7f9d18070a30 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:42:04.661 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:04.656+0000 7f9d1d3bd700 1 -- 172.21.15.29:0/477519599 >> 172.21.15.29:0/477519599 conn(0x7f9d180f74c0 msgr2=0x7f9d180f98e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:42:04.661 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:04.656+0000 7f9d1d3bd700 1 -- 172.21.15.29:0/477519599 shutdown_connections 2024-01-21T11:42:04.661 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:04.656+0000 7f9d1d3bd700 1 -- 172.21.15.29:0/477519599 wait complete. 2024-01-21T11:42:04.661 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:04.657+0000 7f9d1d3bd700 1 Processor -- start 2024-01-21T11:42:04.661 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:04.657+0000 7f9d1d3bd700 1 -- start start 2024-01-21T11:42:04.661 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:04.657+0000 7f9d1d3bd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d18072490 0x7f9d1810ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:42:04.661 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:04.657+0000 7f9d1d3bd700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d1810d1c0 con 0x7f9d18072490 2024-01-21T11:42:04.661 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:04.657+0000 7f9d16ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d18072490 0x7f9d1810ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:42:04.661 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:04.657+0000 7f9d16ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d18072490 0x7f9d1810ccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60034/0 (socket says 172.21.15.29:60034) 2024-01-21T11:42:04.661 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:04.657+0000 7f9d16ffd700 1 -- 172.21.15.29:0/4178203079 learned_addr learned my addr 172.21.15.29:0/4178203079 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-21T11:42:04.661 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:04.658+0000 7f9d16ffd700 1 -- 172.21.15.29:0/4178203079 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d0000b200 con 0x7f9d18072490 2024-01-21T11:42:04.661 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:04.658+0000 7f9d16ffd700 1 --2- 172.21.15.29:0/4178203079 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d18072490 0x7f9d1810ccd0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f9d00011820 tx=0x7f9d00006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:42:04.661 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:04.658+0000 7f9d0f7fe700 1 -- 172.21.15.29:0/4178203079 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d0001b410 con 0x7f9d18072490 2024-01-21T11:42:04.661 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:04.658+0000 7f9d0f7fe700 1 -- 172.21.15.29:0/4178203079 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f9d0001b9e0 con 0x7f9d18072490 2024-01-21T11:42:04.662 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:04.658+0000 7f9d1d3bd700 1 -- 172.21.15.29:0/4178203079 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d1810d3c0 con 0x7f9d18072490 2024-01-21T11:42:04.662 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:04.658+0000 7f9d0f7fe700 1 -- 172.21.15.29:0/4178203079 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d0001a8b0 con 0x7f9d18072490 2024-01-21T11:42:04.662 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:04.658+0000 7f9d1d3bd700 1 -- 172.21.15.29:0/4178203079 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d1810d7d0 con 0x7f9d18072490 2024-01-21T11:42:04.662 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:04.659+0000 7f9d0f7fe700 1 -- 172.21.15.29:0/4178203079 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f9d0000f070 con 0x7f9d18072490 2024-01-21T11:42:04.662 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:04.659+0000 7f9d0f7fe700 1 --2- 172.21.15.29:0/4178203079 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f9cf802feb0 0x7f9cf8032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:42:04.662 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:04.659+0000 7f9d0f7fe700 1 -- 172.21.15.29:0/4178203079 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9d00041590 con 0x7f9d18072490 2024-01-21T11:42:04.662 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:04.659+0000 7f9d167fc700 1 --2- 172.21.15.29:0/4178203079 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f9cf802feb0 0x7f9cf8032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:42:04.662 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:04.659+0000 7f9d167fc700 1 --2- 172.21.15.29:0/4178203079 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f9cf802feb0 0x7f9cf8032370 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f9d080096a0 tx=0x7f9d08006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:42:04.662 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:04.660+0000 7f9d1d3bd700 1 -- 172.21.15.29:0/4178203079 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d18101dd0 con 0x7f9d18072490 2024-01-21T11:42:04.665 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:04.664+0000 7f9d0f7fe700 1 -- 172.21.15.29:0/4178203079 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9d00016070 con 0x7f9d18072490 2024-01-21T11:42:04.840 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:04.839+0000 7f9d1d3bd700 1 -- 172.21.15.29:0/4178203079 --> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f9d18101f90 con 0x7f9cf802feb0 2024-01-21T11:42:05.085 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:05.084+0000 7f9d0f7fe700 1 -- 172.21.15.29:0/4178203079 <== mgr.14164 v2:172.21.15.29:6800/3399348778 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f9d18101f90 con 0x7f9cf802feb0 2024-01-21T11:42:05.085 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$OesYg.6cRkkYHyj/Mupxnehfcv4CR191e7FVp/2upJAl8VGSBrE4G", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1705837325, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-01-21T11:42:05.087 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:05.085+0000 7f9d1d3bd700 1 -- 172.21.15.29:0/4178203079 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f9cf802feb0 msgr2=0x7f9cf8032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:42:05.087 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:05.085+0000 7f9d1d3bd700 1 --2- 172.21.15.29:0/4178203079 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f9cf802feb0 0x7f9cf8032370 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f9d080096a0 tx=0x7f9d08006b40).stop 2024-01-21T11:42:05.087 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:05.085+0000 7f9d1d3bd700 1 -- 172.21.15.29:0/4178203079 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d18072490 msgr2=0x7f9d1810ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:42:05.088 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:05.085+0000 7f9d1d3bd700 1 --2- 172.21.15.29:0/4178203079 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d18072490 0x7f9d1810ccd0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f9d00011820 tx=0x7f9d00006ee0).stop 2024-01-21T11:42:05.088 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:05.085+0000 7f9d1d3bd700 1 -- 172.21.15.29:0/4178203079 shutdown_connections 2024-01-21T11:42:05.088 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:05.085+0000 7f9d1d3bd700 1 --2- 172.21.15.29:0/4178203079 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f9cf802feb0 0x7f9cf8032370 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:42:05.088 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:05.085+0000 7f9d1d3bd700 1 --2- 172.21.15.29:0/4178203079 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d18072490 0x7f9d1810ccd0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:42:05.088 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:05.086+0000 7f9d1d3bd700 1 -- 172.21.15.29:0/4178203079 >> 172.21.15.29:0/4178203079 conn(0x7f9d180f74c0 msgr2=0x7f9d180f90b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:42:05.088 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:05.086+0000 7f9d1d3bd700 1 -- 172.21.15.29:0/4178203079 shutdown_connections 2024-01-21T11:42:05.088 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:05.086+0000 7f9d1d3bd700 1 -- 172.21.15.29:0/4178203079 wait complete. 2024-01-21T11:42:05.109 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:42:04 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:42:03.804678+0000 mon.smithi029 (mon.0) 104 : cluster [DBG] mgrmap e16: smithi029.blhxxk(active, since 2s) 2024-01-21T11:42:05.109 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:42:04 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:42:03.825970+0000 mgr.smithi029.blhxxk (mgr.14164) 3 : audit [DBG] from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T11:42:05.109 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:42:04 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:42:03.969172+0000 mon.smithi029 (mon.0) 105 : audit [INF] from='mgr.14164 172.21.15.29:0/3252518221' entity='mgr.smithi029.blhxxk' 2024-01-21T11:42:05.109 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:42:04 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:42:03.970994+0000 mon.smithi029 (mon.0) 106 : audit [INF] from='mgr.14164 172.21.15.29:0/3252518221' entity='mgr.smithi029.blhxxk' 2024-01-21T11:42:05.457 INFO:teuthology.orchestra.run.smithi029.stderr:Fetching dashboard port number... 2024-01-21T11:42:05.457 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e4084074-b851-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp77cpgrhz:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpea05xzin:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config get mgr mgr/dashboard/ssl_server_port 2024-01-21T11:42:05.807 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:05.805+0000 7fc8373fb700 1 Processor -- start 2024-01-21T11:42:05.807 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:05.805+0000 7fc8373fb700 1 -- start start 2024-01-21T11:42:05.808 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:05.805+0000 7fc8373fb700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8300703b0 0x7fc830100d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T11:42:05.808 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:05.805+0000 7fc8373fb700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8300fc750 con 0x7fc8300703b0 2024-01-21T11:42:05.808 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:05.806+0000 7fc835197700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8300703b0 0x7fc830100d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:42:05.808 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:05.806+0000 7fc835197700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8300703b0 0x7fc830100d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60046/0 (socket says 172.21.15.29:60046) 2024-01-21T11:42:05.808 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:05.806+0000 7fc835197700 1 -- 172.21.15.29:0/2604738562 learned_addr learned my addr 172.21.15.29:0/2604738562 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-21T11:42:05.808 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:05.806+0000 7fc835197700 1 -- 172.21.15.29:0/2604738562 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc830101250 con 0x7fc8300703b0 2024-01-21T11:42:05.808 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:05.806+0000 7fc835197700 1 --2- 172.21.15.29:0/2604738562 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8300703b0 0x7fc830100d60 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7fc82000b5f0 tx=0x7fc820006ee0).ready entity=mon.0 client_cookie=e20b9dd64923dbdc server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:42:05.808 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:05.806+0000 7fc81ffff700 1 -- 172.21.15.29:0/2604738562 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc82000d600 con 0x7fc8300703b0 2024-01-21T11:42:05.808 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:05.806+0000 7fc81ffff700 1 -- 172.21.15.29:0/2604738562 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fc82000dbd0 con 0x7fc8300703b0 2024-01-21T11:42:05.808 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:05.806+0000 7fc81ffff700 1 -- 172.21.15.29:0/2604738562 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc820011c00 con 0x7fc8300703b0 2024-01-21T11:42:05.810 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:05.807+0000 7fc8373fb700 1 -- 172.21.15.29:0/2604738562 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8300703b0 msgr2=0x7fc830100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:42:05.810 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:05.807+0000 7fc8373fb700 1 --2- 172.21.15.29:0/2604738562 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8300703b0 0x7fc830100d60 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7fc82000b5f0 tx=0x7fc820006ee0).stop 2024-01-21T11:42:05.811 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:05.807+0000 7fc8373fb700 1 -- 172.21.15.29:0/2604738562 shutdown_connections 2024-01-21T11:42:05.811 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:05.807+0000 7fc8373fb700 1 --2- 172.21.15.29:0/2604738562 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8300703b0 0x7fc830100d60 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:42:05.811 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:05.807+0000 7fc8373fb700 1 -- 172.21.15.29:0/2604738562 >> 172.21.15.29:0/2604738562 conn(0x7fc8300f63c0 msgr2=0x7fc8300f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:42:05.811 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:05.807+0000 7fc8373fb700 1 -- 172.21.15.29:0/2604738562 shutdown_connections 2024-01-21T11:42:05.811 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:05.807+0000 7fc8373fb700 1 -- 172.21.15.29:0/2604738562 wait complete. 2024-01-21T11:42:05.811 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:05.808+0000 7fc8373fb700 1 Processor -- start 2024-01-21T11:42:05.811 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:05.808+0000 7fc8373fb700 1 -- start start 2024-01-21T11:42:05.811 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:05.808+0000 7fc8373fb700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8300703b0 0x7fc83010ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:42:05.811 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:05.808+0000 7fc8373fb700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc83010d1e0 con 0x7fc8300703b0 2024-01-21T11:42:05.811 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:05.808+0000 7fc835197700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8300703b0 0x7fc83010ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:42:05.811 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:05.808+0000 7fc835197700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8300703b0 0x7fc83010ccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60052/0 (socket says 172.21.15.29:60052) 2024-01-21T11:42:05.811 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:05.808+0000 7fc835197700 1 -- 172.21.15.29:0/181677724 learned_addr learned my addr 172.21.15.29:0/181677724 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-21T11:42:05.811 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:05.808+0000 7fc835197700 1 -- 172.21.15.29:0/181677724 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc82000b200 con 0x7fc8300703b0 2024-01-21T11:42:05.811 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:05.808+0000 7fc835197700 1 --2- 172.21.15.29:0/181677724 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8300703b0 0x7fc83010ccf0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fc82000e040 tx=0x7fc820006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:42:05.812 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:05.809+0000 7fc81dffb700 1 -- 172.21.15.29:0/181677724 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc82000d9f0 con 0x7fc8300703b0 2024-01-21T11:42:05.812 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:05.809+0000 7fc81dffb700 1 -- 172.21.15.29:0/181677724 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fc82001a9f0 con 0x7fc8300703b0 2024-01-21T11:42:05.812 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:05.809+0000 7fc8373fb700 1 -- 172.21.15.29:0/181677724 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc83010d3e0 con 0x7fc8300703b0 2024-01-21T11:42:05.812 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:05.809+0000 7fc81dffb700 1 -- 172.21.15.29:0/181677724 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc82001b8c0 con 0x7fc8300703b0 2024-01-21T11:42:05.812 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:05.809+0000 7fc8373fb700 1 -- 172.21.15.29:0/181677724 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc83010d7f0 con 0x7fc8300703b0 2024-01-21T11:42:05.812 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:05.810+0000 7fc81dffb700 1 -- 172.21.15.29:0/181677724 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7fc82000f070 con 0x7fc8300703b0 2024-01-21T11:42:05.813 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:05.810+0000 7fc81dffb700 1 --2- 172.21.15.29:0/181677724 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fc81802ff00 0x7fc8180323c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:42:05.813 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:05.810+0000 7fc81dffb700 1 -- 172.21.15.29:0/181677724 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc820041530 con 0x7fc8300703b0 2024-01-21T11:42:05.813 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:05.810+0000 7fc834996700 1 --2- 172.21.15.29:0/181677724 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fc81802ff00 0x7fc8180323c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:42:05.813 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:05.810+0000 7fc834996700 1 --2- 172.21.15.29:0/181677724 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fc81802ff00 0x7fc8180323c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fc82c0096a0 tx=0x7fc82c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:42:05.813 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:05.810+0000 7fc8373fb700 1 -- 172.21.15.29:0/181677724 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc830045ad0 con 0x7fc8300703b0 2024-01-21T11:42:05.816 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:05.815+0000 7fc81dffb700 1 -- 172.21.15.29:0/181677724 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc820016070 con 0x7fc8300703b0 2024-01-21T11:42:05.973 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:05.972+0000 7fc8373fb700 1 -- 172.21.15.29:0/181677724 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7fc830101db0 con 0x7fc8300703b0 2024-01-21T11:42:05.973 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:05.972+0000 7fc81dffb700 1 -- 172.21.15.29:0/181677724 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v9) v1 ==== 112+0+5 (secure 0 0 0) 0x7fc830101db0 con 0x7fc8300703b0 2024-01-21T11:42:05.973 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 8443 2024-01-21T11:42:05.976 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:05.974+0000 7fc8373fb700 1 -- 172.21.15.29:0/181677724 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fc81802ff00 msgr2=0x7fc8180323c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:42:05.976 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:05.974+0000 7fc8373fb700 1 --2- 172.21.15.29:0/181677724 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fc81802ff00 0x7fc8180323c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fc82c0096a0 tx=0x7fc82c006b40).stop 2024-01-21T11:42:05.976 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:05.974+0000 7fc8373fb700 1 -- 172.21.15.29:0/181677724 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8300703b0 msgr2=0x7fc83010ccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:42:05.976 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:05.974+0000 7fc8373fb700 1 --2- 172.21.15.29:0/181677724 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8300703b0 0x7fc83010ccf0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fc82000e040 tx=0x7fc820006ee0).stop 2024-01-21T11:42:05.976 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:05.974+0000 7fc8373fb700 1 -- 172.21.15.29:0/181677724 shutdown_connections 2024-01-21T11:42:05.976 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:05.974+0000 7fc8373fb700 1 --2- 172.21.15.29:0/181677724 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fc81802ff00 0x7fc8180323c0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:42:05.976 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:05.974+0000 7fc8373fb700 1 --2- 172.21.15.29:0/181677724 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8300703b0 0x7fc83010ccf0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:42:05.976 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:05.974+0000 7fc8373fb700 1 -- 172.21.15.29:0/181677724 >> 172.21.15.29:0/181677724 conn(0x7fc8300f63c0 msgr2=0x7fc8300f7f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:42:05.976 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:05.974+0000 7fc8373fb700 1 -- 172.21.15.29:0/181677724 shutdown_connections 2024-01-21T11:42:05.976 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-01-21T11:42:05.974+0000 7fc8373fb700 1 -- 172.21.15.29:0/181677724 wait complete. 2024-01-21T11:42:06.258 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:42:06 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:42:04.841453+0000 mgr.smithi029.blhxxk (mgr.14164) 4 : audit [DBG] from='client.14176 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T11:42:06.259 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:42:06 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:42:05.085751+0000 mon.smithi029 (mon.0) 107 : audit [INF] from='mgr.14164 172.21.15.29:0/3252518221' entity='mgr.smithi029.blhxxk' 2024-01-21T11:42:06.259 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:42:06 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:42:05.974356+0000 mon.smithi029 (mon.0) 108 : audit [DBG] from='client.? 172.21.15.29:0/181677724' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-01-21T11:42:06.331 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl is-enabled firewalld.service 2024-01-21T11:42:06.340 INFO:teuthology.orchestra.run.smithi029.stderr:systemctl: disabled 2024-01-21T11:42:06.340 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl is-active firewalld.service 2024-01-21T11:42:06.348 INFO:teuthology.orchestra.run.smithi029.stderr:systemctl: inactive 2024-01-21T11:42:06.348 INFO:teuthology.orchestra.run.smithi029.stderr:firewalld.service is not enabled 2024-01-21T11:42:06.348 INFO:teuthology.orchestra.run.smithi029.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-01-21T11:42:06.352 INFO:teuthology.orchestra.run.smithi029.stderr:Ceph Dashboard is now available at: 2024-01-21T11:42:06.352 INFO:teuthology.orchestra.run.smithi029.stderr: 2024-01-21T11:42:06.352 INFO:teuthology.orchestra.run.smithi029.stderr: URL: https://smithi029.front.sepia.ceph.com:8443/ 2024-01-21T11:42:06.352 INFO:teuthology.orchestra.run.smithi029.stderr: User: admin 2024-01-21T11:42:06.352 INFO:teuthology.orchestra.run.smithi029.stderr: Password: clg4hoex5v 2024-01-21T11:42:06.352 INFO:teuthology.orchestra.run.smithi029.stderr: 2024-01-21T11:42:06.352 INFO:teuthology.orchestra.run.smithi029.stderr:You can access the Ceph CLI with: 2024-01-21T11:42:06.352 INFO:teuthology.orchestra.run.smithi029.stderr: 2024-01-21T11:42:06.352 INFO:teuthology.orchestra.run.smithi029.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid e4084074-b851-11ee-95b0-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-01-21T11:42:06.352 INFO:teuthology.orchestra.run.smithi029.stderr: 2024-01-21T11:42:06.352 INFO:teuthology.orchestra.run.smithi029.stderr:Please consider enabling telemetry to help improve Ceph: 2024-01-21T11:42:06.352 INFO:teuthology.orchestra.run.smithi029.stderr: 2024-01-21T11:42:06.352 INFO:teuthology.orchestra.run.smithi029.stderr: ceph telemetry on 2024-01-21T11:42:06.353 INFO:teuthology.orchestra.run.smithi029.stderr: 2024-01-21T11:42:06.353 INFO:teuthology.orchestra.run.smithi029.stderr:For more information see: 2024-01-21T11:42:06.353 INFO:teuthology.orchestra.run.smithi029.stderr: 2024-01-21T11:42:06.353 INFO:teuthology.orchestra.run.smithi029.stderr: https://docs.ceph.com/docs/pacific/mgr/telemetry/ 2024-01-21T11:42:06.353 INFO:teuthology.orchestra.run.smithi029.stderr: 2024-01-21T11:42:06.353 INFO:teuthology.orchestra.run.smithi029.stderr:Bootstrap complete. 2024-01-21T11:42:06.353 INFO:teuthology.orchestra.run.smithi029.stderr:Releasing lock 139660192618984 on /run/cephadm/e4084074-b851-11ee-95b0-87774f69a715.lock 2024-01-21T11:42:06.353 INFO:teuthology.orchestra.run.smithi029.stderr:Lock 139660192618984 released on /run/cephadm/e4084074-b851-11ee-95b0-87774f69a715.lock 2024-01-21T11:42:06.384 INFO:tasks.cephadm:Fetching config... 2024-01-21T11:42:06.385 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-01-21T11:42:06.385 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-01-21T11:42:06.443 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-01-21T11:42:06.443 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-01-21T11:42:06.443 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-01-21T11:42:06.500 INFO:tasks.cephadm:Fetching mon keyring... 2024-01-21T11:42:06.501 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-01-21T11:42:06.501 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/e4084074-b851-11ee-95b0-87774f69a715/mon.smithi029/keyring of=/dev/stdout 2024-01-21T11:42:06.567 INFO:tasks.cephadm:Fetching pub ssh key... 2024-01-21T11:42:06.568 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-01-21T11:42:06.568 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-01-21T11:42:06.626 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-01-21T11:42:06.626 DEBUG:teuthology.orchestra.run.smithi029:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCzYY7NOXmrvZvCsV3lwxHRFOYUUFRZNS1WnQdeVkaNeylw7qpYI0RNpM6hpzRd0EiOUSPtSpc5hofN+fnIk0VDvelykm5CUqaHLx+GlPMXMi9Euq7IfZUqf/BCjGlwZDb9bOPV1kDu6cNHZf98ys3fyyvJSFraDinff83Y2bza7ERRbOHDrCRfN5A74Uwk3oCxNbFOETe0hgeI3AUDk7OT8hpYy8hlj4ALsz5b6BO6k4eqrmgtQSMI8wOMx5dhplgNlSbkWA3BGn7IxO6KDH7gwcge4K+nLOE79+r74AnRfZC0kTEJQt+n4ovjkWFV36LTv3vX0u83FR4EcBqdokb3JprCgfeHO1fvUmofF3+IK0sEn7jDnf6zgCKwNaWWk261wbdGkFXcR/38VF8OV+9X4UP+y9a+j/s8BoERlmXpAVTG4HDBDwdi4KRjxXSwO010Bk719KOkLSzcYKwltySTo9Rca7Vj733blQpnuWl6GkeTSol++flMwBgnj3s8YS0= ceph-e4084074-b851-11ee-95b0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-01-21T11:42:06.702 INFO:teuthology.orchestra.run.smithi029.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCzYY7NOXmrvZvCsV3lwxHRFOYUUFRZNS1WnQdeVkaNeylw7qpYI0RNpM6hpzRd0EiOUSPtSpc5hofN+fnIk0VDvelykm5CUqaHLx+GlPMXMi9Euq7IfZUqf/BCjGlwZDb9bOPV1kDu6cNHZf98ys3fyyvJSFraDinff83Y2bza7ERRbOHDrCRfN5A74Uwk3oCxNbFOETe0hgeI3AUDk7OT8hpYy8hlj4ALsz5b6BO6k4eqrmgtQSMI8wOMx5dhplgNlSbkWA3BGn7IxO6KDH7gwcge4K+nLOE79+r74AnRfZC0kTEJQt+n4ovjkWFV36LTv3vX0u83FR4EcBqdokb3JprCgfeHO1fvUmofF3+IK0sEn7jDnf6zgCKwNaWWk261wbdGkFXcR/38VF8OV+9X4UP+y9a+j/s8BoERlmXpAVTG4HDBDwdi4KRjxXSwO010Bk719KOkLSzcYKwltySTo9Rca7Vj733blQpnuWl6GkeTSol++flMwBgnj3s8YS0= ceph-e4084074-b851-11ee-95b0-87774f69a715 2024-01-21T11:42:06.713 DEBUG:teuthology.orchestra.run.smithi134:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCzYY7NOXmrvZvCsV3lwxHRFOYUUFRZNS1WnQdeVkaNeylw7qpYI0RNpM6hpzRd0EiOUSPtSpc5hofN+fnIk0VDvelykm5CUqaHLx+GlPMXMi9Euq7IfZUqf/BCjGlwZDb9bOPV1kDu6cNHZf98ys3fyyvJSFraDinff83Y2bza7ERRbOHDrCRfN5A74Uwk3oCxNbFOETe0hgeI3AUDk7OT8hpYy8hlj4ALsz5b6BO6k4eqrmgtQSMI8wOMx5dhplgNlSbkWA3BGn7IxO6KDH7gwcge4K+nLOE79+r74AnRfZC0kTEJQt+n4ovjkWFV36LTv3vX0u83FR4EcBqdokb3JprCgfeHO1fvUmofF3+IK0sEn7jDnf6zgCKwNaWWk261wbdGkFXcR/38VF8OV+9X4UP+y9a+j/s8BoERlmXpAVTG4HDBDwdi4KRjxXSwO010Bk719KOkLSzcYKwltySTo9Rca7Vj733blQpnuWl6GkeTSol++flMwBgnj3s8YS0= ceph-e4084074-b851-11ee-95b0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-01-21T11:42:06.756 INFO:teuthology.orchestra.run.smithi134.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCzYY7NOXmrvZvCsV3lwxHRFOYUUFRZNS1WnQdeVkaNeylw7qpYI0RNpM6hpzRd0EiOUSPtSpc5hofN+fnIk0VDvelykm5CUqaHLx+GlPMXMi9Euq7IfZUqf/BCjGlwZDb9bOPV1kDu6cNHZf98ys3fyyvJSFraDinff83Y2bza7ERRbOHDrCRfN5A74Uwk3oCxNbFOETe0hgeI3AUDk7OT8hpYy8hlj4ALsz5b6BO6k4eqrmgtQSMI8wOMx5dhplgNlSbkWA3BGn7IxO6KDH7gwcge4K+nLOE79+r74AnRfZC0kTEJQt+n4ovjkWFV36LTv3vX0u83FR4EcBqdokb3JprCgfeHO1fvUmofF3+IK0sEn7jDnf6zgCKwNaWWk261wbdGkFXcR/38VF8OV+9X4UP+y9a+j/s8BoERlmXpAVTG4HDBDwdi4KRjxXSwO010Bk719KOkLSzcYKwltySTo9Rca7Vj733blQpnuWl6GkeTSol++flMwBgnj3s8YS0= ceph-e4084074-b851-11ee-95b0-87774f69a715 2024-01-21T11:42:06.768 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-01-21T11:42:07.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-21T11:42:07.843+0000 7f4b9adb3700 1 -- 172.21.15.29:0/2576460544 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b940f51a0 msgr2=0x7f4b940f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:42:07.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-21T11:42:07.843+0000 7f4b9adb3700 1 --2- 172.21.15.29:0/2576460544 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b940f51a0 0x7f4b940f55c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f4b90009fb0 tx=0x7f4b90009b30).stop 2024-01-21T11:42:07.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-21T11:42:07.843+0000 7f4b9adb3700 1 -- 172.21.15.29:0/2576460544 shutdown_connections 2024-01-21T11:42:07.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-21T11:42:07.843+0000 7f4b9adb3700 1 --2- 172.21.15.29:0/2576460544 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b940f51a0 0x7f4b940f55c0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:42:07.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-21T11:42:07.843+0000 7f4b9adb3700 1 -- 172.21.15.29:0/2576460544 >> 172.21.15.29:0/2576460544 conn(0x7f4b940f09a0 msgr2=0x7f4b940f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:42:07.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-21T11:42:07.843+0000 7f4b9adb3700 1 -- 172.21.15.29:0/2576460544 shutdown_connections 2024-01-21T11:42:07.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-21T11:42:07.843+0000 7f4b9adb3700 1 -- 172.21.15.29:0/2576460544 wait complete. 2024-01-21T11:42:07.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-21T11:42:07.844+0000 7f4b9adb3700 1 Processor -- start 2024-01-21T11:42:07.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-21T11:42:07.844+0000 7f4b9adb3700 1 -- start start 2024-01-21T11:42:07.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-21T11:42:07.844+0000 7f4b9adb3700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b940f51a0 0x7f4b94105390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:42:07.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-21T11:42:07.844+0000 7f4b9adb3700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b94105880 con 0x7f4b940f51a0 2024-01-21T11:42:07.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-21T11:42:07.844+0000 7f4b99db1700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b940f51a0 0x7f4b94105390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:42:07.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-21T11:42:07.844+0000 7f4b99db1700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b940f51a0 0x7f4b94105390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60060/0 (socket says 172.21.15.29:60060) 2024-01-21T11:42:07.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-21T11:42:07.844+0000 7f4b99db1700 1 -- 172.21.15.29:0/2539771633 learned_addr learned my addr 172.21.15.29:0/2539771633 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-21T11:42:07.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-21T11:42:07.845+0000 7f4b99db1700 1 -- 172.21.15.29:0/2539771633 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b90005040 con 0x7f4b940f51a0 2024-01-21T11:42:07.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-21T11:42:07.845+0000 7f4b99db1700 1 --2- 172.21.15.29:0/2539771633 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b940f51a0 0x7f4b94105390 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f4b90002040 tx=0x7f4b9000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:42:07.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-21T11:42:07.845+0000 7f4b8a7fc700 1 -- 172.21.15.29:0/2539771633 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b900095f0 con 0x7f4b940f51a0 2024-01-21T11:42:07.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-21T11:42:07.845+0000 7f4b8a7fc700 1 -- 172.21.15.29:0/2539771633 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f4b90007e30 con 0x7f4b940f51a0 2024-01-21T11:42:07.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-21T11:42:07.845+0000 7f4b9adb3700 1 -- 172.21.15.29:0/2539771633 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b94105a80 con 0x7f4b940f51a0 2024-01-21T11:42:07.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-21T11:42:07.845+0000 7f4b8a7fc700 1 -- 172.21.15.29:0/2539771633 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b90014d20 con 0x7f4b940f51a0 2024-01-21T11:42:07.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-21T11:42:07.845+0000 7f4b9adb3700 1 -- 172.21.15.29:0/2539771633 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b94105f10 con 0x7f4b940f51a0 2024-01-21T11:42:07.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-21T11:42:07.846+0000 7f4b8a7fc700 1 -- 172.21.15.29:0/2539771633 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4b90025070 con 0x7f4b940f51a0 2024-01-21T11:42:07.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-21T11:42:07.846+0000 7f4b8a7fc700 1 --2- 172.21.15.29:0/2539771633 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f4b8002ffb0 0x7f4b80032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:42:07.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-21T11:42:07.846+0000 7f4b8a7fc700 1 -- 172.21.15.29:0/2539771633 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4b90040bf0 con 0x7f4b940f51a0 2024-01-21T11:42:07.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-21T11:42:07.846+0000 7f4b9adb3700 1 -- 172.21.15.29:0/2539771633 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b94045ad0 con 0x7f4b940f51a0 2024-01-21T11:42:07.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-21T11:42:07.846+0000 7f4b995b0700 1 --2- 172.21.15.29:0/2539771633 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f4b8002ffb0 0x7f4b80032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:42:07.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-21T11:42:07.846+0000 7f4b995b0700 1 --2- 172.21.15.29:0/2539771633 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f4b8002ffb0 0x7f4b80032470 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f4b840096a0 tx=0x7f4b84006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:42:07.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-21T11:42:07.850+0000 7f4b8a7fc700 1 -- 172.21.15.29:0/2539771633 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4b9000f300 con 0x7f4b940f51a0 2024-01-21T11:42:08.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-21T11:42:08.010+0000 7f4b9adb3700 1 -- 172.21.15.29:0/2539771633 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f4b940fabd0 con 0x7f4b940f51a0 2024-01-21T11:42:08.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-21T11:42:08.013+0000 7f4b8a7fc700 1 -- 172.21.15.29:0/2539771633 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v10)=0 v10) v1 ==== 125+0+0 (secure 0 0 0) 0x7f4b90025af0 con 0x7f4b940f51a0 2024-01-21T11:42:08.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-21T11:42:08.015+0000 7f4b9adb3700 1 -- 172.21.15.29:0/2539771633 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f4b8002ffb0 msgr2=0x7f4b80032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:42:08.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-21T11:42:08.015+0000 7f4b9adb3700 1 --2- 172.21.15.29:0/2539771633 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f4b8002ffb0 0x7f4b80032470 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f4b840096a0 tx=0x7f4b84006b40).stop 2024-01-21T11:42:08.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-21T11:42:08.015+0000 7f4b9adb3700 1 -- 172.21.15.29:0/2539771633 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b940f51a0 msgr2=0x7f4b94105390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:42:08.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-21T11:42:08.015+0000 7f4b9adb3700 1 --2- 172.21.15.29:0/2539771633 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b940f51a0 0x7f4b94105390 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f4b90002040 tx=0x7f4b9000b040).stop 2024-01-21T11:42:08.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-21T11:42:08.015+0000 7f4b9adb3700 1 -- 172.21.15.29:0/2539771633 shutdown_connections 2024-01-21T11:42:08.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-21T11:42:08.015+0000 7f4b9adb3700 1 --2- 172.21.15.29:0/2539771633 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f4b8002ffb0 0x7f4b80032470 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:42:08.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-21T11:42:08.015+0000 7f4b9adb3700 1 --2- 172.21.15.29:0/2539771633 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b940f51a0 0x7f4b94105390 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:42:08.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-21T11:42:08.015+0000 7f4b9adb3700 1 -- 172.21.15.29:0/2539771633 >> 172.21.15.29:0/2539771633 conn(0x7f4b940f09a0 msgr2=0x7f4b940f14c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:42:08.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-21T11:42:08.015+0000 7f4b9adb3700 1 -- 172.21.15.29:0/2539771633 shutdown_connections 2024-01-21T11:42:08.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-21T11:42:08.015+0000 7f4b9adb3700 1 -- 172.21.15.29:0/2539771633 wait complete. 2024-01-21T11:42:08.428 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi134 2024-01-21T11:42:08.429 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-01-21T11:42:08.429 DEBUG:teuthology.orchestra.run.smithi134:> dd of=/etc/ceph/ceph.conf 2024-01-21T11:42:08.450 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-01-21T11:42:08.567 DEBUG:teuthology.orchestra.run.smithi134:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-01-21T11:42:08.588 INFO:tasks.cephadm:Adding host smithi134 to orchestrator... 2024-01-21T11:42:08.588 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph orch host add smithi134 2024-01-21T11:42:09.168 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:42:08 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:42:07.824386+0000 mon.smithi029 (mon.0) 109 : cluster [DBG] mgrmap e17: smithi029.blhxxk(active, since 6s) 2024-01-21T11:42:09.169 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:42:08 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:42:08.015377+0000 mon.smithi029 (mon.0) 110 : audit [INF] from='client.? 172.21.15.29:0/2539771633' entity='client.admin' 2024-01-21T11:42:11.344 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:42:11 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:42:10.020274+0000 mon.smithi029 (mon.0) 111 : audit [INF] from='mgr.14164 172.21.15.29:0/3252518221' entity='mgr.smithi029.blhxxk' 2024-01-21T11:42:11.345 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:42:11 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:42:10.243377+0000 mon.smithi029 (mon.0) 112 : audit [INF] from='mgr.14164 172.21.15.29:0/3252518221' entity='mgr.smithi029.blhxxk' 2024-01-21T11:42:11.345 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:42:11 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:42:10.886778+0000 mon.smithi029 (mon.0) 113 : audit [INF] from='mgr.14164 172.21.15.29:0/3252518221' entity='mgr.smithi029.blhxxk' 2024-01-21T11:42:14.668 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:42:14 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:42:13.171918+0000 mon.smithi029 (mon.0) 114 : audit [INF] from='mgr.14164 172.21.15.29:0/3252518221' entity='mgr.smithi029.blhxxk' 2024-01-21T11:42:14.669 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:42:14 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:42:13.451723+0000 mon.smithi029 (mon.0) 115 : audit [INF] from='mgr.14164 172.21.15.29:0/3252518221' entity='mgr.smithi029.blhxxk' 2024-01-21T11:42:14.669 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:42:14 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:42:13.453836+0000 mon.smithi029 (mon.0) 116 : audit [INF] from='mgr.14164 172.21.15.29:0/3252518221' entity='mgr.smithi029.blhxxk' 2024-01-21T11:42:14.669 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:42:14 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cephadm 2024-01-21T11:42:13.460437+0000 mgr.smithi029.blhxxk (mgr.14164) 5 : cephadm [INF] Deploying daemon alertmanager.smithi029 on smithi029 2024-01-21T11:42:23.168 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:42:22 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:42:21.803444+0000 mgr.smithi029.blhxxk (mgr.14164) 6 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:42:25.168 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:42:24 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:42:23.803690+0000 mgr.smithi029.blhxxk (mgr.14164) 7 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:42:27.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:42:27 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:42:25.803952+0000 mgr.smithi029.blhxxk (mgr.14164) 8 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:42:29.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:42:29 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:42:27.804287+0000 mgr.smithi029.blhxxk (mgr.14164) 9 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:42:30.259 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:30.256+0000 7f8796146700 1 -- 172.21.15.134:0/3248388865 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87900f6510 msgr2=0x7f87900f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:42:30.259 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:30.256+0000 7f8796146700 1 --2- 172.21.15.134:0/3248388865 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87900f6510 0x7f87900f6930 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f8784009fb0 tx=0x7f8784009b30).stop 2024-01-21T11:42:30.259 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:30.256+0000 7f8796146700 1 -- 172.21.15.134:0/3248388865 shutdown_connections 2024-01-21T11:42:30.260 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:30.256+0000 7f8796146700 1 --2- 172.21.15.134:0/3248388865 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87900f6510 0x7f87900f6930 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:42:30.260 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:30.256+0000 7f8796146700 1 -- 172.21.15.134:0/3248388865 >> 172.21.15.134:0/3248388865 conn(0x7f87900f1c70 msgr2=0x7f87900f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:42:30.260 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:30.256+0000 7f8796146700 1 -- 172.21.15.134:0/3248388865 shutdown_connections 2024-01-21T11:42:30.260 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:30.257+0000 7f8796146700 1 -- 172.21.15.134:0/3248388865 wait complete. 2024-01-21T11:42:30.260 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:30.257+0000 7f8796146700 1 Processor -- start 2024-01-21T11:42:30.260 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:30.257+0000 7f8796146700 1 -- start start 2024-01-21T11:42:30.260 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:30.257+0000 7f8796146700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87900f6510 0x7f8790106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:42:30.260 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:30.257+0000 7f8796146700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8790106c20 con 0x7f87900f6510 2024-01-21T11:42:30.260 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:30.258+0000 7f878f7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87900f6510 0x7f8790106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:42:30.260 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:30.258+0000 7f878f7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87900f6510 0x7f8790106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:46294/0 (socket says 172.21.15.134:46294) 2024-01-21T11:42:30.260 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:30.258+0000 7f878f7fe700 1 -- 172.21.15.134:0/1950432100 learned_addr learned my addr 172.21.15.134:0/1950432100 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:42:30.260 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:30.258+0000 7f878f7fe700 1 -- 172.21.15.134:0/1950432100 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8784005040 con 0x7f87900f6510 2024-01-21T11:42:30.260 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:30.258+0000 7f878f7fe700 1 --2- 172.21.15.134:0/1950432100 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87900f6510 0x7f8790106730 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f8784013510 tx=0x7f878400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:42:30.261 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:30.258+0000 7f8777fff700 1 -- 172.21.15.134:0/1950432100 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8784009320 con 0x7f87900f6510 2024-01-21T11:42:30.261 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:30.258+0000 7f8796146700 1 -- 172.21.15.134:0/1950432100 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8790106e20 con 0x7f87900f6510 2024-01-21T11:42:30.261 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:30.258+0000 7f8777fff700 1 -- 172.21.15.134:0/1950432100 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8784013f00 con 0x7f87900f6510 2024-01-21T11:42:30.261 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:30.258+0000 7f8777fff700 1 -- 172.21.15.134:0/1950432100 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8784014d90 con 0x7f87900f6510 2024-01-21T11:42:30.261 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:30.258+0000 7f8796146700 1 -- 172.21.15.134:0/1950432100 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87901072b0 con 0x7f87900f6510 2024-01-21T11:42:30.261 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:30.259+0000 7f8777fff700 1 -- 172.21.15.134:0/1950432100 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8784014420 con 0x7f87900f6510 2024-01-21T11:42:30.261 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:30.259+0000 7f8796146700 1 -- 172.21.15.134:0/1950432100 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8790045ad0 con 0x7f87900f6510 2024-01-21T11:42:30.261 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:30.259+0000 7f8777fff700 1 --2- 172.21.15.134:0/1950432100 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f877802ff60 0x7f8778032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:42:30.261 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:30.259+0000 7f8777fff700 1 -- 172.21.15.134:0/1950432100 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8784041070 con 0x7f87900f6510 2024-01-21T11:42:30.261 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:30.259+0000 7f878effd700 1 --2- 172.21.15.134:0/1950432100 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f877802ff60 0x7f8778032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:42:30.261 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:30.260+0000 7f878effd700 1 --2- 172.21.15.134:0/1950432100 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f877802ff60 0x7f8778032420 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f87800096a0 tx=0x7f8780006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:42:30.264 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:30.264+0000 7f8777fff700 1 -- 172.21.15.134:0/1950432100 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f87840095f0 con 0x7f87900f6510 2024-01-21T11:42:30.434 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:30.434+0000 7f8796146700 1 -- 172.21.15.134:0/1950432100 --> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi134", "target": ["mon-mgr", ""]}) v1 -- 0x7f87900f2e70 con 0x7f877802ff60 2024-01-21T11:42:31.036 INFO:teuthology.orchestra.run.smithi134.stdout:Added host 'smithi134' 2024-01-21T11:42:31.036 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:31.035+0000 7f8777fff700 1 -- 172.21.15.134:0/1950432100 <== mgr.14164 v2:172.21.15.29:6800/3399348778 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7f87900f2e70 con 0x7f877802ff60 2024-01-21T11:42:31.038 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:31.037+0000 7f8796146700 1 -- 172.21.15.134:0/1950432100 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f877802ff60 msgr2=0x7f8778032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:42:31.038 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:31.037+0000 7f8796146700 1 --2- 172.21.15.134:0/1950432100 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f877802ff60 0x7f8778032420 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f87800096a0 tx=0x7f8780006b40).stop 2024-01-21T11:42:31.038 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:31.037+0000 7f8796146700 1 -- 172.21.15.134:0/1950432100 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87900f6510 msgr2=0x7f8790106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:42:31.038 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:31.037+0000 7f8796146700 1 --2- 172.21.15.134:0/1950432100 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87900f6510 0x7f8790106730 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f8784013510 tx=0x7f878400b040).stop 2024-01-21T11:42:31.038 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:31.037+0000 7f8796146700 1 -- 172.21.15.134:0/1950432100 shutdown_connections 2024-01-21T11:42:31.038 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:31.037+0000 7f8796146700 1 --2- 172.21.15.134:0/1950432100 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f877802ff60 0x7f8778032420 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:42:31.038 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:31.037+0000 7f8796146700 1 --2- 172.21.15.134:0/1950432100 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87900f6510 0x7f8790106730 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:42:31.038 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:31.037+0000 7f8796146700 1 -- 172.21.15.134:0/1950432100 >> 172.21.15.134:0/1950432100 conn(0x7f87900f1c70 msgr2=0x7f87900f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:42:31.038 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:31.037+0000 7f8796146700 1 -- 172.21.15.134:0/1950432100 shutdown_connections 2024-01-21T11:42:31.038 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:31.037+0000 7f8796146700 1 -- 172.21.15.134:0/1950432100 wait complete. 2024-01-21T11:42:31.413 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph orch host ls --format=json 2024-01-21T11:42:31.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:42:31 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:42:29.804691+0000 mgr.smithi029.blhxxk (mgr.14164) 10 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:42:31.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:42:31 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:42:30.436763+0000 mgr.smithi029.blhxxk (mgr.14164) 11 : audit [DBG] from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi134", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T11:42:31.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:42:31 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:42:31.035544+0000 mon.smithi029 (mon.0) 117 : audit [INF] from='mgr.14164 172.21.15.29:0/3252518221' entity='mgr.smithi029.blhxxk' 2024-01-21T11:42:32.518 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:32.517+0000 7f982edc8700 1 -- 172.21.15.134:0/2783770796 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98280f6510 msgr2=0x7f98280f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:42:32.518 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:32.517+0000 7f982edc8700 1 --2- 172.21.15.134:0/2783770796 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98280f6510 0x7f98280f6930 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f981c009fb0 tx=0x7f981c009b30).stop 2024-01-21T11:42:32.518 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:32.517+0000 7f982edc8700 1 -- 172.21.15.134:0/2783770796 shutdown_connections 2024-01-21T11:42:32.519 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:32.517+0000 7f982edc8700 1 --2- 172.21.15.134:0/2783770796 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98280f6510 0x7f98280f6930 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:42:32.519 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:32.517+0000 7f982edc8700 1 -- 172.21.15.134:0/2783770796 >> 172.21.15.134:0/2783770796 conn(0x7f98280f1c70 msgr2=0x7f98280f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:42:32.519 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:32.517+0000 7f982edc8700 1 -- 172.21.15.134:0/2783770796 shutdown_connections 2024-01-21T11:42:32.519 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:32.517+0000 7f982edc8700 1 -- 172.21.15.134:0/2783770796 wait complete. 2024-01-21T11:42:32.519 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:32.518+0000 7f982edc8700 1 Processor -- start 2024-01-21T11:42:32.519 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:32.518+0000 7f982edc8700 1 -- start start 2024-01-21T11:42:32.519 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:32.518+0000 7f982edc8700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98280f6510 0x7f9828106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:42:32.519 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:32.518+0000 7f982edc8700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9828106c20 con 0x7f98280f6510 2024-01-21T11:42:32.519 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:32.518+0000 7f982cb64700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98280f6510 0x7f9828106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:42:32.519 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:32.518+0000 7f982cb64700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98280f6510 0x7f9828106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:46304/0 (socket says 172.21.15.134:46304) 2024-01-21T11:42:32.519 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:32.518+0000 7f982cb64700 1 -- 172.21.15.134:0/4164973379 learned_addr learned my addr 172.21.15.134:0/4164973379 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:42:32.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:32.519+0000 7f982cb64700 1 -- 172.21.15.134:0/4164973379 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f981c005040 con 0x7f98280f6510 2024-01-21T11:42:32.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:32.519+0000 7f982cb64700 1 --2- 172.21.15.134:0/4164973379 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98280f6510 0x7f9828106730 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f981c013510 tx=0x7f981c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:42:32.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:32.519+0000 7f98257fa700 1 -- 172.21.15.134:0/4164973379 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f981c009320 con 0x7f98280f6510 2024-01-21T11:42:32.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:32.519+0000 7f982edc8700 1 -- 172.21.15.134:0/4164973379 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9828106e20 con 0x7f98280f6510 2024-01-21T11:42:32.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:32.519+0000 7f98257fa700 1 -- 172.21.15.134:0/4164973379 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f981c013f00 con 0x7f98280f6510 2024-01-21T11:42:32.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:32.519+0000 7f98257fa700 1 -- 172.21.15.134:0/4164973379 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f981c014d90 con 0x7f98280f6510 2024-01-21T11:42:32.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:32.519+0000 7f982edc8700 1 -- 172.21.15.134:0/4164973379 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98281072b0 con 0x7f98280f6510 2024-01-21T11:42:32.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:32.520+0000 7f98257fa700 1 -- 172.21.15.134:0/4164973379 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f981c014420 con 0x7f98280f6510 2024-01-21T11:42:32.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:32.520+0000 7f982edc8700 1 -- 172.21.15.134:0/4164973379 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9828045ad0 con 0x7f98280f6510 2024-01-21T11:42:32.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:32.520+0000 7f98257fa700 1 --2- 172.21.15.134:0/4164973379 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f981802ff60 0x7f9818032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:42:32.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:32.520+0000 7f98257fa700 1 -- 172.21.15.134:0/4164973379 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f981c041000 con 0x7f98280f6510 2024-01-21T11:42:32.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:32.520+0000 7f9827fff700 1 --2- 172.21.15.134:0/4164973379 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f981802ff60 0x7f9818032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:42:32.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:32.521+0000 7f9827fff700 1 --2- 172.21.15.134:0/4164973379 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f981802ff60 0x7f9818032420 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f98140096a0 tx=0x7f9814006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:42:32.525 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:32.525+0000 7f98257fa700 1 -- 172.21.15.134:0/4164973379 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f981c0095f0 con 0x7f98280f6510 2024-01-21T11:42:32.696 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:32.696+0000 7f982edc8700 1 -- 172.21.15.134:0/4164973379 --> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f98280f2e70 con 0x7f981802ff60 2024-01-21T11:42:32.697 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:32.697+0000 7f98257fa700 1 -- 172.21.15.134:0/4164973379 <== mgr.14164 v2:172.21.15.29:6800/3399348778 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+156 (secure 0 0 0) 0x7f98280f2e70 con 0x7f981802ff60 2024-01-21T11:42:32.698 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:42:32.698 INFO:teuthology.orchestra.run.smithi134.stdout:[{"addr": "172.21.15.29", "hostname": "smithi029", "labels": [], "status": ""}, {"addr": "smithi134", "hostname": "smithi134", "labels": [], "status": ""}] 2024-01-21T11:42:32.699 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:32.699+0000 7f982edc8700 1 -- 172.21.15.134:0/4164973379 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f981802ff60 msgr2=0x7f9818032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:42:32.699 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:32.699+0000 7f982edc8700 1 --2- 172.21.15.134:0/4164973379 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f981802ff60 0x7f9818032420 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f98140096a0 tx=0x7f9814006b40).stop 2024-01-21T11:42:32.700 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:32.699+0000 7f982edc8700 1 -- 172.21.15.134:0/4164973379 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98280f6510 msgr2=0x7f9828106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:42:32.700 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:32.699+0000 7f982edc8700 1 --2- 172.21.15.134:0/4164973379 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98280f6510 0x7f9828106730 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f981c013510 tx=0x7f981c00b040).stop 2024-01-21T11:42:32.700 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:32.699+0000 7f982edc8700 1 -- 172.21.15.134:0/4164973379 shutdown_connections 2024-01-21T11:42:32.700 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:32.699+0000 7f982edc8700 1 --2- 172.21.15.134:0/4164973379 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f981802ff60 0x7f9818032420 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:42:32.700 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:32.699+0000 7f982edc8700 1 --2- 172.21.15.134:0/4164973379 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98280f6510 0x7f9828106730 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:42:32.700 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:32.699+0000 7f982edc8700 1 -- 172.21.15.134:0/4164973379 >> 172.21.15.134:0/4164973379 conn(0x7f98280f1c70 msgr2=0x7f98280f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:42:32.700 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:32.699+0000 7f982edc8700 1 -- 172.21.15.134:0/4164973379 shutdown_connections 2024-01-21T11:42:32.700 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:32.699+0000 7f982edc8700 1 -- 172.21.15.134:0/4164973379 wait complete. 2024-01-21T11:42:32.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:42:32 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cephadm 2024-01-21T11:42:31.035958+0000 mgr.smithi029.blhxxk (mgr.14164) 12 : cephadm [INF] Added host smithi134 2024-01-21T11:42:33.088 INFO:tasks.cephadm:Setting crush tunables to default 2024-01-21T11:42:33.088 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph osd crush tunables default 2024-01-21T11:42:33.669 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:42:33 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:42:31.804959+0000 mgr.smithi029.blhxxk (mgr.14164) 13 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:42:34.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-21T11:42:34.280+0000 7fc93d424700 1 -- 172.21.15.29:0/2388754281 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9380f5410 msgr2=0x7fc9380f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:42:34.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-21T11:42:34.280+0000 7fc93d424700 1 --2- 172.21.15.29:0/2388754281 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9380f5410 0x7fc9380f5830 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7fc920009fb0 tx=0x7fc920009b30).stop 2024-01-21T11:42:34.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-21T11:42:34.280+0000 7fc93d424700 1 -- 172.21.15.29:0/2388754281 shutdown_connections 2024-01-21T11:42:34.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-21T11:42:34.280+0000 7fc93d424700 1 --2- 172.21.15.29:0/2388754281 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9380f5410 0x7fc9380f5830 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:42:34.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-21T11:42:34.280+0000 7fc93d424700 1 -- 172.21.15.29:0/2388754281 >> 172.21.15.29:0/2388754281 conn(0x7fc9380f0b70 msgr2=0x7fc9380f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:42:34.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-21T11:42:34.280+0000 7fc93d424700 1 -- 172.21.15.29:0/2388754281 shutdown_connections 2024-01-21T11:42:34.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-21T11:42:34.280+0000 7fc93d424700 1 -- 172.21.15.29:0/2388754281 wait complete. 2024-01-21T11:42:34.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-21T11:42:34.281+0000 7fc93d424700 1 Processor -- start 2024-01-21T11:42:34.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-21T11:42:34.281+0000 7fc93d424700 1 -- start start 2024-01-21T11:42:34.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-21T11:42:34.281+0000 7fc93d424700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9380f5410 0x7fc938109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:42:34.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-21T11:42:34.281+0000 7fc93d424700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc938060220 con 0x7fc9380f5410 2024-01-21T11:42:34.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-21T11:42:34.281+0000 7fc936ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9380f5410 0x7fc938109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:42:34.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-21T11:42:34.281+0000 7fc936ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9380f5410 0x7fc938109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58062/0 (socket says 172.21.15.29:58062) 2024-01-21T11:42:34.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-21T11:42:34.281+0000 7fc936ffd700 1 -- 172.21.15.29:0/3401005765 learned_addr learned my addr 172.21.15.29:0/3401005765 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-21T11:42:34.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-21T11:42:34.282+0000 7fc936ffd700 1 -- 172.21.15.29:0/3401005765 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc920005040 con 0x7fc9380f5410 2024-01-21T11:42:34.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-21T11:42:34.282+0000 7fc936ffd700 1 --2- 172.21.15.29:0/3401005765 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9380f5410 0x7fc938109450 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7fc920013510 tx=0x7fc92000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:42:34.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-21T11:42:34.282+0000 7fc92f7fe700 1 -- 172.21.15.29:0/3401005765 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc920009320 con 0x7fc9380f5410 2024-01-21T11:42:34.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-21T11:42:34.282+0000 7fc92f7fe700 1 -- 172.21.15.29:0/3401005765 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc920013f00 con 0x7fc9380f5410 2024-01-21T11:42:34.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-21T11:42:34.282+0000 7fc92f7fe700 1 -- 172.21.15.29:0/3401005765 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc920014d90 con 0x7fc9380f5410 2024-01-21T11:42:34.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-21T11:42:34.282+0000 7fc93d424700 1 -- 172.21.15.29:0/3401005765 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc938060420 con 0x7fc9380f5410 2024-01-21T11:42:34.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-21T11:42:34.282+0000 7fc93d424700 1 -- 172.21.15.29:0/3401005765 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc938109b00 con 0x7fc9380f5410 2024-01-21T11:42:34.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-21T11:42:34.283+0000 7fc92f7fe700 1 -- 172.21.15.29:0/3401005765 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc9200263a0 con 0x7fc9380f5410 2024-01-21T11:42:34.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-21T11:42:34.283+0000 7fc92f7fe700 1 --2- 172.21.15.29:0/3401005765 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fc92402ff60 0x7fc924032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:42:34.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-21T11:42:34.283+0000 7fc93d424700 1 -- 172.21.15.29:0/3401005765 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc938059100 con 0x7fc9380f5410 2024-01-21T11:42:34.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-21T11:42:34.283+0000 7fc92f7fe700 1 -- 172.21.15.29:0/3401005765 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc9200412f0 con 0x7fc9380f5410 2024-01-21T11:42:34.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-21T11:42:34.283+0000 7fc9367fc700 1 --2- 172.21.15.29:0/3401005765 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fc92402ff60 0x7fc924032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:42:34.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-21T11:42:34.284+0000 7fc9367fc700 1 --2- 172.21.15.29:0/3401005765 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fc92402ff60 0x7fc924032420 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7fc9280096a0 tx=0x7fc928006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:42:34.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-21T11:42:34.287+0000 7fc92f7fe700 1 -- 172.21.15.29:0/3401005765 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc92001cb10 con 0x7fc9380f5410 2024-01-21T11:42:34.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-21T11:42:34.452+0000 7fc93d424700 1 -- 172.21.15.29:0/3401005765 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fc938045ad0 con 0x7fc9380f5410 2024-01-21T11:42:34.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-21T11:42:34.510+0000 7fc92f7fe700 1 -- 172.21.15.29:0/3401005765 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v5) v1 ==== 124+0+0 (secure 0 0 0) 0x7fc92002b070 con 0x7fc9380f5410 2024-01-21T11:42:34.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-21T11:42:34.512+0000 7fc93d424700 1 -- 172.21.15.29:0/3401005765 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fc92402ff60 msgr2=0x7fc924032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:42:34.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-21T11:42:34.512+0000 7fc93d424700 1 --2- 172.21.15.29:0/3401005765 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fc92402ff60 0x7fc924032420 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7fc9280096a0 tx=0x7fc928006b40).stop 2024-01-21T11:42:34.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-21T11:42:34.512+0000 7fc93d424700 1 -- 172.21.15.29:0/3401005765 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9380f5410 msgr2=0x7fc938109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:42:34.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-21T11:42:34.512+0000 7fc93d424700 1 --2- 172.21.15.29:0/3401005765 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9380f5410 0x7fc938109450 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7fc920013510 tx=0x7fc92000b040).stop 2024-01-21T11:42:34.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-21T11:42:34.512+0000 7fc93d424700 1 -- 172.21.15.29:0/3401005765 shutdown_connections 2024-01-21T11:42:34.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-21T11:42:34.512+0000 7fc93d424700 1 --2- 172.21.15.29:0/3401005765 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fc92402ff60 0x7fc924032420 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:42:34.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-21T11:42:34.512+0000 7fc93d424700 1 --2- 172.21.15.29:0/3401005765 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9380f5410 0x7fc938109450 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:42:34.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-21T11:42:34.512+0000 7fc93d424700 1 -- 172.21.15.29:0/3401005765 >> 172.21.15.29:0/3401005765 conn(0x7fc9380f0b70 msgr2=0x7fc9380f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:42:34.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-21T11:42:34.512+0000 7fc93d424700 1 -- 172.21.15.29:0/3401005765 shutdown_connections 2024-01-21T11:42:34.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-21T11:42:34.512+0000 7fc93d424700 1 -- 172.21.15.29:0/3401005765 wait complete. 2024-01-21T11:42:34.513 INFO:teuthology.orchestra.run.smithi029.stderr:adjusted tunables profile to default 2024-01-21T11:42:34.780 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:42:34 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:42:32.698178+0000 mgr.smithi029.blhxxk (mgr.14164) 14 : audit [DBG] from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-01-21T11:42:34.780 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:42:34 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:42:34.453205+0000 mon.smithi029 (mon.0) 118 : audit [INF] from='client.? 172.21.15.29:0/3401005765' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-01-21T11:42:34.890 INFO:tasks.cephadm:Adding mon.smithi029 on smithi029 2024-01-21T11:42:34.890 INFO:tasks.cephadm:Adding mon.smithi134 on smithi134 2024-01-21T11:42:34.890 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph orch apply mon '2;smithi029:172.21.15.29=smithi029;smithi134:172.21.15.134=smithi134' 2024-01-21T11:42:35.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:42:35 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:42:33.805343+0000 mgr.smithi029.blhxxk (mgr.14164) 15 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:42:35.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:42:35 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:42:34.511521+0000 mon.smithi029 (mon.0) 119 : audit [INF] from='client.? 172.21.15.29:0/3401005765' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-01-21T11:42:35.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:42:35 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:42:34.511564+0000 mon.smithi029 (mon.0) 120 : cluster [DBG] osdmap e5: 0 total, 0 up, 0 in 2024-01-21T11:42:36.055 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:36.054+0000 7f936839d700 1 -- 172.21.15.134:0/526865564 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93600f5410 msgr2=0x7f93600f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:42:36.055 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:36.054+0000 7f936839d700 1 --2- 172.21.15.134:0/526865564 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93600f5410 0x7f93600f5830 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f935c009fb0 tx=0x7f935c009b30).stop 2024-01-21T11:42:36.055 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:36.054+0000 7f936839d700 1 -- 172.21.15.134:0/526865564 shutdown_connections 2024-01-21T11:42:36.055 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:36.054+0000 7f936839d700 1 --2- 172.21.15.134:0/526865564 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93600f5410 0x7f93600f5830 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:42:36.055 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:36.054+0000 7f936839d700 1 -- 172.21.15.134:0/526865564 >> 172.21.15.134:0/526865564 conn(0x7f93600f0b70 msgr2=0x7f93600f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:42:36.056 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:36.054+0000 7f936839d700 1 -- 172.21.15.134:0/526865564 shutdown_connections 2024-01-21T11:42:36.056 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:36.054+0000 7f936839d700 1 -- 172.21.15.134:0/526865564 wait complete. 2024-01-21T11:42:36.056 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:36.054+0000 7f936839d700 1 Processor -- start 2024-01-21T11:42:36.056 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:36.054+0000 7f936839d700 1 -- start start 2024-01-21T11:42:36.056 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:36.055+0000 7f936839d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93600f5410 0x7f9360105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:42:36.056 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:36.055+0000 7f936839d700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9360105b20 con 0x7f93600f5410 2024-01-21T11:42:36.056 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:36.055+0000 7f9366139700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93600f5410 0x7f9360105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:42:36.056 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:36.055+0000 7f9366139700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93600f5410 0x7f9360105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:45338/0 (socket says 172.21.15.134:45338) 2024-01-21T11:42:36.056 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:36.055+0000 7f9366139700 1 -- 172.21.15.134:0/4196934600 learned_addr learned my addr 172.21.15.134:0/4196934600 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:42:36.056 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:36.055+0000 7f9366139700 1 -- 172.21.15.134:0/4196934600 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f935c005040 con 0x7f93600f5410 2024-01-21T11:42:36.056 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:36.055+0000 7f9366139700 1 --2- 172.21.15.134:0/4196934600 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93600f5410 0x7f9360105630 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f935c013510 tx=0x7f935c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:42:36.057 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:36.055+0000 7f9356ffd700 1 -- 172.21.15.134:0/4196934600 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f935c009320 con 0x7f93600f5410 2024-01-21T11:42:36.057 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:36.056+0000 7f936839d700 1 -- 172.21.15.134:0/4196934600 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9360105d20 con 0x7f93600f5410 2024-01-21T11:42:36.058 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:36.056+0000 7f9356ffd700 1 -- 172.21.15.134:0/4196934600 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f935c013f00 con 0x7f93600f5410 2024-01-21T11:42:36.058 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:36.056+0000 7f9356ffd700 1 -- 172.21.15.134:0/4196934600 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f935c014d90 con 0x7f93600f5410 2024-01-21T11:42:36.058 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:36.056+0000 7f936839d700 1 -- 172.21.15.134:0/4196934600 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93601061b0 con 0x7f93600f5410 2024-01-21T11:42:36.058 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:36.056+0000 7f9356ffd700 1 -- 172.21.15.134:0/4196934600 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f935c0095f0 con 0x7f93600f5410 2024-01-21T11:42:36.058 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:36.056+0000 7f936839d700 1 -- 172.21.15.134:0/4196934600 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9360045ad0 con 0x7f93600f5410 2024-01-21T11:42:36.058 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:36.056+0000 7f9356ffd700 1 --2- 172.21.15.134:0/4196934600 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f934c02ff60 0x7f934c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:42:36.058 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:36.057+0000 7f9356ffd700 1 -- 172.21.15.134:0/4196934600 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f935c041000 con 0x7f93600f5410 2024-01-21T11:42:36.058 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:36.057+0000 7f9365938700 1 --2- 172.21.15.134:0/4196934600 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f934c02ff60 0x7f934c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:42:36.058 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:36.057+0000 7f9365938700 1 --2- 172.21.15.134:0/4196934600 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f934c02ff60 0x7f934c032420 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f93500096a0 tx=0x7f9350006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:42:36.061 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:36.061+0000 7f9356ffd700 1 -- 172.21.15.134:0/4196934600 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f935c014420 con 0x7f93600f5410 2024-01-21T11:42:36.240 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:36.240+0000 7f936839d700 1 -- 172.21.15.134:0/4196934600 --> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi029:172.21.15.29=smithi029;smithi134:172.21.15.134=smithi134", "target": ["mon-mgr", ""]}) v1 -- 0x7f93600f1e20 con 0x7f934c02ff60 2024-01-21T11:42:36.245 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:36.245+0000 7f9356ffd700 1 -- 172.21.15.134:0/4196934600 <== mgr.14164 v2:172.21.15.29:6800/3399348778 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f93600f1e20 con 0x7f934c02ff60 2024-01-21T11:42:36.245 INFO:teuthology.orchestra.run.smithi134.stdout:Scheduled mon update... 2024-01-21T11:42:36.247 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:36.246+0000 7f936839d700 1 -- 172.21.15.134:0/4196934600 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f934c02ff60 msgr2=0x7f934c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:42:36.247 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:36.246+0000 7f936839d700 1 --2- 172.21.15.134:0/4196934600 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f934c02ff60 0x7f934c032420 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f93500096a0 tx=0x7f9350006b40).stop 2024-01-21T11:42:36.247 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:36.246+0000 7f936839d700 1 -- 172.21.15.134:0/4196934600 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93600f5410 msgr2=0x7f9360105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:42:36.247 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:36.246+0000 7f936839d700 1 --2- 172.21.15.134:0/4196934600 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93600f5410 0x7f9360105630 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f935c013510 tx=0x7f935c00b040).stop 2024-01-21T11:42:36.247 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:36.247+0000 7f936839d700 1 -- 172.21.15.134:0/4196934600 shutdown_connections 2024-01-21T11:42:36.247 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:36.247+0000 7f936839d700 1 --2- 172.21.15.134:0/4196934600 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f934c02ff60 0x7f934c032420 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:42:36.247 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:36.247+0000 7f936839d700 1 --2- 172.21.15.134:0/4196934600 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93600f5410 0x7f9360105630 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:42:36.247 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:36.247+0000 7f936839d700 1 -- 172.21.15.134:0/4196934600 >> 172.21.15.134:0/4196934600 conn(0x7f93600f0b70 msgr2=0x7f93600f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:42:36.247 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:36.247+0000 7f936839d700 1 -- 172.21.15.134:0/4196934600 shutdown_connections 2024-01-21T11:42:36.248 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:36.247+0000 7f936839d700 1 -- 172.21.15.134:0/4196934600 wait complete. 2024-01-21T11:42:36.624 DEBUG:teuthology.orchestra.run.smithi134:mon.smithi134> sudo journalctl -f -n 0 -u ceph-e4084074-b851-11ee-95b0-87774f69a715@mon.smithi134.service 2024-01-21T11:42:36.627 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:42:36.627 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:42:36.655 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:-- Logs begin at Sun 2024-01-21 11:24:35 UTC. -- 2024-01-21T11:42:37.668 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:42:37 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:42:35.805674+0000 mgr.smithi029.blhxxk (mgr.14164) 16 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:42:37.669 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:42:37 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:42:36.241876+0000 mgr.smithi029.blhxxk (mgr.14164) 17 : audit [DBG] from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi029:172.21.15.29=smithi029;smithi134:172.21.15.134=smithi134", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T11:42:37.669 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:42:37 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cephadm 2024-01-21T11:42:36.243281+0000 mgr.smithi029.blhxxk (mgr.14164) 18 : cephadm [INF] Saving service mon spec with placement smithi029:172.21.15.29=smithi029;smithi134:172.21.15.134=smithi134;count:2 2024-01-21T11:42:37.669 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:42:37 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:42:36.245962+0000 mon.smithi029 (mon.0) 121 : audit [INF] from='mgr.14164 172.21.15.29:0/3252518221' entity='mgr.smithi029.blhxxk' 2024-01-21T11:42:37.783 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:37.782+0000 7fb7c10d3700 1 -- 172.21.15.134:0/4241719481 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7bc0f5410 msgr2=0x7fb7bc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:42:37.783 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:37.782+0000 7fb7c10d3700 1 --2- 172.21.15.134:0/4241719481 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7bc0f5410 0x7fb7bc0f5830 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7fb7b0009fb0 tx=0x7fb7b0009b30).stop 2024-01-21T11:42:37.783 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:37.782+0000 7fb7c10d3700 1 -- 172.21.15.134:0/4241719481 shutdown_connections 2024-01-21T11:42:37.783 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:37.782+0000 7fb7c10d3700 1 --2- 172.21.15.134:0/4241719481 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7bc0f5410 0x7fb7bc0f5830 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:42:37.783 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:37.782+0000 7fb7c10d3700 1 -- 172.21.15.134:0/4241719481 >> 172.21.15.134:0/4241719481 conn(0x7fb7bc0f0b70 msgr2=0x7fb7bc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:42:37.784 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:37.782+0000 7fb7c10d3700 1 -- 172.21.15.134:0/4241719481 shutdown_connections 2024-01-21T11:42:37.784 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:37.782+0000 7fb7c10d3700 1 -- 172.21.15.134:0/4241719481 wait complete. 2024-01-21T11:42:37.784 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:37.783+0000 7fb7c10d3700 1 Processor -- start 2024-01-21T11:42:37.784 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:37.783+0000 7fb7c10d3700 1 -- start start 2024-01-21T11:42:37.784 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:37.783+0000 7fb7c10d3700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7bc0f5410 0x7fb7bc109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:42:37.784 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:37.783+0000 7fb7c10d3700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7bc060220 con 0x7fb7bc0f5410 2024-01-21T11:42:37.784 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:37.783+0000 7fb7bad9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7bc0f5410 0x7fb7bc109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:42:37.784 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:37.783+0000 7fb7bad9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7bc0f5410 0x7fb7bc109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:45352/0 (socket says 172.21.15.134:45352) 2024-01-21T11:42:37.784 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:37.783+0000 7fb7bad9d700 1 -- 172.21.15.134:0/527152289 learned_addr learned my addr 172.21.15.134:0/527152289 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:42:37.784 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:37.783+0000 7fb7bad9d700 1 -- 172.21.15.134:0/527152289 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7b0005040 con 0x7fb7bc0f5410 2024-01-21T11:42:37.784 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:37.783+0000 7fb7bad9d700 1 --2- 172.21.15.134:0/527152289 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7bc0f5410 0x7fb7bc109450 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7fb7b0013510 tx=0x7fb7b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:42:37.785 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:37.784+0000 7fb7a37fe700 1 -- 172.21.15.134:0/527152289 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb7b0009320 con 0x7fb7bc0f5410 2024-01-21T11:42:37.785 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:37.784+0000 7fb7c10d3700 1 -- 172.21.15.134:0/527152289 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7bc060420 con 0x7fb7bc0f5410 2024-01-21T11:42:37.786 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:37.784+0000 7fb7a37fe700 1 -- 172.21.15.134:0/527152289 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb7b0013f00 con 0x7fb7bc0f5410 2024-01-21T11:42:37.786 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:37.784+0000 7fb7a37fe700 1 -- 172.21.15.134:0/527152289 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb7b0014d90 con 0x7fb7bc0f5410 2024-01-21T11:42:37.786 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:37.784+0000 7fb7c10d3700 1 -- 172.21.15.134:0/527152289 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7bc109b00 con 0x7fb7bc0f5410 2024-01-21T11:42:37.786 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:37.784+0000 7fb7a37fe700 1 -- 172.21.15.134:0/527152289 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb7b00263a0 con 0x7fb7bc0f5410 2024-01-21T11:42:37.786 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:37.785+0000 7fb7a37fe700 1 --2- 172.21.15.134:0/527152289 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fb7a402ff60 0x7fb7a4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:42:37.786 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:37.785+0000 7fb7c10d3700 1 -- 172.21.15.134:0/527152289 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7bc059100 con 0x7fb7bc0f5410 2024-01-21T11:42:37.786 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:37.785+0000 7fb7a37fe700 1 -- 172.21.15.134:0/527152289 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb7b0040ff0 con 0x7fb7bc0f5410 2024-01-21T11:42:37.786 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:37.785+0000 7fb7ba59c700 1 --2- 172.21.15.134:0/527152289 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fb7a402ff60 0x7fb7a4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:42:37.786 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:37.785+0000 7fb7ba59c700 1 --2- 172.21.15.134:0/527152289 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fb7a402ff60 0x7fb7a4032420 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fb7ac0096a0 tx=0x7fb7ac006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:42:37.789 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:37.789+0000 7fb7a37fe700 1 -- 172.21.15.134:0/527152289 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb7b0012920 con 0x7fb7bc0f5410 2024-01-21T11:42:37.992 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:37.991+0000 7fb7c10d3700 1 -- 172.21.15.134:0/527152289 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb7bc045ad0 con 0x7fb7bc0f5410 2024-01-21T11:42:37.992 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:37.992+0000 7fb7a37fe700 1 -- 172.21.15.134:0/527152289 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb7b002b020 con 0x7fb7bc0f5410 2024-01-21T11:42:37.993 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:42:37.993 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:42:37.994 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:37.994+0000 7fb7c10d3700 1 -- 172.21.15.134:0/527152289 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fb7a402ff60 msgr2=0x7fb7a4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:42:37.994 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:37.994+0000 7fb7c10d3700 1 --2- 172.21.15.134:0/527152289 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fb7a402ff60 0x7fb7a4032420 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fb7ac0096a0 tx=0x7fb7ac006b40).stop 2024-01-21T11:42:37.994 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:37.994+0000 7fb7c10d3700 1 -- 172.21.15.134:0/527152289 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7bc0f5410 msgr2=0x7fb7bc109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:42:37.994 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:37.994+0000 7fb7c10d3700 1 --2- 172.21.15.134:0/527152289 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7bc0f5410 0x7fb7bc109450 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7fb7b0013510 tx=0x7fb7b000b040).stop 2024-01-21T11:42:37.994 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:37.994+0000 7fb7c10d3700 1 -- 172.21.15.134:0/527152289 shutdown_connections 2024-01-21T11:42:37.994 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:37.994+0000 7fb7c10d3700 1 --2- 172.21.15.134:0/527152289 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fb7a402ff60 0x7fb7a4032420 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:42:37.994 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:37.994+0000 7fb7c10d3700 1 --2- 172.21.15.134:0/527152289 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7bc0f5410 0x7fb7bc109450 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:42:37.994 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:37.994+0000 7fb7c10d3700 1 -- 172.21.15.134:0/527152289 >> 172.21.15.134:0/527152289 conn(0x7fb7bc0f0b70 msgr2=0x7fb7bc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:42:37.995 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:37.994+0000 7fb7c10d3700 1 -- 172.21.15.134:0/527152289 shutdown_connections 2024-01-21T11:42:37.995 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:37.994+0000 7fb7c10d3700 1 -- 172.21.15.134:0/527152289 wait complete. 2024-01-21T11:42:37.995 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:42:38.668 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:42:38 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:42:37.993298+0000 mon.smithi029 (mon.0) 122 : audit [DBG] from='client.? 172.21.15.134:0/527152289' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:42:39.412 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:42:39.412 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:42:39.668 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:42:39 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:42:37.805951+0000 mgr.smithi029.blhxxk (mgr.14164) 19 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:42:40.939 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:40.937+0000 7f2e18f80700 1 -- 172.21.15.134:0/674269102 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2e140f5410 msgr2=0x7f2e140f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:42:40.939 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:40.937+0000 7f2e18f80700 1 --2- 172.21.15.134:0/674269102 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2e140f5410 0x7f2e140f5830 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f2dfc009fb0 tx=0x7f2dfc009b30).stop 2024-01-21T11:42:40.939 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:40.937+0000 7f2e18f80700 1 -- 172.21.15.134:0/674269102 shutdown_connections 2024-01-21T11:42:40.939 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:40.937+0000 7f2e18f80700 1 --2- 172.21.15.134:0/674269102 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2e140f5410 0x7f2e140f5830 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:42:40.940 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:40.937+0000 7f2e18f80700 1 -- 172.21.15.134:0/674269102 >> 172.21.15.134:0/674269102 conn(0x7f2e140f0b70 msgr2=0x7f2e140f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:42:40.940 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:40.937+0000 7f2e18f80700 1 -- 172.21.15.134:0/674269102 shutdown_connections 2024-01-21T11:42:40.940 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:40.937+0000 7f2e18f80700 1 -- 172.21.15.134:0/674269102 wait complete. 2024-01-21T11:42:40.940 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:40.938+0000 7f2e18f80700 1 Processor -- start 2024-01-21T11:42:40.940 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:40.938+0000 7f2e18f80700 1 -- start start 2024-01-21T11:42:40.940 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:40.938+0000 7f2e18f80700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2e140f5410 0x7f2e14105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:42:40.940 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:40.938+0000 7f2e18f80700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e14105b20 con 0x7f2e140f5410 2024-01-21T11:42:40.940 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:40.938+0000 7f2e1259c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2e140f5410 0x7f2e14105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:42:40.940 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:40.938+0000 7f2e1259c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2e140f5410 0x7f2e14105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:45374/0 (socket says 172.21.15.134:45374) 2024-01-21T11:42:40.940 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:40.938+0000 7f2e1259c700 1 -- 172.21.15.134:0/1136520912 learned_addr learned my addr 172.21.15.134:0/1136520912 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:42:40.940 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:40.939+0000 7f2e1259c700 1 -- 172.21.15.134:0/1136520912 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dfc005040 con 0x7f2e140f5410 2024-01-21T11:42:40.940 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:40.939+0000 7f2e1259c700 1 --2- 172.21.15.134:0/1136520912 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2e140f5410 0x7f2e14105630 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f2dfc013510 tx=0x7f2dfc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:42:40.940 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:40.939+0000 7f2e0affd700 1 -- 172.21.15.134:0/1136520912 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2dfc009320 con 0x7f2e140f5410 2024-01-21T11:42:40.940 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:40.939+0000 7f2e0affd700 1 -- 172.21.15.134:0/1136520912 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2dfc013f00 con 0x7f2e140f5410 2024-01-21T11:42:40.941 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:40.939+0000 7f2e0affd700 1 -- 172.21.15.134:0/1136520912 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2dfc014d90 con 0x7f2e140f5410 2024-01-21T11:42:40.941 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:40.939+0000 7f2e18f80700 1 -- 172.21.15.134:0/1136520912 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e14105d20 con 0x7f2e140f5410 2024-01-21T11:42:40.941 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:40.939+0000 7f2e18f80700 1 -- 172.21.15.134:0/1136520912 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e141061b0 con 0x7f2e140f5410 2024-01-21T11:42:40.941 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:40.940+0000 7f2e0affd700 1 -- 172.21.15.134:0/1136520912 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2dfc0263a0 con 0x7f2e140f5410 2024-01-21T11:42:40.942 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:40.940+0000 7f2e0affd700 1 --2- 172.21.15.134:0/1136520912 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f2e0002ff60 0x7f2e00032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:42:40.942 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:40.940+0000 7f2e0affd700 1 -- 172.21.15.134:0/1136520912 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2dfc041300 con 0x7f2e140f5410 2024-01-21T11:42:40.942 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:40.940+0000 7f2e18f80700 1 -- 172.21.15.134:0/1136520912 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e14045ad0 con 0x7f2e140f5410 2024-01-21T11:42:40.942 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:40.940+0000 7f2e11d9b700 1 --2- 172.21.15.134:0/1136520912 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f2e0002ff60 0x7f2e00032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:42:40.942 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:40.941+0000 7f2e11d9b700 1 --2- 172.21.15.134:0/1136520912 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f2e0002ff60 0x7f2e00032420 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f2e040096a0 tx=0x7f2e04006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:42:40.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:40.945+0000 7f2e0affd700 1 -- 172.21.15.134:0/1136520912 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2dfc012920 con 0x7f2e140f5410 2024-01-21T11:42:41.148 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:41.148+0000 7f2e18f80700 1 -- 172.21.15.134:0/1136520912 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2e140fae60 con 0x7f2e140f5410 2024-01-21T11:42:41.149 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:41.149+0000 7f2e0affd700 1 -- 172.21.15.134:0/1136520912 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2dfc02b030 con 0x7f2e140f5410 2024-01-21T11:42:41.149 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:42:41.149 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:42:41.151 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:41.151+0000 7f2e18f80700 1 -- 172.21.15.134:0/1136520912 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f2e0002ff60 msgr2=0x7f2e00032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:42:41.151 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:41.151+0000 7f2e18f80700 1 --2- 172.21.15.134:0/1136520912 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f2e0002ff60 0x7f2e00032420 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f2e040096a0 tx=0x7f2e04006b40).stop 2024-01-21T11:42:41.151 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:41.151+0000 7f2e18f80700 1 -- 172.21.15.134:0/1136520912 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2e140f5410 msgr2=0x7f2e14105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:42:41.151 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:41.151+0000 7f2e18f80700 1 --2- 172.21.15.134:0/1136520912 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2e140f5410 0x7f2e14105630 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f2dfc013510 tx=0x7f2dfc00b040).stop 2024-01-21T11:42:41.151 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:41.151+0000 7f2e18f80700 1 -- 172.21.15.134:0/1136520912 shutdown_connections 2024-01-21T11:42:41.151 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:41.151+0000 7f2e18f80700 1 --2- 172.21.15.134:0/1136520912 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f2e0002ff60 0x7f2e00032420 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:42:41.151 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:41.151+0000 7f2e18f80700 1 --2- 172.21.15.134:0/1136520912 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2e140f5410 0x7f2e14105630 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:42:41.151 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:41.151+0000 7f2e18f80700 1 -- 172.21.15.134:0/1136520912 >> 172.21.15.134:0/1136520912 conn(0x7f2e140f0b70 msgr2=0x7f2e140f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:42:41.151 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:41.151+0000 7f2e18f80700 1 -- 172.21.15.134:0/1136520912 shutdown_connections 2024-01-21T11:42:41.152 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:41.151+0000 7f2e18f80700 1 -- 172.21.15.134:0/1136520912 wait complete. 2024-01-21T11:42:41.152 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:42:41.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:42:41 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:42:39.806214+0000 mgr.smithi029.blhxxk (mgr.14164) 20 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:42:41.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:42:41 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:42:41.150375+0000 mon.smithi029 (mon.0) 123 : audit [DBG] from='client.? 172.21.15.134:0/1136520912' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:42:42.550 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:42:42.550 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:42:43.741 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:43.739+0000 7fc7fba7d700 1 -- 172.21.15.134:0/1327536747 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7f40f5410 msgr2=0x7fc7f40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:42:43.741 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:43.739+0000 7fc7fba7d700 1 --2- 172.21.15.134:0/1327536747 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7f40f5410 0x7fc7f40f5830 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7fc7e4009fb0 tx=0x7fc7e4009b30).stop 2024-01-21T11:42:43.741 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:43.739+0000 7fc7fba7d700 1 -- 172.21.15.134:0/1327536747 shutdown_connections 2024-01-21T11:42:43.741 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:43.739+0000 7fc7fba7d700 1 --2- 172.21.15.134:0/1327536747 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7f40f5410 0x7fc7f40f5830 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:42:43.741 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:43.739+0000 7fc7fba7d700 1 -- 172.21.15.134:0/1327536747 >> 172.21.15.134:0/1327536747 conn(0x7fc7f40f0b70 msgr2=0x7fc7f40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:42:43.741 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:43.739+0000 7fc7fba7d700 1 -- 172.21.15.134:0/1327536747 shutdown_connections 2024-01-21T11:42:43.741 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:43.739+0000 7fc7fba7d700 1 -- 172.21.15.134:0/1327536747 wait complete. 2024-01-21T11:42:43.742 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:43.740+0000 7fc7fba7d700 1 Processor -- start 2024-01-21T11:42:43.742 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:43.740+0000 7fc7fba7d700 1 -- start start 2024-01-21T11:42:43.742 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:43.740+0000 7fc7fba7d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7f40f5410 0x7fc7f4106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:42:43.742 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:43.740+0000 7fc7fba7d700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7f4106c10 con 0x7fc7f40f5410 2024-01-21T11:42:43.742 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:43.740+0000 7fc7f9819700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7f40f5410 0x7fc7f4106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:42:43.742 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:43.740+0000 7fc7f9819700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7f40f5410 0x7fc7f4106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:45396/0 (socket says 172.21.15.134:45396) 2024-01-21T11:42:43.742 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:43.740+0000 7fc7f9819700 1 -- 172.21.15.134:0/2380703546 learned_addr learned my addr 172.21.15.134:0/2380703546 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:42:43.742 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:43.740+0000 7fc7f9819700 1 -- 172.21.15.134:0/2380703546 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7e4005040 con 0x7fc7f40f5410 2024-01-21T11:42:43.742 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:43.740+0000 7fc7f9819700 1 --2- 172.21.15.134:0/2380703546 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7f40f5410 0x7fc7f4106720 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7fc7e4014000 tx=0x7fc7e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:42:43.742 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:43.741+0000 7fc7ea7fc700 1 -- 172.21.15.134:0/2380703546 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc7e40095f0 con 0x7fc7f40f5410 2024-01-21T11:42:43.742 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:43.741+0000 7fc7ea7fc700 1 -- 172.21.15.134:0/2380703546 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc7e4007e70 con 0x7fc7f40f5410 2024-01-21T11:42:43.742 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:43.741+0000 7fc7fba7d700 1 -- 172.21.15.134:0/2380703546 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7f4106e10 con 0x7fc7f40f5410 2024-01-21T11:42:43.742 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:43.741+0000 7fc7ea7fc700 1 -- 172.21.15.134:0/2380703546 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc7e4014ea0 con 0x7fc7f40f5410 2024-01-21T11:42:43.742 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:43.741+0000 7fc7fba7d700 1 -- 172.21.15.134:0/2380703546 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7f41072a0 con 0x7fc7f40f5410 2024-01-21T11:42:43.743 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:43.741+0000 7fc7ea7fc700 1 -- 172.21.15.134:0/2380703546 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc7e40079c0 con 0x7fc7f40f5410 2024-01-21T11:42:43.743 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:43.742+0000 7fc7fba7d700 1 -- 172.21.15.134:0/2380703546 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7f4045ad0 con 0x7fc7f40f5410 2024-01-21T11:42:43.743 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:43.742+0000 7fc7ea7fc700 1 --2- 172.21.15.134:0/2380703546 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fc7e002ff60 0x7fc7e0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:42:43.743 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:43.742+0000 7fc7ea7fc700 1 -- 172.21.15.134:0/2380703546 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc7e4040860 con 0x7fc7f40f5410 2024-01-21T11:42:43.743 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:43.742+0000 7fc7f9018700 1 --2- 172.21.15.134:0/2380703546 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fc7e002ff60 0x7fc7e0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:42:43.743 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:43.742+0000 7fc7f9018700 1 --2- 172.21.15.134:0/2380703546 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fc7e002ff60 0x7fc7e0032420 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fc7f00096a0 tx=0x7fc7f0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:42:43.746 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:43.746+0000 7fc7ea7fc700 1 -- 172.21.15.134:0/2380703546 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc7e400f950 con 0x7fc7f40f5410 2024-01-21T11:42:43.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:42:43 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:42:41.806440+0000 mgr.smithi029.blhxxk (mgr.14164) 21 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:42:43.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:43.950+0000 7fc7fba7d700 1 -- 172.21.15.134:0/2380703546 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc7f4108150 con 0x7fc7f40f5410 2024-01-21T11:42:43.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:43.951+0000 7fc7ea7fc700 1 -- 172.21.15.134:0/2380703546 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc7e4010230 con 0x7fc7f40f5410 2024-01-21T11:42:43.951 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:42:43.951 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:42:43.953 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:43.952+0000 7fc7fba7d700 1 -- 172.21.15.134:0/2380703546 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fc7e002ff60 msgr2=0x7fc7e0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:42:43.953 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:43.952+0000 7fc7fba7d700 1 --2- 172.21.15.134:0/2380703546 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fc7e002ff60 0x7fc7e0032420 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fc7f00096a0 tx=0x7fc7f0006b40).stop 2024-01-21T11:42:43.953 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:43.952+0000 7fc7fba7d700 1 -- 172.21.15.134:0/2380703546 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7f40f5410 msgr2=0x7fc7f4106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:42:43.953 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:43.952+0000 7fc7fba7d700 1 --2- 172.21.15.134:0/2380703546 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7f40f5410 0x7fc7f4106720 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7fc7e4014000 tx=0x7fc7e400b040).stop 2024-01-21T11:42:43.953 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:43.952+0000 7fc7fba7d700 1 -- 172.21.15.134:0/2380703546 shutdown_connections 2024-01-21T11:42:43.953 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:43.952+0000 7fc7fba7d700 1 --2- 172.21.15.134:0/2380703546 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fc7e002ff60 0x7fc7e0032420 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:42:43.953 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:43.952+0000 7fc7fba7d700 1 --2- 172.21.15.134:0/2380703546 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7f40f5410 0x7fc7f4106720 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:42:43.954 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:43.952+0000 7fc7fba7d700 1 -- 172.21.15.134:0/2380703546 >> 172.21.15.134:0/2380703546 conn(0x7fc7f40f0b70 msgr2=0x7fc7f405fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:42:43.954 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:43.953+0000 7fc7fba7d700 1 -- 172.21.15.134:0/2380703546 shutdown_connections 2024-01-21T11:42:43.954 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:43.953+0000 7fc7fba7d700 1 -- 172.21.15.134:0/2380703546 wait complete. 2024-01-21T11:42:43.954 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:42:44.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:42:44 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:42:43.952138+0000 mon.smithi029 (mon.0) 124 : audit [DBG] from='client.? 172.21.15.134:0/2380703546' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:42:45.334 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:42:45.335 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:42:45.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:42:45 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:42:43.806748+0000 mgr.smithi029.blhxxk (mgr.14164) 22 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:42:46.570 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:46.568+0000 7efff5418700 1 -- 172.21.15.134:0/799228655 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efff00f62a0 msgr2=0x7efff00f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:42:46.570 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:46.568+0000 7efff5418700 1 --2- 172.21.15.134:0/799228655 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efff00f62a0 0x7efff00f66c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7effe0009fb0 tx=0x7effe0009b30).stop 2024-01-21T11:42:46.570 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:46.568+0000 7efff5418700 1 -- 172.21.15.134:0/799228655 shutdown_connections 2024-01-21T11:42:46.570 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:46.568+0000 7efff5418700 1 --2- 172.21.15.134:0/799228655 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efff00f62a0 0x7efff00f66c0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:42:46.570 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:46.568+0000 7efff5418700 1 -- 172.21.15.134:0/799228655 >> 172.21.15.134:0/799228655 conn(0x7efff00f1aa0 msgr2=0x7efff00f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:42:46.570 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:46.568+0000 7efff5418700 1 -- 172.21.15.134:0/799228655 shutdown_connections 2024-01-21T11:42:46.571 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:46.568+0000 7efff5418700 1 -- 172.21.15.134:0/799228655 wait complete. 2024-01-21T11:42:46.571 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:46.568+0000 7efff5418700 1 Processor -- start 2024-01-21T11:42:46.571 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:46.569+0000 7efff5418700 1 -- start start 2024-01-21T11:42:46.571 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:46.569+0000 7efff5418700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efff00f62a0 0x7efff01064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:42:46.571 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:46.569+0000 7efff5418700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efff01069a0 con 0x7efff00f62a0 2024-01-21T11:42:46.571 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:46.569+0000 7effeffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efff00f62a0 0x7efff01064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:42:46.571 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:46.569+0000 7effeffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efff00f62a0 0x7efff01064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:40696/0 (socket says 172.21.15.134:40696) 2024-01-21T11:42:46.571 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:46.569+0000 7effeffff700 1 -- 172.21.15.134:0/1697163946 learned_addr learned my addr 172.21.15.134:0/1697163946 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:42:46.571 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:46.569+0000 7effeffff700 1 -- 172.21.15.134:0/1697163946 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effe0005040 con 0x7efff00f62a0 2024-01-21T11:42:46.571 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:46.569+0000 7effeffff700 1 --2- 172.21.15.134:0/1697163946 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efff00f62a0 0x7efff01064b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7effe0013510 tx=0x7effe000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:42:46.571 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:46.570+0000 7effecff9700 1 -- 172.21.15.134:0/1697163946 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7effe0009320 con 0x7efff00f62a0 2024-01-21T11:42:46.571 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:46.570+0000 7efff5418700 1 -- 172.21.15.134:0/1697163946 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efff0106ba0 con 0x7efff00f62a0 2024-01-21T11:42:46.571 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:46.570+0000 7effecff9700 1 -- 172.21.15.134:0/1697163946 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7effe0013f00 con 0x7efff00f62a0 2024-01-21T11:42:46.572 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:46.570+0000 7effecff9700 1 -- 172.21.15.134:0/1697163946 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7effe0014dc0 con 0x7efff00f62a0 2024-01-21T11:42:46.572 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:46.570+0000 7efff5418700 1 -- 172.21.15.134:0/1697163946 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efff0107030 con 0x7efff00f62a0 2024-01-21T11:42:46.572 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:46.570+0000 7effecff9700 1 -- 172.21.15.134:0/1697163946 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7effe0014420 con 0x7efff00f62a0 2024-01-21T11:42:46.573 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:46.570+0000 7effecff9700 1 --2- 172.21.15.134:0/1697163946 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7effdc02ff60 0x7effdc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:42:46.573 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:46.570+0000 7efff5418700 1 -- 172.21.15.134:0/1697163946 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efff0045ad0 con 0x7efff00f62a0 2024-01-21T11:42:46.573 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:46.571+0000 7effecff9700 1 -- 172.21.15.134:0/1697163946 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7effe0041130 con 0x7efff00f62a0 2024-01-21T11:42:46.573 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:46.571+0000 7effef7fe700 1 --2- 172.21.15.134:0/1697163946 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7effdc02ff60 0x7effdc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:42:46.573 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:46.571+0000 7effef7fe700 1 --2- 172.21.15.134:0/1697163946 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7effdc02ff60 0x7effdc032420 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7effd80096a0 tx=0x7effd8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:42:46.575 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:46.575+0000 7effecff9700 1 -- 172.21.15.134:0/1697163946 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7effe001cb10 con 0x7efff00f62a0 2024-01-21T11:42:46.779 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:46.778+0000 7efff5418700 1 -- 172.21.15.134:0/1697163946 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efff0107ee0 con 0x7efff00f62a0 2024-01-21T11:42:46.779 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:46.779+0000 7effecff9700 1 -- 172.21.15.134:0/1697163946 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7effe001cb10 con 0x7efff00f62a0 2024-01-21T11:42:46.780 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:42:46.780 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:42:46.781 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:46.781+0000 7efff5418700 1 -- 172.21.15.134:0/1697163946 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7effdc02ff60 msgr2=0x7effdc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:42:46.782 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:46.781+0000 7efff5418700 1 --2- 172.21.15.134:0/1697163946 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7effdc02ff60 0x7effdc032420 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7effd80096a0 tx=0x7effd8006b40).stop 2024-01-21T11:42:46.782 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:46.781+0000 7efff5418700 1 -- 172.21.15.134:0/1697163946 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efff00f62a0 msgr2=0x7efff01064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:42:46.782 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:46.781+0000 7efff5418700 1 --2- 172.21.15.134:0/1697163946 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efff00f62a0 0x7efff01064b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7effe0013510 tx=0x7effe000b040).stop 2024-01-21T11:42:46.782 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:46.781+0000 7efff5418700 1 -- 172.21.15.134:0/1697163946 shutdown_connections 2024-01-21T11:42:46.782 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:46.781+0000 7efff5418700 1 --2- 172.21.15.134:0/1697163946 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7effdc02ff60 0x7effdc032420 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:42:46.782 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:46.781+0000 7efff5418700 1 --2- 172.21.15.134:0/1697163946 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efff00f62a0 0x7efff01064b0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:42:46.782 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:46.781+0000 7efff5418700 1 -- 172.21.15.134:0/1697163946 >> 172.21.15.134:0/1697163946 conn(0x7efff00f1aa0 msgr2=0x7efff00f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:42:46.782 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:46.781+0000 7efff5418700 1 -- 172.21.15.134:0/1697163946 shutdown_connections 2024-01-21T11:42:46.782 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:46.781+0000 7efff5418700 1 -- 172.21.15.134:0/1697163946 wait complete. 2024-01-21T11:42:46.782 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:42:47.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:42:47 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:42:45.807105+0000 mgr.smithi029.blhxxk (mgr.14164) 23 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:42:47.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:42:47 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:42:46.780484+0000 mon.smithi029 (mon.0) 125 : audit [DBG] from='client.? 172.21.15.134:0/1697163946' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:42:48.172 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:42:48.172 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:42:49.366 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:49.365+0000 7f16368c2700 1 -- 172.21.15.134:0/276757090 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1630060220 msgr2=0x7f16300f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:42:49.367 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:49.365+0000 7f16368c2700 1 --2- 172.21.15.134:0/276757090 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1630060220 0x7f16300f9150 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f1624009fb0 tx=0x7f1624009b30).stop 2024-01-21T11:42:49.367 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:49.365+0000 7f16368c2700 1 -- 172.21.15.134:0/276757090 shutdown_connections 2024-01-21T11:42:49.367 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:49.365+0000 7f16368c2700 1 --2- 172.21.15.134:0/276757090 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1630060220 0x7f16300f9150 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:42:49.367 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:49.365+0000 7f16368c2700 1 -- 172.21.15.134:0/276757090 >> 172.21.15.134:0/276757090 conn(0x7f16300f0b70 msgr2=0x7f16300f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:42:49.367 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:49.365+0000 7f16368c2700 1 -- 172.21.15.134:0/276757090 shutdown_connections 2024-01-21T11:42:49.367 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:49.365+0000 7f16368c2700 1 -- 172.21.15.134:0/276757090 wait complete. 2024-01-21T11:42:49.367 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:49.366+0000 7f16368c2700 1 Processor -- start 2024-01-21T11:42:49.367 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:49.366+0000 7f16368c2700 1 -- start start 2024-01-21T11:42:49.367 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:49.366+0000 7f16368c2700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1630060220 0x7f16301066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:42:49.367 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:49.366+0000 7f16368c2700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1630106bd0 con 0x7f1630060220 2024-01-21T11:42:49.367 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:49.366+0000 7f162ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1630060220 0x7f16301066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:42:49.368 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:49.366+0000 7f162ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1630060220 0x7f16301066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:40710/0 (socket says 172.21.15.134:40710) 2024-01-21T11:42:49.368 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:49.366+0000 7f162ffff700 1 -- 172.21.15.134:0/1868251834 learned_addr learned my addr 172.21.15.134:0/1868251834 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:42:49.368 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:49.367+0000 7f162ffff700 1 -- 172.21.15.134:0/1868251834 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1624005040 con 0x7f1630060220 2024-01-21T11:42:49.369 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:49.367+0000 7f162ffff700 1 --2- 172.21.15.134:0/1868251834 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1630060220 0x7f16301066e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f1624013510 tx=0x7f162400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:42:49.369 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:49.367+0000 7f162cff9700 1 -- 172.21.15.134:0/1868251834 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1624009320 con 0x7f1630060220 2024-01-21T11:42:49.369 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:49.367+0000 7f162cff9700 1 -- 172.21.15.134:0/1868251834 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1624013f00 con 0x7f1630060220 2024-01-21T11:42:49.369 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:49.367+0000 7f16368c2700 1 -- 172.21.15.134:0/1868251834 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1630106dd0 con 0x7f1630060220 2024-01-21T11:42:49.369 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:49.367+0000 7f162cff9700 1 -- 172.21.15.134:0/1868251834 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1624014dc0 con 0x7f1630060220 2024-01-21T11:42:49.369 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:49.367+0000 7f16368c2700 1 -- 172.21.15.134:0/1868251834 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1630107260 con 0x7f1630060220 2024-01-21T11:42:49.369 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:49.368+0000 7f162cff9700 1 -- 172.21.15.134:0/1868251834 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1624014420 con 0x7f1630060220 2024-01-21T11:42:49.370 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:49.368+0000 7f16368c2700 1 -- 172.21.15.134:0/1868251834 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1630045ad0 con 0x7f1630060220 2024-01-21T11:42:49.370 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:49.368+0000 7f162cff9700 1 --2- 172.21.15.134:0/1868251834 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f161802ff60 0x7f1618032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:42:49.370 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:49.368+0000 7f162cff9700 1 -- 172.21.15.134:0/1868251834 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1624041110 con 0x7f1630060220 2024-01-21T11:42:49.370 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:49.368+0000 7f162f7fe700 1 --2- 172.21.15.134:0/1868251834 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f161802ff60 0x7f1618032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:42:49.370 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:49.369+0000 7f162f7fe700 1 --2- 172.21.15.134:0/1868251834 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f161802ff60 0x7f1618032420 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f16200096a0 tx=0x7f1620006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:42:49.375 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:49.373+0000 7f162cff9700 1 -- 172.21.15.134:0/1868251834 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f16240095f0 con 0x7f1630060220 2024-01-21T11:42:49.579 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:49.578+0000 7f16368c2700 1 -- 172.21.15.134:0/1868251834 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1630108110 con 0x7f1630060220 2024-01-21T11:42:49.580 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:49.578+0000 7f162cff9700 1 -- 172.21.15.134:0/1868251834 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f162401cb10 con 0x7f1630060220 2024-01-21T11:42:49.580 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:42:49.580 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:42:49.581 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:49.580+0000 7f16368c2700 1 -- 172.21.15.134:0/1868251834 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f161802ff60 msgr2=0x7f1618032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:42:49.581 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:49.580+0000 7f16368c2700 1 --2- 172.21.15.134:0/1868251834 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f161802ff60 0x7f1618032420 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f16200096a0 tx=0x7f1620006b40).stop 2024-01-21T11:42:49.581 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:49.580+0000 7f16368c2700 1 -- 172.21.15.134:0/1868251834 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1630060220 msgr2=0x7f16301066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:42:49.581 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:49.580+0000 7f16368c2700 1 --2- 172.21.15.134:0/1868251834 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1630060220 0x7f16301066e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f1624013510 tx=0x7f162400b040).stop 2024-01-21T11:42:49.581 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:49.580+0000 7f16368c2700 1 -- 172.21.15.134:0/1868251834 shutdown_connections 2024-01-21T11:42:49.581 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:49.580+0000 7f16368c2700 1 --2- 172.21.15.134:0/1868251834 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f161802ff60 0x7f1618032420 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:42:49.581 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:49.580+0000 7f16368c2700 1 --2- 172.21.15.134:0/1868251834 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1630060220 0x7f16301066e0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:42:49.581 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:49.580+0000 7f16368c2700 1 -- 172.21.15.134:0/1868251834 >> 172.21.15.134:0/1868251834 conn(0x7f16300f0b70 msgr2=0x7f16300f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:42:49.582 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:49.580+0000 7f16368c2700 1 -- 172.21.15.134:0/1868251834 shutdown_connections 2024-01-21T11:42:49.582 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:49.580+0000 7f16368c2700 1 -- 172.21.15.134:0/1868251834 wait complete. 2024-01-21T11:42:49.582 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:42:49.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:42:49 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:42:47.807523+0000 mgr.smithi029.blhxxk (mgr.14164) 24 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:42:50.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:42:50 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:42:49.579414+0000 mon.smithi029 (mon.0) 126 : audit [DBG] from='client.? 172.21.15.134:0/1868251834' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:42:50.974 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:42:50.974 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:42:51.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:42:51 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:42:49.807812+0000 mgr.smithi029.blhxxk (mgr.14164) 25 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:42:52.147 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:52.145+0000 7f675359e700 1 -- 172.21.15.134:0/3507892627 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67540f51a0 msgr2=0x7f67540f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:42:52.147 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:52.145+0000 7f675359e700 1 --2- 172.21.15.134:0/3507892627 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67540f51a0 0x7f67540f55c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f6748009fb0 tx=0x7f6748009b30).stop 2024-01-21T11:42:52.147 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:52.146+0000 7f675359e700 1 -- 172.21.15.134:0/3507892627 shutdown_connections 2024-01-21T11:42:52.147 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:52.146+0000 7f675359e700 1 --2- 172.21.15.134:0/3507892627 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67540f51a0 0x7f67540f55c0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:42:52.148 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:52.146+0000 7f675359e700 1 -- 172.21.15.134:0/3507892627 >> 172.21.15.134:0/3507892627 conn(0x7f67540f09a0 msgr2=0x7f67540f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:42:52.148 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:52.146+0000 7f675359e700 1 -- 172.21.15.134:0/3507892627 shutdown_connections 2024-01-21T11:42:52.148 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:52.146+0000 7f675359e700 1 -- 172.21.15.134:0/3507892627 wait complete. 2024-01-21T11:42:52.148 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:52.146+0000 7f675359e700 1 Processor -- start 2024-01-21T11:42:52.148 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:52.146+0000 7f675359e700 1 -- start start 2024-01-21T11:42:52.148 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:52.146+0000 7f675359e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67540f51a0 0x7f67541053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:42:52.148 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:52.146+0000 7f675359e700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67541058a0 con 0x7f67540f51a0 2024-01-21T11:42:52.148 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:52.146+0000 7f675259c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67540f51a0 0x7f67541053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:42:52.148 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:52.147+0000 7f675259c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67540f51a0 0x7f67541053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:40728/0 (socket says 172.21.15.134:40728) 2024-01-21T11:42:52.148 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:52.147+0000 7f675259c700 1 -- 172.21.15.134:0/249719249 learned_addr learned my addr 172.21.15.134:0/249719249 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:42:52.148 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:52.147+0000 7f675259c700 1 -- 172.21.15.134:0/249719249 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6748005040 con 0x7f67540f51a0 2024-01-21T11:42:52.148 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:52.147+0000 7f675259c700 1 --2- 172.21.15.134:0/249719249 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67540f51a0 0x7f67541053b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f6748013510 tx=0x7f674800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:42:52.148 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:52.147+0000 7f6742ffd700 1 -- 172.21.15.134:0/249719249 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6748009320 con 0x7f67540f51a0 2024-01-21T11:42:52.149 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:52.147+0000 7f675359e700 1 -- 172.21.15.134:0/249719249 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6754105aa0 con 0x7f67540f51a0 2024-01-21T11:42:52.149 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:52.147+0000 7f6742ffd700 1 -- 172.21.15.134:0/249719249 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6748013f00 con 0x7f67540f51a0 2024-01-21T11:42:52.149 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:52.147+0000 7f6742ffd700 1 -- 172.21.15.134:0/249719249 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6748014dc0 con 0x7f67540f51a0 2024-01-21T11:42:52.149 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:52.147+0000 7f675359e700 1 -- 172.21.15.134:0/249719249 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6754105f30 con 0x7f67540f51a0 2024-01-21T11:42:52.150 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:52.148+0000 7f6742ffd700 1 -- 172.21.15.134:0/249719249 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f67480095f0 con 0x7f67540f51a0 2024-01-21T11:42:52.150 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:52.148+0000 7f675359e700 1 -- 172.21.15.134:0/249719249 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6754045ad0 con 0x7f67540f51a0 2024-01-21T11:42:52.150 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:52.148+0000 7f6742ffd700 1 --2- 172.21.15.134:0/249719249 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f673c02ff60 0x7f673c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:42:52.150 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:52.148+0000 7f6742ffd700 1 -- 172.21.15.134:0/249719249 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f67480410c0 con 0x7f67540f51a0 2024-01-21T11:42:52.150 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:52.148+0000 7f6751d9b700 1 --2- 172.21.15.134:0/249719249 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f673c02ff60 0x7f673c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:42:52.150 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:52.149+0000 7f6751d9b700 1 --2- 172.21.15.134:0/249719249 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f673c02ff60 0x7f673c032420 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f67440096a0 tx=0x7f6744006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:42:52.153 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:52.153+0000 7f6742ffd700 1 -- 172.21.15.134:0/249719249 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6748014420 con 0x7f67540f51a0 2024-01-21T11:42:52.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:52.357+0000 7f675359e700 1 -- 172.21.15.134:0/249719249 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6754106de0 con 0x7f67540f51a0 2024-01-21T11:42:52.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:52.357+0000 7f6742ffd700 1 -- 172.21.15.134:0/249719249 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6748010230 con 0x7f67540f51a0 2024-01-21T11:42:52.357 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:42:52.358 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:42:52.359 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:52.359+0000 7f675359e700 1 -- 172.21.15.134:0/249719249 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f673c02ff60 msgr2=0x7f673c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:42:52.359 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:52.359+0000 7f675359e700 1 --2- 172.21.15.134:0/249719249 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f673c02ff60 0x7f673c032420 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f67440096a0 tx=0x7f6744006b40).stop 2024-01-21T11:42:52.359 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:52.359+0000 7f675359e700 1 -- 172.21.15.134:0/249719249 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67540f51a0 msgr2=0x7f67541053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:42:52.359 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:52.359+0000 7f675359e700 1 --2- 172.21.15.134:0/249719249 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67540f51a0 0x7f67541053b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f6748013510 tx=0x7f674800b040).stop 2024-01-21T11:42:52.359 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:52.359+0000 7f675359e700 1 -- 172.21.15.134:0/249719249 shutdown_connections 2024-01-21T11:42:52.359 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:52.359+0000 7f675359e700 1 --2- 172.21.15.134:0/249719249 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f673c02ff60 0x7f673c032420 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:42:52.359 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:52.359+0000 7f675359e700 1 --2- 172.21.15.134:0/249719249 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67540f51a0 0x7f67541053b0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:42:52.359 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:52.359+0000 7f675359e700 1 -- 172.21.15.134:0/249719249 >> 172.21.15.134:0/249719249 conn(0x7f67540f09a0 msgr2=0x7f67540f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:42:52.359 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:52.359+0000 7f675359e700 1 -- 172.21.15.134:0/249719249 shutdown_connections 2024-01-21T11:42:52.360 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:52.359+0000 7f675359e700 1 -- 172.21.15.134:0/249719249 wait complete. 2024-01-21T11:42:52.360 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:42:52.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:42:52 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:42:52.358518+0000 mon.smithi029 (mon.0) 127 : audit [DBG] from='client.? 172.21.15.134:0/249719249' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:42:53.739 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:42:53.739 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:42:53.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:42:53 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:42:51.808049+0000 mgr.smithi029.blhxxk (mgr.14164) 26 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:42:54.855 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:54.854+0000 7fbad907c700 1 -- 172.21.15.134:0/139035000 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbad40f5410 msgr2=0x7fbad40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:42:54.855 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:54.854+0000 7fbad907c700 1 --2- 172.21.15.134:0/139035000 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbad40f5410 0x7fbad40f5830 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7fbac8009fb0 tx=0x7fbac8009b30).stop 2024-01-21T11:42:54.855 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:54.854+0000 7fbad907c700 1 -- 172.21.15.134:0/139035000 shutdown_connections 2024-01-21T11:42:54.855 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:54.854+0000 7fbad907c700 1 --2- 172.21.15.134:0/139035000 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbad40f5410 0x7fbad40f5830 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:42:54.855 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:54.854+0000 7fbad907c700 1 -- 172.21.15.134:0/139035000 >> 172.21.15.134:0/139035000 conn(0x7fbad40f0b70 msgr2=0x7fbad40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:42:54.855 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:54.854+0000 7fbad907c700 1 -- 172.21.15.134:0/139035000 shutdown_connections 2024-01-21T11:42:54.855 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:54.854+0000 7fbad907c700 1 -- 172.21.15.134:0/139035000 wait complete. 2024-01-21T11:42:54.857 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:54.855+0000 7fbad907c700 1 Processor -- start 2024-01-21T11:42:54.857 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:54.855+0000 7fbad907c700 1 -- start start 2024-01-21T11:42:54.857 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:54.855+0000 7fbad907c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbad40f5410 0x7fbad4109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:42:54.857 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:54.855+0000 7fbad907c700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbad4060220 con 0x7fbad40f5410 2024-01-21T11:42:54.857 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:54.855+0000 7fbad2d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbad40f5410 0x7fbad4109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:42:54.858 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:54.855+0000 7fbad2d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbad40f5410 0x7fbad4109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:60664/0 (socket says 172.21.15.134:60664) 2024-01-21T11:42:54.858 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:54.855+0000 7fbad2d9d700 1 -- 172.21.15.134:0/2610870086 learned_addr learned my addr 172.21.15.134:0/2610870086 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:42:54.858 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:54.856+0000 7fbad2d9d700 1 -- 172.21.15.134:0/2610870086 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbac8005040 con 0x7fbad40f5410 2024-01-21T11:42:54.858 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:54.856+0000 7fbad2d9d700 1 --2- 172.21.15.134:0/2610870086 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbad40f5410 0x7fbad4109450 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fbac8013510 tx=0x7fbac800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:42:54.858 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:54.856+0000 7fbabb7fe700 1 -- 172.21.15.134:0/2610870086 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbac8009320 con 0x7fbad40f5410 2024-01-21T11:42:54.858 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:54.856+0000 7fbad907c700 1 -- 172.21.15.134:0/2610870086 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbad4060420 con 0x7fbad40f5410 2024-01-21T11:42:54.858 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:54.856+0000 7fbabb7fe700 1 -- 172.21.15.134:0/2610870086 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbac8013f00 con 0x7fbad40f5410 2024-01-21T11:42:54.858 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:54.856+0000 7fbabb7fe700 1 -- 172.21.15.134:0/2610870086 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbac8014d90 con 0x7fbad40f5410 2024-01-21T11:42:54.858 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:54.856+0000 7fbad907c700 1 -- 172.21.15.134:0/2610870086 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbad4109b00 con 0x7fbad40f5410 2024-01-21T11:42:54.858 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:54.857+0000 7fbabb7fe700 1 -- 172.21.15.134:0/2610870086 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbac80263a0 con 0x7fbad40f5410 2024-01-21T11:42:54.858 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:54.857+0000 7fbad907c700 1 -- 172.21.15.134:0/2610870086 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbad4059100 con 0x7fbad40f5410 2024-01-21T11:42:54.858 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:54.857+0000 7fbabb7fe700 1 --2- 172.21.15.134:0/2610870086 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fbabc02ff60 0x7fbabc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:42:54.858 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:54.857+0000 7fbabb7fe700 1 -- 172.21.15.134:0/2610870086 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbac8041080 con 0x7fbad40f5410 2024-01-21T11:42:54.858 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:54.857+0000 7fbad259c700 1 --2- 172.21.15.134:0/2610870086 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fbabc02ff60 0x7fbabc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:42:54.859 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:54.858+0000 7fbad259c700 1 --2- 172.21.15.134:0/2610870086 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fbabc02ff60 0x7fbabc032420 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7fbac40096a0 tx=0x7fbac4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:42:54.861 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:54.862+0000 7fbabb7fe700 1 -- 172.21.15.134:0/2610870086 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbac80095f0 con 0x7fbad40f5410 2024-01-21T11:42:55.068 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:55.067+0000 7fbad907c700 1 -- 172.21.15.134:0/2610870086 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbad4045ad0 con 0x7fbad40f5410 2024-01-21T11:42:55.068 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:55.068+0000 7fbabb7fe700 1 -- 172.21.15.134:0/2610870086 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbac801db70 con 0x7fbad40f5410 2024-01-21T11:42:55.068 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:42:55.068 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:42:55.070 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:55.069+0000 7fbad907c700 1 -- 172.21.15.134:0/2610870086 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fbabc02ff60 msgr2=0x7fbabc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:42:55.070 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:55.069+0000 7fbad907c700 1 --2- 172.21.15.134:0/2610870086 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fbabc02ff60 0x7fbabc032420 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7fbac40096a0 tx=0x7fbac4006b40).stop 2024-01-21T11:42:55.070 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:55.069+0000 7fbad907c700 1 -- 172.21.15.134:0/2610870086 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbad40f5410 msgr2=0x7fbad4109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:42:55.070 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:55.069+0000 7fbad907c700 1 --2- 172.21.15.134:0/2610870086 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbad40f5410 0x7fbad4109450 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fbac8013510 tx=0x7fbac800b040).stop 2024-01-21T11:42:55.070 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:55.069+0000 7fbad907c700 1 -- 172.21.15.134:0/2610870086 shutdown_connections 2024-01-21T11:42:55.070 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:55.069+0000 7fbad907c700 1 --2- 172.21.15.134:0/2610870086 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fbabc02ff60 0x7fbabc032420 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:42:55.071 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:55.069+0000 7fbad907c700 1 --2- 172.21.15.134:0/2610870086 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbad40f5410 0x7fbad4109450 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:42:55.071 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:55.069+0000 7fbad907c700 1 -- 172.21.15.134:0/2610870086 >> 172.21.15.134:0/2610870086 conn(0x7fbad40f0b70 msgr2=0x7fbad40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:42:55.071 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:55.069+0000 7fbad907c700 1 -- 172.21.15.134:0/2610870086 shutdown_connections 2024-01-21T11:42:55.071 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:42:55.069+0000 7fbad907c700 1 -- 172.21.15.134:0/2610870086 wait complete. 2024-01-21T11:42:55.071 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:42:55.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:42:55 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:42:53.808301+0000 mgr.smithi029.blhxxk (mgr.14164) 27 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:42:55.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:42:55 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:42:55.068921+0000 mon.smithi029 (mon.0) 128 : audit [DBG] from='client.? 172.21.15.134:0/2610870086' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:42:56.479 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:42:56.480 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:42:57.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:42:57 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:42:55.808622+0000 mgr.smithi029.blhxxk (mgr.14164) 28 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:42:59.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:42:59 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:42:57.808890+0000 mgr.smithi029.blhxxk (mgr.14164) 29 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:43:01.191 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:01.115+0000 7f45d24ac700 1 -- 172.21.15.134:0/1106523373 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45cc0f5810 msgr2=0x7f45cc0f5c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:43:01.191 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:01.115+0000 7f45d24ac700 1 --2- 172.21.15.134:0/1106523373 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45cc0f5810 0x7f45cc0f5c30 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f45c8009fb0 tx=0x7f45c8009b30).stop 2024-01-21T11:43:01.191 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:01.115+0000 7f45d24ac700 1 -- 172.21.15.134:0/1106523373 shutdown_connections 2024-01-21T11:43:01.191 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:01.115+0000 7f45d24ac700 1 --2- 172.21.15.134:0/1106523373 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45cc0f5810 0x7f45cc0f5c30 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:43:01.191 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:01.115+0000 7f45d24ac700 1 -- 172.21.15.134:0/1106523373 >> 172.21.15.134:0/1106523373 conn(0x7f45cc0f0120 msgr2=0x7f45cc0f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:43:01.191 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:01.115+0000 7f45d24ac700 1 -- 172.21.15.134:0/1106523373 shutdown_connections 2024-01-21T11:43:01.191 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:01.115+0000 7f45d24ac700 1 -- 172.21.15.134:0/1106523373 wait complete. 2024-01-21T11:43:01.191 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:01.116+0000 7f45d24ac700 1 Processor -- start 2024-01-21T11:43:01.192 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:01.116+0000 7f45d24ac700 1 -- start start 2024-01-21T11:43:01.192 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:01.116+0000 7f45d24ac700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45cc0f5810 0x7f45cc1053e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:43:01.192 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:01.116+0000 7f45d24ac700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45cc1058d0 con 0x7f45cc0f5810 2024-01-21T11:43:01.192 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:01.116+0000 7f45d14aa700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45cc0f5810 0x7f45cc1053e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:43:01.192 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:01.116+0000 7f45d14aa700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45cc0f5810 0x7f45cc1053e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:60670/0 (socket says 172.21.15.134:60670) 2024-01-21T11:43:01.192 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:01.116+0000 7f45d14aa700 1 -- 172.21.15.134:0/511891377 learned_addr learned my addr 172.21.15.134:0/511891377 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:43:01.192 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:01.117+0000 7f45d14aa700 1 -- 172.21.15.134:0/511891377 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45c8005040 con 0x7f45cc0f5810 2024-01-21T11:43:01.192 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:01.117+0000 7f45d14aa700 1 --2- 172.21.15.134:0/511891377 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45cc0f5810 0x7f45cc1053e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f45c8004dd0 tx=0x7f45c8009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:43:01.192 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:01.117+0000 7f45bdffb700 1 -- 172.21.15.134:0/511891377 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f45c8018070 con 0x7f45cc0f5810 2024-01-21T11:43:01.192 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:01.117+0000 7f45d24ac700 1 -- 172.21.15.134:0/511891377 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45cc105ad0 con 0x7f45cc0f5810 2024-01-21T11:43:01.192 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:01.117+0000 7f45bdffb700 1 -- 172.21.15.134:0/511891377 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f45c8013410 con 0x7f45cc0f5810 2024-01-21T11:43:01.192 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:01.117+0000 7f45bdffb700 1 -- 172.21.15.134:0/511891377 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f45c801c3c0 con 0x7f45cc0f5810 2024-01-21T11:43:01.192 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:01.117+0000 7f45d24ac700 1 -- 172.21.15.134:0/511891377 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45cc105f60 con 0x7f45cc0f5810 2024-01-21T11:43:01.192 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:01.118+0000 7f45bdffb700 1 -- 172.21.15.134:0/511891377 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f45c801d420 con 0x7f45cc0f5810 2024-01-21T11:43:01.192 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:01.118+0000 7f45d24ac700 1 -- 172.21.15.134:0/511891377 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45cc045ad0 con 0x7f45cc0f5810 2024-01-21T11:43:01.193 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:01.118+0000 7f45bdffb700 1 --2- 172.21.15.134:0/511891377 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f45b802ff60 0x7f45b8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:43:01.193 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:01.118+0000 7f45bdffb700 1 -- 172.21.15.134:0/511891377 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f45c80413d0 con 0x7f45cc0f5810 2024-01-21T11:43:01.193 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:01.119+0000 7f45d0ca9700 1 --2- 172.21.15.134:0/511891377 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f45b802ff60 0x7f45b8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:43:01.193 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:01.119+0000 7f45d0ca9700 1 --2- 172.21.15.134:0/511891377 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f45b802ff60 0x7f45b8032420 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f45c00096a0 tx=0x7f45c0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:43:01.193 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:01.123+0000 7f45bdffb700 1 -- 172.21.15.134:0/511891377 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f45c801d820 con 0x7f45cc0f5810 2024-01-21T11:43:01.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:01.328+0000 7f45d24ac700 1 -- 172.21.15.134:0/511891377 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f45cc106db0 con 0x7f45cc0f5810 2024-01-21T11:43:01.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:01.328+0000 7f45bdffb700 1 -- 172.21.15.134:0/511891377 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f45c801b370 con 0x7f45cc0f5810 2024-01-21T11:43:01.329 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:43:01.329 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:43:01.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:01.330+0000 7f45d24ac700 1 -- 172.21.15.134:0/511891377 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f45b802ff60 msgr2=0x7f45b8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:43:01.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:01.330+0000 7f45d24ac700 1 --2- 172.21.15.134:0/511891377 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f45b802ff60 0x7f45b8032420 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f45c00096a0 tx=0x7f45c0006b40).stop 2024-01-21T11:43:01.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:01.330+0000 7f45d24ac700 1 -- 172.21.15.134:0/511891377 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45cc0f5810 msgr2=0x7f45cc1053e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:43:01.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:01.330+0000 7f45d24ac700 1 --2- 172.21.15.134:0/511891377 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45cc0f5810 0x7f45cc1053e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f45c8004dd0 tx=0x7f45c8009b30).stop 2024-01-21T11:43:01.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:01.330+0000 7f45d24ac700 1 -- 172.21.15.134:0/511891377 shutdown_connections 2024-01-21T11:43:01.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:01.330+0000 7f45d24ac700 1 --2- 172.21.15.134:0/511891377 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f45b802ff60 0x7f45b8032420 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:43:01.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:01.330+0000 7f45d24ac700 1 --2- 172.21.15.134:0/511891377 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45cc0f5810 0x7f45cc1053e0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:43:01.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:01.330+0000 7f45d24ac700 1 -- 172.21.15.134:0/511891377 >> 172.21.15.134:0/511891377 conn(0x7f45cc0f0120 msgr2=0x7f45cc0f0b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:43:01.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:01.330+0000 7f45d24ac700 1 -- 172.21.15.134:0/511891377 shutdown_connections 2024-01-21T11:43:01.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:01.330+0000 7f45d24ac700 1 -- 172.21.15.134:0/511891377 wait complete. 2024-01-21T11:43:01.332 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:43:01.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:43:01 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:42:59.809171+0000 mgr.smithi029.blhxxk (mgr.14164) 30 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:43:01.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:43:01 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:43:01.329738+0000 mon.smithi029 (mon.0) 129 : audit [DBG] from='client.? 172.21.15.134:0/511891377' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:43:02.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:43:02 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:43:01.937637+0000 mon.smithi029 (mon.0) 130 : audit [INF] from='mgr.14164 172.21.15.29:0/3252518221' entity='mgr.smithi029.blhxxk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi029.blhxxk/mirror_snapshot_schedule"}]: dispatch 2024-01-21T11:43:02.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:43:02 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:43:01.940097+0000 mon.smithi029 (mon.0) 131 : audit [INF] from='mgr.14164 172.21.15.29:0/3252518221' entity='mgr.smithi029.blhxxk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi029.blhxxk/trash_purge_schedule"}]: dispatch 2024-01-21T11:43:03.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:43:03 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:43:01.809427+0000 mgr.smithi029.blhxxk (mgr.14164) 31 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:43:04.858 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:43:04.859 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:43:05.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:43:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:43:03.809926+0000 mgr.smithi029.blhxxk (mgr.14164) 32 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:43:07.436 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:07.434+0000 7fda42b5d700 1 -- 172.21.15.134:0/2601287941 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda3c0f5410 msgr2=0x7fda3c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:43:07.436 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:07.434+0000 7fda42b5d700 1 --2- 172.21.15.134:0/2601287941 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda3c0f5410 0x7fda3c0f5830 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7fda30009fb0 tx=0x7fda30009b30).stop 2024-01-21T11:43:07.436 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:07.434+0000 7fda42b5d700 1 -- 172.21.15.134:0/2601287941 shutdown_connections 2024-01-21T11:43:07.436 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:07.434+0000 7fda42b5d700 1 --2- 172.21.15.134:0/2601287941 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda3c0f5410 0x7fda3c0f5830 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:43:07.436 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:07.434+0000 7fda42b5d700 1 -- 172.21.15.134:0/2601287941 >> 172.21.15.134:0/2601287941 conn(0x7fda3c0f0b70 msgr2=0x7fda3c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:43:07.436 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:07.434+0000 7fda42b5d700 1 -- 172.21.15.134:0/2601287941 shutdown_connections 2024-01-21T11:43:07.436 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:07.434+0000 7fda42b5d700 1 -- 172.21.15.134:0/2601287941 wait complete. 2024-01-21T11:43:07.436 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:07.435+0000 7fda42b5d700 1 Processor -- start 2024-01-21T11:43:07.437 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:07.435+0000 7fda42b5d700 1 -- start start 2024-01-21T11:43:07.437 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:07.435+0000 7fda42b5d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda3c0f5410 0x7fda3c10b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:43:07.437 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:07.435+0000 7fda42b5d700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda3c107240 con 0x7fda3c0f5410 2024-01-21T11:43:07.437 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:07.435+0000 7fda408f9700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda3c0f5410 0x7fda3c10b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:43:07.437 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:07.435+0000 7fda408f9700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda3c0f5410 0x7fda3c10b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:47392/0 (socket says 172.21.15.134:47392) 2024-01-21T11:43:07.437 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:07.435+0000 7fda408f9700 1 -- 172.21.15.134:0/531991084 learned_addr learned my addr 172.21.15.134:0/531991084 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:43:07.437 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:07.436+0000 7fda408f9700 1 -- 172.21.15.134:0/531991084 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda30005040 con 0x7fda3c0f5410 2024-01-21T11:43:07.437 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:07.436+0000 7fda408f9700 1 --2- 172.21.15.134:0/531991084 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda3c0f5410 0x7fda3c10b850 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7fda30013510 tx=0x7fda3000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:43:07.437 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:07.436+0000 7fda397fa700 1 -- 172.21.15.134:0/531991084 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda30009320 con 0x7fda3c0f5410 2024-01-21T11:43:07.437 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:07.436+0000 7fda42b5d700 1 -- 172.21.15.134:0/531991084 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda3c060220 con 0x7fda3c0f5410 2024-01-21T11:43:07.437 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:07.436+0000 7fda397fa700 1 -- 172.21.15.134:0/531991084 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fda30013f00 con 0x7fda3c0f5410 2024-01-21T11:43:07.437 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:07.436+0000 7fda397fa700 1 -- 172.21.15.134:0/531991084 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda30014d90 con 0x7fda3c0f5410 2024-01-21T11:43:07.437 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:07.436+0000 7fda42b5d700 1 -- 172.21.15.134:0/531991084 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda3c10bd40 con 0x7fda3c0f5410 2024-01-21T11:43:07.438 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:07.437+0000 7fda397fa700 1 -- 172.21.15.134:0/531991084 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fda30014420 con 0x7fda3c0f5410 2024-01-21T11:43:07.438 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:07.437+0000 7fda42b5d700 1 -- 172.21.15.134:0/531991084 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda3c045ad0 con 0x7fda3c0f5410 2024-01-21T11:43:07.438 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:07.437+0000 7fda397fa700 1 --2- 172.21.15.134:0/531991084 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fda2c02ff60 0x7fda2c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:43:07.438 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:07.437+0000 7fda397fa700 1 -- 172.21.15.134:0/531991084 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fda30041070 con 0x7fda3c0f5410 2024-01-21T11:43:07.439 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:07.437+0000 7fda3bfff700 1 --2- 172.21.15.134:0/531991084 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fda2c02ff60 0x7fda2c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:43:07.439 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:07.438+0000 7fda3bfff700 1 --2- 172.21.15.134:0/531991084 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fda2c02ff60 0x7fda2c032420 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7fda280096a0 tx=0x7fda28006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:43:07.442 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:07.442+0000 7fda397fa700 1 -- 172.21.15.134:0/531991084 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fda300095f0 con 0x7fda3c0f5410 2024-01-21T11:43:07.646 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:07.646+0000 7fda42b5d700 1 -- 172.21.15.134:0/531991084 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fda3c0fae60 con 0x7fda3c0f5410 2024-01-21T11:43:07.647 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:07.647+0000 7fda397fa700 1 -- 172.21.15.134:0/531991084 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fda3001db70 con 0x7fda3c0f5410 2024-01-21T11:43:07.647 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:43:07.647 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:43:07.649 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:07.648+0000 7fda42b5d700 1 -- 172.21.15.134:0/531991084 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fda2c02ff60 msgr2=0x7fda2c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:43:07.649 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:07.648+0000 7fda42b5d700 1 --2- 172.21.15.134:0/531991084 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fda2c02ff60 0x7fda2c032420 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7fda280096a0 tx=0x7fda28006b40).stop 2024-01-21T11:43:07.649 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:07.648+0000 7fda42b5d700 1 -- 172.21.15.134:0/531991084 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda3c0f5410 msgr2=0x7fda3c10b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:43:07.649 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:07.648+0000 7fda42b5d700 1 --2- 172.21.15.134:0/531991084 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda3c0f5410 0x7fda3c10b850 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7fda30013510 tx=0x7fda3000b040).stop 2024-01-21T11:43:07.649 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:07.648+0000 7fda42b5d700 1 -- 172.21.15.134:0/531991084 shutdown_connections 2024-01-21T11:43:07.649 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:07.648+0000 7fda42b5d700 1 --2- 172.21.15.134:0/531991084 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fda2c02ff60 0x7fda2c032420 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:43:07.649 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:07.648+0000 7fda42b5d700 1 --2- 172.21.15.134:0/531991084 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda3c0f5410 0x7fda3c10b850 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:43:07.649 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:07.648+0000 7fda42b5d700 1 -- 172.21.15.134:0/531991084 >> 172.21.15.134:0/531991084 conn(0x7fda3c0f0b70 msgr2=0x7fda3c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:43:07.649 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:07.648+0000 7fda42b5d700 1 -- 172.21.15.134:0/531991084 shutdown_connections 2024-01-21T11:43:07.649 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:07.648+0000 7fda42b5d700 1 -- 172.21.15.134:0/531991084 wait complete. 2024-01-21T11:43:07.650 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:43:07.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:43:07 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:43:05.810344+0000 mgr.smithi029.blhxxk (mgr.14164) 33 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:43:08.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:43:08 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:43:07.647955+0000 mon.smithi029 (mon.0) 132 : audit [DBG] from='client.? 172.21.15.134:0/531991084' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:43:09.043 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:43:09.044 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:43:09.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:43:09 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:43:07.810685+0000 mgr.smithi029.blhxxk (mgr.14164) 34 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:43:10.207 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:10.205+0000 7fea09589700 1 -- 172.21.15.134:0/3061462253 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea040f5410 msgr2=0x7fea040f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:43:10.207 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:10.205+0000 7fea09589700 1 --2- 172.21.15.134:0/3061462253 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea040f5410 0x7fea040f5830 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7fe9f8009fb0 tx=0x7fe9f8009b30).stop 2024-01-21T11:43:10.207 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:10.205+0000 7fea09589700 1 -- 172.21.15.134:0/3061462253 shutdown_connections 2024-01-21T11:43:10.208 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:10.205+0000 7fea09589700 1 --2- 172.21.15.134:0/3061462253 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea040f5410 0x7fea040f5830 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:43:10.208 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:10.205+0000 7fea09589700 1 -- 172.21.15.134:0/3061462253 >> 172.21.15.134:0/3061462253 conn(0x7fea040f0b70 msgr2=0x7fea040f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:43:10.208 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:10.206+0000 7fea09589700 1 -- 172.21.15.134:0/3061462253 shutdown_connections 2024-01-21T11:43:10.208 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:10.206+0000 7fea09589700 1 -- 172.21.15.134:0/3061462253 wait complete. 2024-01-21T11:43:10.208 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:10.206+0000 7fea09589700 1 Processor -- start 2024-01-21T11:43:10.208 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:10.206+0000 7fea09589700 1 -- start start 2024-01-21T11:43:10.208 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:10.206+0000 7fea09589700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea040f5410 0x7fea040fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:43:10.208 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:10.206+0000 7fea09589700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea040fb820 con 0x7fea040f5410 2024-01-21T11:43:10.208 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:10.206+0000 7fea02ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea040f5410 0x7fea040fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:43:10.208 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:10.207+0000 7fea02ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea040f5410 0x7fea040fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:47406/0 (socket says 172.21.15.134:47406) 2024-01-21T11:43:10.208 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:10.207+0000 7fea02ffd700 1 -- 172.21.15.134:0/2641469846 learned_addr learned my addr 172.21.15.134:0/2641469846 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:43:10.208 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:10.207+0000 7fea02ffd700 1 -- 172.21.15.134:0/2641469846 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9f8005040 con 0x7fea040f5410 2024-01-21T11:43:10.208 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:10.207+0000 7fea02ffd700 1 --2- 172.21.15.134:0/2641469846 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea040f5410 0x7fea040fd3a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fe9f8013510 tx=0x7fe9f800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:43:10.209 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:10.207+0000 7fe9eb7fe700 1 -- 172.21.15.134:0/2641469846 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe9f8009320 con 0x7fea040f5410 2024-01-21T11:43:10.209 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:10.207+0000 7fea09589700 1 -- 172.21.15.134:0/2641469846 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea040fb9c0 con 0x7fea040f5410 2024-01-21T11:43:10.209 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:10.207+0000 7fe9eb7fe700 1 -- 172.21.15.134:0/2641469846 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe9f8013f00 con 0x7fea040f5410 2024-01-21T11:43:10.209 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:10.207+0000 7fea09589700 1 -- 172.21.15.134:0/2641469846 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea040fbe50 con 0x7fea040f5410 2024-01-21T11:43:10.209 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:10.207+0000 7fe9eb7fe700 1 -- 172.21.15.134:0/2641469846 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe9f8014d90 con 0x7fea040f5410 2024-01-21T11:43:10.210 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:10.208+0000 7fe9eb7fe700 1 -- 172.21.15.134:0/2641469846 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe9f8014420 con 0x7fea040f5410 2024-01-21T11:43:10.210 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:10.208+0000 7fea09589700 1 -- 172.21.15.134:0/2641469846 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea04059100 con 0x7fea040f5410 2024-01-21T11:43:10.210 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:10.208+0000 7fe9eb7fe700 1 --2- 172.21.15.134:0/2641469846 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fe9ec02ff60 0x7fe9ec032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:43:10.210 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:10.208+0000 7fe9eb7fe700 1 -- 172.21.15.134:0/2641469846 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe9f80410e0 con 0x7fea040f5410 2024-01-21T11:43:10.210 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:10.208+0000 7fea027fc700 1 --2- 172.21.15.134:0/2641469846 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fe9ec02ff60 0x7fe9ec032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:43:10.210 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:10.209+0000 7fea027fc700 1 --2- 172.21.15.134:0/2641469846 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fe9ec02ff60 0x7fe9ec032420 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7fe9f40096a0 tx=0x7fe9f4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:43:10.213 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:10.213+0000 7fe9eb7fe700 1 -- 172.21.15.134:0/2641469846 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe9f80095f0 con 0x7fea040f5410 2024-01-21T11:43:10.424 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:10.423+0000 7fea09589700 1 -- 172.21.15.134:0/2641469846 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fea04045ad0 con 0x7fea040f5410 2024-01-21T11:43:10.424 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:10.424+0000 7fe9eb7fe700 1 -- 172.21.15.134:0/2641469846 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe9f801db70 con 0x7fea040f5410 2024-01-21T11:43:10.425 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:43:10.425 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:43:10.426 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:10.426+0000 7fea09589700 1 -- 172.21.15.134:0/2641469846 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fe9ec02ff60 msgr2=0x7fe9ec032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:43:10.426 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:10.426+0000 7fea09589700 1 --2- 172.21.15.134:0/2641469846 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fe9ec02ff60 0x7fe9ec032420 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7fe9f40096a0 tx=0x7fe9f4006b40).stop 2024-01-21T11:43:10.427 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:10.426+0000 7fea09589700 1 -- 172.21.15.134:0/2641469846 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea040f5410 msgr2=0x7fea040fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:43:10.427 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:10.426+0000 7fea09589700 1 --2- 172.21.15.134:0/2641469846 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea040f5410 0x7fea040fd3a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fe9f8013510 tx=0x7fe9f800b040).stop 2024-01-21T11:43:10.427 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:10.426+0000 7fea09589700 1 -- 172.21.15.134:0/2641469846 shutdown_connections 2024-01-21T11:43:10.427 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:10.426+0000 7fea09589700 1 --2- 172.21.15.134:0/2641469846 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fe9ec02ff60 0x7fe9ec032420 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:43:10.427 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:10.426+0000 7fea09589700 1 --2- 172.21.15.134:0/2641469846 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea040f5410 0x7fea040fd3a0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:43:10.427 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:10.426+0000 7fea09589700 1 -- 172.21.15.134:0/2641469846 >> 172.21.15.134:0/2641469846 conn(0x7fea040f0b70 msgr2=0x7fea0405fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:43:10.427 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:10.426+0000 7fea09589700 1 -- 172.21.15.134:0/2641469846 shutdown_connections 2024-01-21T11:43:10.427 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:10.426+0000 7fea09589700 1 -- 172.21.15.134:0/2641469846 wait complete. 2024-01-21T11:43:10.427 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:43:10.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:43:10 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:43:10.425404+0000 mon.smithi029 (mon.0) 133 : audit [DBG] from='client.? 172.21.15.134:0/2641469846' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:43:11.811 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:43:11.812 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:43:11.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:43:11 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:43:09.811063+0000 mgr.smithi029.blhxxk (mgr.14164) 35 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:43:13.004 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:13.003+0000 7f0ad934d700 1 -- 172.21.15.134:0/3234652949 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ad40f6510 msgr2=0x7f0ad40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:43:13.004 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:13.003+0000 7f0ad934d700 1 --2- 172.21.15.134:0/3234652949 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ad40f6510 0x7f0ad40f6930 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f0abc009fb0 tx=0x7f0abc009b30).stop 2024-01-21T11:43:13.004 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:13.003+0000 7f0ad934d700 1 -- 172.21.15.134:0/3234652949 shutdown_connections 2024-01-21T11:43:13.004 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:13.003+0000 7f0ad934d700 1 --2- 172.21.15.134:0/3234652949 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ad40f6510 0x7f0ad40f6930 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:43:13.004 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:13.003+0000 7f0ad934d700 1 -- 172.21.15.134:0/3234652949 >> 172.21.15.134:0/3234652949 conn(0x7f0ad40f1c70 msgr2=0x7f0ad40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:43:13.004 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:13.003+0000 7f0ad934d700 1 -- 172.21.15.134:0/3234652949 shutdown_connections 2024-01-21T11:43:13.004 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:13.003+0000 7f0ad934d700 1 -- 172.21.15.134:0/3234652949 wait complete. 2024-01-21T11:43:13.005 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:13.004+0000 7f0ad934d700 1 Processor -- start 2024-01-21T11:43:13.005 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:13.004+0000 7f0ad934d700 1 -- start start 2024-01-21T11:43:13.006 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:13.004+0000 7f0ad934d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ad40f6510 0x7f0ad4106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:43:13.006 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:13.004+0000 7f0ad934d700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad4106c20 con 0x7f0ad40f6510 2024-01-21T11:43:13.006 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:13.004+0000 7f0ad2ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ad40f6510 0x7f0ad4106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:43:13.006 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:13.005+0000 7f0ad2ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ad40f6510 0x7f0ad4106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:47432/0 (socket says 172.21.15.134:47432) 2024-01-21T11:43:13.006 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:13.005+0000 7f0ad2ffd700 1 -- 172.21.15.134:0/1494300939 learned_addr learned my addr 172.21.15.134:0/1494300939 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:43:13.006 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:13.005+0000 7f0ad2ffd700 1 -- 172.21.15.134:0/1494300939 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0abc005040 con 0x7f0ad40f6510 2024-01-21T11:43:13.006 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:13.005+0000 7f0ad2ffd700 1 --2- 172.21.15.134:0/1494300939 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ad40f6510 0x7f0ad4106730 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f0abc013510 tx=0x7f0abc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:43:13.006 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:13.005+0000 7f0acb7fe700 1 -- 172.21.15.134:0/1494300939 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0abc009320 con 0x7f0ad40f6510 2024-01-21T11:43:13.006 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:13.005+0000 7f0ad934d700 1 -- 172.21.15.134:0/1494300939 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ad4106e20 con 0x7f0ad40f6510 2024-01-21T11:43:13.006 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:13.005+0000 7f0acb7fe700 1 -- 172.21.15.134:0/1494300939 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0abc013f00 con 0x7f0ad40f6510 2024-01-21T11:43:13.006 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:13.005+0000 7f0acb7fe700 1 -- 172.21.15.134:0/1494300939 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0abc014d90 con 0x7f0ad40f6510 2024-01-21T11:43:13.006 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:13.005+0000 7f0ad934d700 1 -- 172.21.15.134:0/1494300939 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ad41072b0 con 0x7f0ad40f6510 2024-01-21T11:43:13.007 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:13.006+0000 7f0acb7fe700 1 -- 172.21.15.134:0/1494300939 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0abc014420 con 0x7f0ad40f6510 2024-01-21T11:43:13.007 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:13.006+0000 7f0ad934d700 1 -- 172.21.15.134:0/1494300939 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ad4045ad0 con 0x7f0ad40f6510 2024-01-21T11:43:13.007 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:13.006+0000 7f0acb7fe700 1 --2- 172.21.15.134:0/1494300939 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f0ac002ff60 0x7f0ac0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:43:13.007 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:13.006+0000 7f0acb7fe700 1 -- 172.21.15.134:0/1494300939 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0abc041020 con 0x7f0ad40f6510 2024-01-21T11:43:13.007 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:13.006+0000 7f0ad27fc700 1 --2- 172.21.15.134:0/1494300939 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f0ac002ff60 0x7f0ac0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:43:13.007 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:13.007+0000 7f0ad27fc700 1 --2- 172.21.15.134:0/1494300939 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f0ac002ff60 0x7f0ac0032420 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f0ac40096a0 tx=0x7f0ac4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:43:13.011 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:13.011+0000 7f0acb7fe700 1 -- 172.21.15.134:0/1494300939 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0abc0095f0 con 0x7f0ad40f6510 2024-01-21T11:43:13.217 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:13.216+0000 7f0ad934d700 1 -- 172.21.15.134:0/1494300939 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0ad40fbf60 con 0x7f0ad40f6510 2024-01-21T11:43:13.217 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:13.217+0000 7f0acb7fe700 1 -- 172.21.15.134:0/1494300939 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0abc01db70 con 0x7f0ad40f6510 2024-01-21T11:43:13.217 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:43:13.218 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:43:13.219 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:13.219+0000 7f0ad934d700 1 -- 172.21.15.134:0/1494300939 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f0ac002ff60 msgr2=0x7f0ac0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:43:13.219 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:13.219+0000 7f0ad934d700 1 --2- 172.21.15.134:0/1494300939 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f0ac002ff60 0x7f0ac0032420 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f0ac40096a0 tx=0x7f0ac4006b40).stop 2024-01-21T11:43:13.219 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:13.219+0000 7f0ad934d700 1 -- 172.21.15.134:0/1494300939 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ad40f6510 msgr2=0x7f0ad4106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:43:13.219 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:13.219+0000 7f0ad934d700 1 --2- 172.21.15.134:0/1494300939 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ad40f6510 0x7f0ad4106730 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f0abc013510 tx=0x7f0abc00b040).stop 2024-01-21T11:43:13.219 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:13.219+0000 7f0ad934d700 1 -- 172.21.15.134:0/1494300939 shutdown_connections 2024-01-21T11:43:13.219 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:13.219+0000 7f0ad934d700 1 --2- 172.21.15.134:0/1494300939 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f0ac002ff60 0x7f0ac0032420 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:43:13.220 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:13.219+0000 7f0ad934d700 1 --2- 172.21.15.134:0/1494300939 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ad40f6510 0x7f0ad4106730 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:43:13.220 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:13.219+0000 7f0ad934d700 1 -- 172.21.15.134:0/1494300939 >> 172.21.15.134:0/1494300939 conn(0x7f0ad40f1c70 msgr2=0x7f0ad40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:43:13.220 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:13.219+0000 7f0ad934d700 1 -- 172.21.15.134:0/1494300939 shutdown_connections 2024-01-21T11:43:13.220 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:13.219+0000 7f0ad934d700 1 -- 172.21.15.134:0/1494300939 wait complete. 2024-01-21T11:43:13.220 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:43:13.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:43:13 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:43:11.811465+0000 mgr.smithi029.blhxxk (mgr.14164) 36 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:43:13.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:43:13 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:43:13.218398+0000 mon.smithi029 (mon.0) 134 : audit [DBG] from='client.? 172.21.15.134:0/1494300939' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:43:14.624 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:43:14.625 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:43:15.768 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:15.767+0000 7efc0cfa0700 1 -- 172.21.15.134:0/2089856495 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc080f5410 msgr2=0x7efc080f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:43:15.769 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:15.767+0000 7efc0cfa0700 1 --2- 172.21.15.134:0/2089856495 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc080f5410 0x7efc080f5830 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7efbf0009fb0 tx=0x7efbf0009b30).stop 2024-01-21T11:43:15.769 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:15.767+0000 7efc0cfa0700 1 -- 172.21.15.134:0/2089856495 shutdown_connections 2024-01-21T11:43:15.769 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:15.768+0000 7efc0cfa0700 1 --2- 172.21.15.134:0/2089856495 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc080f5410 0x7efc080f5830 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:43:15.769 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:15.768+0000 7efc0cfa0700 1 -- 172.21.15.134:0/2089856495 >> 172.21.15.134:0/2089856495 conn(0x7efc080f0b70 msgr2=0x7efc080f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:43:15.769 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:15.768+0000 7efc0cfa0700 1 -- 172.21.15.134:0/2089856495 shutdown_connections 2024-01-21T11:43:15.769 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:15.768+0000 7efc0cfa0700 1 -- 172.21.15.134:0/2089856495 wait complete. 2024-01-21T11:43:15.769 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:15.768+0000 7efc0cfa0700 1 Processor -- start 2024-01-21T11:43:15.769 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:15.768+0000 7efc0cfa0700 1 -- start start 2024-01-21T11:43:15.769 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:15.768+0000 7efc0cfa0700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc080f5410 0x7efc080fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:43:15.769 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:15.768+0000 7efc0cfa0700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc080fb820 con 0x7efc080f5410 2024-01-21T11:43:15.769 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:15.768+0000 7efc0659c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc080f5410 0x7efc080fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:43:15.769 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:15.769+0000 7efc0659c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc080f5410 0x7efc080fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:36284/0 (socket says 172.21.15.134:36284) 2024-01-21T11:43:15.769 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:15.769+0000 7efc0659c700 1 -- 172.21.15.134:0/1958028388 learned_addr learned my addr 172.21.15.134:0/1958028388 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:43:15.769 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:15.769+0000 7efc0659c700 1 -- 172.21.15.134:0/1958028388 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbf0005040 con 0x7efc080f5410 2024-01-21T11:43:15.770 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:15.769+0000 7efc0659c700 1 --2- 172.21.15.134:0/1958028388 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc080f5410 0x7efc080fd3a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7efbf0014000 tx=0x7efbf000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:43:15.770 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:15.769+0000 7efbfeffd700 1 -- 172.21.15.134:0/1958028388 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efbf00095f0 con 0x7efc080f5410 2024-01-21T11:43:15.770 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:15.769+0000 7efc0cfa0700 1 -- 172.21.15.134:0/1958028388 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc080fb9c0 con 0x7efc080f5410 2024-01-21T11:43:15.770 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:15.769+0000 7efbfeffd700 1 -- 172.21.15.134:0/1958028388 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efbf0007e70 con 0x7efc080f5410 2024-01-21T11:43:15.770 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:15.769+0000 7efbfeffd700 1 -- 172.21.15.134:0/1958028388 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efbf0014ea0 con 0x7efc080f5410 2024-01-21T11:43:15.770 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:15.769+0000 7efc0cfa0700 1 -- 172.21.15.134:0/1958028388 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc080fbe50 con 0x7efc080f5410 2024-01-21T11:43:15.771 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:15.770+0000 7efbfeffd700 1 -- 172.21.15.134:0/1958028388 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efbf00079c0 con 0x7efc080f5410 2024-01-21T11:43:15.771 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:15.770+0000 7efc0cfa0700 1 -- 172.21.15.134:0/1958028388 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc08059100 con 0x7efc080f5410 2024-01-21T11:43:15.771 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:15.770+0000 7efbfeffd700 1 --2- 172.21.15.134:0/1958028388 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7efbf402ff60 0x7efbf4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:43:15.771 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:15.770+0000 7efbfeffd700 1 -- 172.21.15.134:0/1958028388 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efbf0040920 con 0x7efc080f5410 2024-01-21T11:43:15.771 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:15.770+0000 7efc05d9b700 1 --2- 172.21.15.134:0/1958028388 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7efbf402ff60 0x7efbf4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:43:15.771 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:15.771+0000 7efc05d9b700 1 --2- 172.21.15.134:0/1958028388 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7efbf402ff60 0x7efbf4032420 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7efbf80096a0 tx=0x7efbf8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:43:15.774 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:15.775+0000 7efbfeffd700 1 -- 172.21.15.134:0/1958028388 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efbf000f950 con 0x7efc080f5410 2024-01-21T11:43:15.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:43:15 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:43:13.811817+0000 mgr.smithi029.blhxxk (mgr.14164) 37 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:43:15.981 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:15.982+0000 7efc0cfa0700 1 -- 172.21.15.134:0/1958028388 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efc080fae60 con 0x7efc080f5410 2024-01-21T11:43:15.982 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:15.983+0000 7efbfeffd700 1 -- 172.21.15.134:0/1958028388 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7efbf0010230 con 0x7efc080f5410 2024-01-21T11:43:15.983 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:43:15.983 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:43:15.984 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:15.985+0000 7efc0cfa0700 1 -- 172.21.15.134:0/1958028388 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7efbf402ff60 msgr2=0x7efbf4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:43:15.984 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:15.985+0000 7efc0cfa0700 1 --2- 172.21.15.134:0/1958028388 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7efbf402ff60 0x7efbf4032420 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7efbf80096a0 tx=0x7efbf8006b40).stop 2024-01-21T11:43:15.984 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:15.985+0000 7efc0cfa0700 1 -- 172.21.15.134:0/1958028388 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc080f5410 msgr2=0x7efc080fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:43:15.984 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:15.985+0000 7efc0cfa0700 1 --2- 172.21.15.134:0/1958028388 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc080f5410 0x7efc080fd3a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7efbf0014000 tx=0x7efbf000b040).stop 2024-01-21T11:43:15.984 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:15.985+0000 7efc0cfa0700 1 -- 172.21.15.134:0/1958028388 shutdown_connections 2024-01-21T11:43:15.984 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:15.985+0000 7efc0cfa0700 1 --2- 172.21.15.134:0/1958028388 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7efbf402ff60 0x7efbf4032420 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:43:15.984 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:15.985+0000 7efc0cfa0700 1 --2- 172.21.15.134:0/1958028388 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc080f5410 0x7efc080fd3a0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:43:15.984 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:15.985+0000 7efc0cfa0700 1 -- 172.21.15.134:0/1958028388 >> 172.21.15.134:0/1958028388 conn(0x7efc080f0b70 msgr2=0x7efc0805fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:43:15.984 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:15.985+0000 7efc0cfa0700 1 -- 172.21.15.134:0/1958028388 shutdown_connections 2024-01-21T11:43:15.985 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:15.985+0000 7efc0cfa0700 1 -- 172.21.15.134:0/1958028388 wait complete. 2024-01-21T11:43:15.985 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:43:16.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:43:16 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:43:15.983467+0000 mon.smithi029 (mon.0) 135 : audit [DBG] from='client.? 172.21.15.134:0/1958028388' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:43:17.375 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:43:17.376 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:43:17.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:43:17 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:43:15.812182+0000 mgr.smithi029.blhxxk (mgr.14164) 38 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:43:18.568 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:18.567+0000 7f6e071e1700 1 -- 172.21.15.134:0/66751713 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e000f5410 msgr2=0x7f6e000f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:43:18.568 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:18.567+0000 7f6e071e1700 1 --2- 172.21.15.134:0/66751713 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e000f5410 0x7f6e000f5830 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f6df4009fb0 tx=0x7f6df4009b30).stop 2024-01-21T11:43:18.568 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:18.567+0000 7f6e071e1700 1 -- 172.21.15.134:0/66751713 shutdown_connections 2024-01-21T11:43:18.568 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:18.567+0000 7f6e071e1700 1 --2- 172.21.15.134:0/66751713 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e000f5410 0x7f6e000f5830 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:43:18.568 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:18.567+0000 7f6e071e1700 1 -- 172.21.15.134:0/66751713 >> 172.21.15.134:0/66751713 conn(0x7f6e000f0b70 msgr2=0x7f6e000f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:43:18.568 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:18.567+0000 7f6e071e1700 1 -- 172.21.15.134:0/66751713 shutdown_connections 2024-01-21T11:43:18.568 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:18.567+0000 7f6e071e1700 1 -- 172.21.15.134:0/66751713 wait complete. 2024-01-21T11:43:18.568 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:18.567+0000 7f6e071e1700 1 Processor -- start 2024-01-21T11:43:18.568 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:18.567+0000 7f6e071e1700 1 -- start start 2024-01-21T11:43:18.569 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:18.568+0000 7f6e071e1700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e000f5410 0x7f6e000fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:43:18.569 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:18.568+0000 7f6e071e1700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e000fb820 con 0x7f6e000f5410 2024-01-21T11:43:18.569 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:18.568+0000 7f6e04f7d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e000f5410 0x7f6e000fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:43:18.569 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:18.568+0000 7f6e04f7d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e000f5410 0x7f6e000fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:36298/0 (socket says 172.21.15.134:36298) 2024-01-21T11:43:18.569 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:18.568+0000 7f6e04f7d700 1 -- 172.21.15.134:0/3930772931 learned_addr learned my addr 172.21.15.134:0/3930772931 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:43:18.570 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:18.568+0000 7f6e04f7d700 1 -- 172.21.15.134:0/3930772931 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6df4005040 con 0x7f6e000f5410 2024-01-21T11:43:18.570 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:18.568+0000 7f6e04f7d700 1 --2- 172.21.15.134:0/3930772931 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e000f5410 0x7f6e000fd3a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f6df4014000 tx=0x7f6df400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:43:18.570 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:18.568+0000 7f6dfd7fa700 1 -- 172.21.15.134:0/3930772931 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6df40095f0 con 0x7f6e000f5410 2024-01-21T11:43:18.570 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:18.569+0000 7f6dfd7fa700 1 -- 172.21.15.134:0/3930772931 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6df4007e70 con 0x7f6e000f5410 2024-01-21T11:43:18.571 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:18.569+0000 7f6e071e1700 1 -- 172.21.15.134:0/3930772931 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e000fb9c0 con 0x7f6e000f5410 2024-01-21T11:43:18.571 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:18.569+0000 7f6dfd7fa700 1 -- 172.21.15.134:0/3930772931 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6df4014ea0 con 0x7f6e000f5410 2024-01-21T11:43:18.571 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:18.569+0000 7f6e071e1700 1 -- 172.21.15.134:0/3930772931 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e000fbe50 con 0x7f6e000f5410 2024-01-21T11:43:18.571 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:18.569+0000 7f6dfd7fa700 1 -- 172.21.15.134:0/3930772931 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6df4025070 con 0x7f6e000f5410 2024-01-21T11:43:18.571 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:18.569+0000 7f6e071e1700 1 -- 172.21.15.134:0/3930772931 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e00059100 con 0x7f6e000f5410 2024-01-21T11:43:18.571 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:18.569+0000 7f6dfd7fa700 1 --2- 172.21.15.134:0/3930772931 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f6df002ff60 0x7f6df0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:43:18.571 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:18.569+0000 7f6dfd7fa700 1 -- 172.21.15.134:0/3930772931 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6df40409f0 con 0x7f6e000f5410 2024-01-21T11:43:18.571 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:18.570+0000 7f6dfffff700 1 --2- 172.21.15.134:0/3930772931 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f6df002ff60 0x7f6df0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:43:18.571 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:18.570+0000 7f6dfffff700 1 --2- 172.21.15.134:0/3930772931 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f6df002ff60 0x7f6df0032420 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f6dec0096a0 tx=0x7f6dec006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:43:18.574 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:18.574+0000 7f6dfd7fa700 1 -- 172.21.15.134:0/3930772931 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6df400f4e0 con 0x7f6e000f5410 2024-01-21T11:43:18.781 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:18.781+0000 7f6e071e1700 1 -- 172.21.15.134:0/3930772931 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6e000fae60 con 0x7f6e000f5410 2024-01-21T11:43:18.782 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:18.782+0000 7f6dfd7fa700 1 -- 172.21.15.134:0/3930772931 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6df400f300 con 0x7f6e000f5410 2024-01-21T11:43:18.782 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:43:18.782 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:43:18.784 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:18.784+0000 7f6e071e1700 1 -- 172.21.15.134:0/3930772931 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f6df002ff60 msgr2=0x7f6df0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:43:18.784 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:18.784+0000 7f6e071e1700 1 --2- 172.21.15.134:0/3930772931 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f6df002ff60 0x7f6df0032420 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f6dec0096a0 tx=0x7f6dec006b40).stop 2024-01-21T11:43:18.784 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:18.784+0000 7f6e071e1700 1 -- 172.21.15.134:0/3930772931 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e000f5410 msgr2=0x7f6e000fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:43:18.784 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:18.784+0000 7f6e071e1700 1 --2- 172.21.15.134:0/3930772931 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e000f5410 0x7f6e000fd3a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f6df4014000 tx=0x7f6df400b040).stop 2024-01-21T11:43:18.784 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:18.784+0000 7f6e071e1700 1 -- 172.21.15.134:0/3930772931 shutdown_connections 2024-01-21T11:43:18.784 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:18.784+0000 7f6e071e1700 1 --2- 172.21.15.134:0/3930772931 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f6df002ff60 0x7f6df0032420 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:43:18.784 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:18.784+0000 7f6e071e1700 1 --2- 172.21.15.134:0/3930772931 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e000f5410 0x7f6e000fd3a0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:43:18.785 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:18.784+0000 7f6e071e1700 1 -- 172.21.15.134:0/3930772931 >> 172.21.15.134:0/3930772931 conn(0x7f6e000f0b70 msgr2=0x7f6e0005fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:43:18.785 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:18.784+0000 7f6e071e1700 1 -- 172.21.15.134:0/3930772931 shutdown_connections 2024-01-21T11:43:18.785 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:18.784+0000 7f6e071e1700 1 -- 172.21.15.134:0/3930772931 wait complete. 2024-01-21T11:43:18.785 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:43:19.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:43:19 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:43:17.812615+0000 mgr.smithi029.blhxxk (mgr.14164) 39 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:43:19.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:43:19 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:43:18.783234+0000 mon.smithi029 (mon.0) 136 : audit [DBG] from='client.? 172.21.15.134:0/3930772931' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:43:20.165 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:43:20.165 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:43:21.362 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:21.360+0000 7fc08f34f700 1 -- 172.21.15.134:0/2936768165 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0880f6510 msgr2=0x7fc0880f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:43:21.362 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:21.360+0000 7fc08f34f700 1 --2- 172.21.15.134:0/2936768165 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0880f6510 0x7fc0880f6930 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7fc078009fb0 tx=0x7fc078009b30).stop 2024-01-21T11:43:21.362 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:21.360+0000 7fc08f34f700 1 -- 172.21.15.134:0/2936768165 shutdown_connections 2024-01-21T11:43:21.362 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:21.360+0000 7fc08f34f700 1 --2- 172.21.15.134:0/2936768165 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0880f6510 0x7fc0880f6930 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:43:21.362 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:21.360+0000 7fc08f34f700 1 -- 172.21.15.134:0/2936768165 >> 172.21.15.134:0/2936768165 conn(0x7fc0880f1c70 msgr2=0x7fc0880f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:43:21.362 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:21.360+0000 7fc08f34f700 1 -- 172.21.15.134:0/2936768165 shutdown_connections 2024-01-21T11:43:21.362 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:21.360+0000 7fc08f34f700 1 -- 172.21.15.134:0/2936768165 wait complete. 2024-01-21T11:43:21.362 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:21.361+0000 7fc08f34f700 1 Processor -- start 2024-01-21T11:43:21.362 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:21.361+0000 7fc08f34f700 1 -- start start 2024-01-21T11:43:21.362 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:21.361+0000 7fc08f34f700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0880f6510 0x7fc088106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:43:21.363 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:21.361+0000 7fc08f34f700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc088106c20 con 0x7fc0880f6510 2024-01-21T11:43:21.363 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:21.361+0000 7fc08d0eb700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0880f6510 0x7fc088106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:43:21.363 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:21.361+0000 7fc08d0eb700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0880f6510 0x7fc088106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:36328/0 (socket says 172.21.15.134:36328) 2024-01-21T11:43:21.363 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:21.361+0000 7fc08d0eb700 1 -- 172.21.15.134:0/3182783219 learned_addr learned my addr 172.21.15.134:0/3182783219 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:43:21.363 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:21.361+0000 7fc08d0eb700 1 -- 172.21.15.134:0/3182783219 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc078005040 con 0x7fc0880f6510 2024-01-21T11:43:21.363 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:21.361+0000 7fc08d0eb700 1 --2- 172.21.15.134:0/3182783219 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0880f6510 0x7fc088106730 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7fc078013510 tx=0x7fc07800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:43:21.363 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:21.362+0000 7fc07dffb700 1 -- 172.21.15.134:0/3182783219 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc078009320 con 0x7fc0880f6510 2024-01-21T11:43:21.363 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:21.362+0000 7fc07dffb700 1 -- 172.21.15.134:0/3182783219 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc078013f00 con 0x7fc0880f6510 2024-01-21T11:43:21.363 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:21.362+0000 7fc08f34f700 1 -- 172.21.15.134:0/3182783219 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc088106e20 con 0x7fc0880f6510 2024-01-21T11:43:21.363 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:21.362+0000 7fc07dffb700 1 -- 172.21.15.134:0/3182783219 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc078014d90 con 0x7fc0880f6510 2024-01-21T11:43:21.363 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:21.362+0000 7fc08f34f700 1 -- 172.21.15.134:0/3182783219 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0881072b0 con 0x7fc0880f6510 2024-01-21T11:43:21.364 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:21.363+0000 7fc07dffb700 1 -- 172.21.15.134:0/3182783219 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc0780095f0 con 0x7fc0880f6510 2024-01-21T11:43:21.364 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:21.363+0000 7fc08f34f700 1 -- 172.21.15.134:0/3182783219 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc088045ad0 con 0x7fc0880f6510 2024-01-21T11:43:21.364 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:21.363+0000 7fc07dffb700 1 --2- 172.21.15.134:0/3182783219 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fc07402ff60 0x7fc074032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:43:21.364 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:21.363+0000 7fc07dffb700 1 -- 172.21.15.134:0/3182783219 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc078040fc0 con 0x7fc0880f6510 2024-01-21T11:43:21.364 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:21.363+0000 7fc08c8ea700 1 --2- 172.21.15.134:0/3182783219 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fc07402ff60 0x7fc074032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:43:21.364 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:21.364+0000 7fc08c8ea700 1 --2- 172.21.15.134:0/3182783219 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fc07402ff60 0x7fc074032420 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7fc0840096a0 tx=0x7fc084006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:43:21.368 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:21.368+0000 7fc07dffb700 1 -- 172.21.15.134:0/3182783219 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc078014420 con 0x7fc0880f6510 2024-01-21T11:43:21.573 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:21.573+0000 7fc08f34f700 1 -- 172.21.15.134:0/3182783219 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc0880fbf60 con 0x7fc0880f6510 2024-01-21T11:43:21.574 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:21.574+0000 7fc07dffb700 1 -- 172.21.15.134:0/3182783219 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc078010230 con 0x7fc0880f6510 2024-01-21T11:43:21.574 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:43:21.574 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:43:21.576 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:21.575+0000 7fc08f34f700 1 -- 172.21.15.134:0/3182783219 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fc07402ff60 msgr2=0x7fc074032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:43:21.576 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:21.575+0000 7fc08f34f700 1 --2- 172.21.15.134:0/3182783219 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fc07402ff60 0x7fc074032420 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7fc0840096a0 tx=0x7fc084006b40).stop 2024-01-21T11:43:21.576 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:21.575+0000 7fc08f34f700 1 -- 172.21.15.134:0/3182783219 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0880f6510 msgr2=0x7fc088106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:43:21.576 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:21.575+0000 7fc08f34f700 1 --2- 172.21.15.134:0/3182783219 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0880f6510 0x7fc088106730 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7fc078013510 tx=0x7fc07800b040).stop 2024-01-21T11:43:21.576 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:21.576+0000 7fc08f34f700 1 -- 172.21.15.134:0/3182783219 shutdown_connections 2024-01-21T11:43:21.576 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:21.576+0000 7fc08f34f700 1 --2- 172.21.15.134:0/3182783219 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fc07402ff60 0x7fc074032420 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:43:21.576 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:21.576+0000 7fc08f34f700 1 --2- 172.21.15.134:0/3182783219 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0880f6510 0x7fc088106730 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:43:21.576 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:21.576+0000 7fc08f34f700 1 -- 172.21.15.134:0/3182783219 >> 172.21.15.134:0/3182783219 conn(0x7fc0880f1c70 msgr2=0x7fc0880f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:43:21.576 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:21.576+0000 7fc08f34f700 1 -- 172.21.15.134:0/3182783219 shutdown_connections 2024-01-21T11:43:21.576 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:21.576+0000 7fc08f34f700 1 -- 172.21.15.134:0/3182783219 wait complete. 2024-01-21T11:43:21.577 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:43:21.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:43:21 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:43:19.813070+0000 mgr.smithi029.blhxxk (mgr.14164) 40 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:43:22.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:43:22 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:43:21.575036+0000 mon.smithi029 (mon.0) 137 : audit [DBG] from='client.? 172.21.15.134:0/3182783219' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:43:22.978 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:43:22.978 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:43:23.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:43:23 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:43:21.813494+0000 mgr.smithi029.blhxxk (mgr.14164) 41 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:43:24.175 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:24.174+0000 7f9f6bfb2700 1 -- 172.21.15.134:0/4137278203 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f640f6510 msgr2=0x7f9f640f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:43:24.175 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:24.174+0000 7f9f6bfb2700 1 --2- 172.21.15.134:0/4137278203 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f640f6510 0x7f9f640f6930 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f9f54009fb0 tx=0x7f9f54009b30).stop 2024-01-21T11:43:24.175 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:24.174+0000 7f9f6bfb2700 1 -- 172.21.15.134:0/4137278203 shutdown_connections 2024-01-21T11:43:24.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:24.174+0000 7f9f6bfb2700 1 --2- 172.21.15.134:0/4137278203 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f640f6510 0x7f9f640f6930 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:43:24.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:24.174+0000 7f9f6bfb2700 1 -- 172.21.15.134:0/4137278203 >> 172.21.15.134:0/4137278203 conn(0x7f9f640f1c70 msgr2=0x7f9f640f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:43:24.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:24.174+0000 7f9f6bfb2700 1 -- 172.21.15.134:0/4137278203 shutdown_connections 2024-01-21T11:43:24.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:24.174+0000 7f9f6bfb2700 1 -- 172.21.15.134:0/4137278203 wait complete. 2024-01-21T11:43:24.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:24.175+0000 7f9f6bfb2700 1 Processor -- start 2024-01-21T11:43:24.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:24.175+0000 7f9f6bfb2700 1 -- start start 2024-01-21T11:43:24.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:24.175+0000 7f9f6bfb2700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f640f6510 0x7f9f64106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:43:24.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:24.175+0000 7f9f6bfb2700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f64106c20 con 0x7f9f640f6510 2024-01-21T11:43:24.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:24.175+0000 7f9f69d4e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f640f6510 0x7f9f64106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:43:24.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:24.175+0000 7f9f69d4e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f640f6510 0x7f9f64106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:36350/0 (socket says 172.21.15.134:36350) 2024-01-21T11:43:24.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:24.175+0000 7f9f69d4e700 1 -- 172.21.15.134:0/2837871994 learned_addr learned my addr 172.21.15.134:0/2837871994 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:43:24.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:24.175+0000 7f9f69d4e700 1 -- 172.21.15.134:0/2837871994 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f54005040 con 0x7f9f640f6510 2024-01-21T11:43:24.177 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:24.176+0000 7f9f69d4e700 1 --2- 172.21.15.134:0/2837871994 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f640f6510 0x7f9f64106730 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f9f54013510 tx=0x7f9f5400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:43:24.178 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:24.176+0000 7f9f5a7fc700 1 -- 172.21.15.134:0/2837871994 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9f54009320 con 0x7f9f640f6510 2024-01-21T11:43:24.178 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:24.176+0000 7f9f6bfb2700 1 -- 172.21.15.134:0/2837871994 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f64106e20 con 0x7f9f640f6510 2024-01-21T11:43:24.178 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:24.176+0000 7f9f6bfb2700 1 -- 172.21.15.134:0/2837871994 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f641072b0 con 0x7f9f640f6510 2024-01-21T11:43:24.178 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:24.176+0000 7f9f5a7fc700 1 -- 172.21.15.134:0/2837871994 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9f54013f00 con 0x7f9f640f6510 2024-01-21T11:43:24.178 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:24.176+0000 7f9f5a7fc700 1 -- 172.21.15.134:0/2837871994 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9f54014d60 con 0x7f9f640f6510 2024-01-21T11:43:24.178 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:24.177+0000 7f9f5a7fc700 1 -- 172.21.15.134:0/2837871994 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9f540095f0 con 0x7f9f640f6510 2024-01-21T11:43:24.178 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:24.177+0000 7f9f6bfb2700 1 -- 172.21.15.134:0/2837871994 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f4c004fa0 con 0x7f9f640f6510 2024-01-21T11:43:24.178 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:24.177+0000 7f9f5a7fc700 1 --2- 172.21.15.134:0/2837871994 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f9f5002ff60 0x7f9f50032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:43:24.178 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:24.177+0000 7f9f5a7fc700 1 -- 172.21.15.134:0/2837871994 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9f54040f80 con 0x7f9f640f6510 2024-01-21T11:43:24.178 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:24.177+0000 7f9f6954d700 1 --2- 172.21.15.134:0/2837871994 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f9f5002ff60 0x7f9f50032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:43:24.179 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:24.178+0000 7f9f6954d700 1 --2- 172.21.15.134:0/2837871994 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f9f5002ff60 0x7f9f50032420 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f9f600096a0 tx=0x7f9f60006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:43:24.182 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:24.182+0000 7f9f5a7fc700 1 -- 172.21.15.134:0/2837871994 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9f54014420 con 0x7f9f640f6510 2024-01-21T11:43:24.385 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:24.385+0000 7f9f6bfb2700 1 -- 172.21.15.134:0/2837871994 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9f4c005670 con 0x7f9f640f6510 2024-01-21T11:43:24.386 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:24.386+0000 7f9f5a7fc700 1 -- 172.21.15.134:0/2837871994 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9f54012980 con 0x7f9f640f6510 2024-01-21T11:43:24.386 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:43:24.386 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:43:24.388 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:24.387+0000 7f9f6bfb2700 1 -- 172.21.15.134:0/2837871994 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f9f5002ff60 msgr2=0x7f9f50032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:43:24.388 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:24.387+0000 7f9f6bfb2700 1 --2- 172.21.15.134:0/2837871994 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f9f5002ff60 0x7f9f50032420 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f9f600096a0 tx=0x7f9f60006b40).stop 2024-01-21T11:43:24.388 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:24.387+0000 7f9f6bfb2700 1 -- 172.21.15.134:0/2837871994 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f640f6510 msgr2=0x7f9f64106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:43:24.388 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:24.387+0000 7f9f6bfb2700 1 --2- 172.21.15.134:0/2837871994 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f640f6510 0x7f9f64106730 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f9f54013510 tx=0x7f9f5400b040).stop 2024-01-21T11:43:24.388 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:24.387+0000 7f9f6bfb2700 1 -- 172.21.15.134:0/2837871994 shutdown_connections 2024-01-21T11:43:24.388 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:24.387+0000 7f9f6bfb2700 1 --2- 172.21.15.134:0/2837871994 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f9f5002ff60 0x7f9f50032420 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:43:24.388 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:24.387+0000 7f9f6bfb2700 1 --2- 172.21.15.134:0/2837871994 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f640f6510 0x7f9f64106730 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:43:24.388 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:24.387+0000 7f9f6bfb2700 1 -- 172.21.15.134:0/2837871994 >> 172.21.15.134:0/2837871994 conn(0x7f9f640f1c70 msgr2=0x7f9f640f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:43:24.388 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:24.387+0000 7f9f6bfb2700 1 -- 172.21.15.134:0/2837871994 shutdown_connections 2024-01-21T11:43:24.388 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:24.387+0000 7f9f6bfb2700 1 -- 172.21.15.134:0/2837871994 wait complete. 2024-01-21T11:43:24.389 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:43:24.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:43:24 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:43:24.386994+0000 mon.smithi029 (mon.0) 138 : audit [DBG] from='client.? 172.21.15.134:0/2837871994' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:43:25.777 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:43:25.777 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:43:25.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:43:25 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:43:23.813848+0000 mgr.smithi029.blhxxk (mgr.14164) 42 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:43:26.970 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:26.968+0000 7f3cf60b7700 1 -- 172.21.15.134:0/1739905830 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3cf00f6510 msgr2=0x7f3cf00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:43:26.970 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:26.968+0000 7f3cf60b7700 1 --2- 172.21.15.134:0/1739905830 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3cf00f6510 0x7f3cf00f6930 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f3ce4009fb0 tx=0x7f3ce4009b30).stop 2024-01-21T11:43:26.971 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:26.968+0000 7f3cf60b7700 1 -- 172.21.15.134:0/1739905830 shutdown_connections 2024-01-21T11:43:26.971 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:26.968+0000 7f3cf60b7700 1 --2- 172.21.15.134:0/1739905830 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3cf00f6510 0x7f3cf00f6930 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:43:26.971 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:26.968+0000 7f3cf60b7700 1 -- 172.21.15.134:0/1739905830 >> 172.21.15.134:0/1739905830 conn(0x7f3cf00f1c70 msgr2=0x7f3cf00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:43:26.971 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:26.968+0000 7f3cf60b7700 1 -- 172.21.15.134:0/1739905830 shutdown_connections 2024-01-21T11:43:26.971 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:26.968+0000 7f3cf60b7700 1 -- 172.21.15.134:0/1739905830 wait complete. 2024-01-21T11:43:26.971 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:26.969+0000 7f3cf60b7700 1 Processor -- start 2024-01-21T11:43:26.971 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:26.969+0000 7f3cf60b7700 1 -- start start 2024-01-21T11:43:26.971 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:26.969+0000 7f3cf60b7700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3cf00f6510 0x7f3cf0106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:43:26.971 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:26.969+0000 7f3cf60b7700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cf0106c20 con 0x7f3cf00f6510 2024-01-21T11:43:26.971 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:26.969+0000 7f3cef7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3cf00f6510 0x7f3cf0106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:43:26.971 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:26.969+0000 7f3cef7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3cf00f6510 0x7f3cf0106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:58220/0 (socket says 172.21.15.134:58220) 2024-01-21T11:43:26.971 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:26.969+0000 7f3cef7fe700 1 -- 172.21.15.134:0/1172499728 learned_addr learned my addr 172.21.15.134:0/1172499728 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:43:26.971 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:26.970+0000 7f3cef7fe700 1 -- 172.21.15.134:0/1172499728 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ce4005040 con 0x7f3cf00f6510 2024-01-21T11:43:26.972 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:26.970+0000 7f3cef7fe700 1 --2- 172.21.15.134:0/1172499728 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3cf00f6510 0x7f3cf0106730 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f3ce4014000 tx=0x7f3ce400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:43:26.972 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:26.970+0000 7f3cd7fff700 1 -- 172.21.15.134:0/1172499728 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ce40095f0 con 0x7f3cf00f6510 2024-01-21T11:43:26.972 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:26.970+0000 7f3cf60b7700 1 -- 172.21.15.134:0/1172499728 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cf0106e20 con 0x7f3cf00f6510 2024-01-21T11:43:26.972 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:26.970+0000 7f3cd7fff700 1 -- 172.21.15.134:0/1172499728 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3ce4007e70 con 0x7f3cf00f6510 2024-01-21T11:43:26.972 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:26.970+0000 7f3cd7fff700 1 -- 172.21.15.134:0/1172499728 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ce4014ea0 con 0x7f3cf00f6510 2024-01-21T11:43:26.972 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:26.970+0000 7f3cf60b7700 1 -- 172.21.15.134:0/1172499728 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cf01072b0 con 0x7f3cf00f6510 2024-01-21T11:43:26.973 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:26.971+0000 7f3cd7fff700 1 -- 172.21.15.134:0/1172499728 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3ce4014690 con 0x7f3cf00f6510 2024-01-21T11:43:26.973 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:26.971+0000 7f3cf60b7700 1 -- 172.21.15.134:0/1172499728 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cf0045ad0 con 0x7f3cf00f6510 2024-01-21T11:43:26.973 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:26.971+0000 7f3cd7fff700 1 --2- 172.21.15.134:0/1172499728 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f3cd802ffb0 0x7f3cd8032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:43:26.973 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:26.971+0000 7f3cd7fff700 1 -- 172.21.15.134:0/1172499728 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3ce40408f0 con 0x7f3cf00f6510 2024-01-21T11:43:26.973 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:26.971+0000 7f3ceeffd700 1 --2- 172.21.15.134:0/1172499728 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f3cd802ffb0 0x7f3cd8032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:43:26.973 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:26.972+0000 7f3ceeffd700 1 --2- 172.21.15.134:0/1172499728 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f3cd802ffb0 0x7f3cd8032470 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f3ce00096a0 tx=0x7f3ce0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:43:26.976 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:26.976+0000 7f3cd7fff700 1 -- 172.21.15.134:0/1172499728 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3ce400f300 con 0x7f3cf00f6510 2024-01-21T11:43:27.181 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:27.180+0000 7f3cf60b7700 1 -- 172.21.15.134:0/1172499728 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3cf00fbf60 con 0x7f3cf00f6510 2024-01-21T11:43:27.181 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:27.181+0000 7f3cd7fff700 1 -- 172.21.15.134:0/1172499728 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3ce400f300 con 0x7f3cf00f6510 2024-01-21T11:43:27.182 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:43:27.182 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:43:27.183 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:27.183+0000 7f3cf60b7700 1 -- 172.21.15.134:0/1172499728 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f3cd802ffb0 msgr2=0x7f3cd8032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:43:27.183 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:27.183+0000 7f3cf60b7700 1 --2- 172.21.15.134:0/1172499728 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f3cd802ffb0 0x7f3cd8032470 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f3ce00096a0 tx=0x7f3ce0006b40).stop 2024-01-21T11:43:27.183 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:27.183+0000 7f3cf60b7700 1 -- 172.21.15.134:0/1172499728 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3cf00f6510 msgr2=0x7f3cf0106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:43:27.183 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:27.183+0000 7f3cf60b7700 1 --2- 172.21.15.134:0/1172499728 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3cf00f6510 0x7f3cf0106730 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f3ce4014000 tx=0x7f3ce400b040).stop 2024-01-21T11:43:27.183 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:27.183+0000 7f3cf60b7700 1 -- 172.21.15.134:0/1172499728 shutdown_connections 2024-01-21T11:43:27.183 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:27.183+0000 7f3cf60b7700 1 --2- 172.21.15.134:0/1172499728 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f3cd802ffb0 0x7f3cd8032470 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:43:27.183 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:27.183+0000 7f3cf60b7700 1 --2- 172.21.15.134:0/1172499728 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3cf00f6510 0x7f3cf0106730 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:43:27.183 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:27.183+0000 7f3cf60b7700 1 -- 172.21.15.134:0/1172499728 >> 172.21.15.134:0/1172499728 conn(0x7f3cf00f1c70 msgr2=0x7f3cf00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:43:27.184 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:27.183+0000 7f3cf60b7700 1 -- 172.21.15.134:0/1172499728 shutdown_connections 2024-01-21T11:43:27.184 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:27.183+0000 7f3cf60b7700 1 -- 172.21.15.134:0/1172499728 wait complete. 2024-01-21T11:43:27.184 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:43:27.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:43:27 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:43:25.814248+0000 mgr.smithi029.blhxxk (mgr.14164) 43 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:43:27.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:43:27 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:43:27.182137+0000 mon.smithi029 (mon.0) 139 : audit [DBG] from='client.? 172.21.15.134:0/1172499728' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:43:28.579 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:43:28.580 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:43:29.828 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:29.826+0000 7f55adcca700 1 -- 172.21.15.134:0/1404378431 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55a80f5410 msgr2=0x7f55a80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:43:29.829 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:29.826+0000 7f55adcca700 1 --2- 172.21.15.134:0/1404378431 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55a80f5410 0x7f55a80f5830 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f5598009fb0 tx=0x7f5598009b30).stop 2024-01-21T11:43:29.829 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:29.827+0000 7f55adcca700 1 -- 172.21.15.134:0/1404378431 shutdown_connections 2024-01-21T11:43:29.829 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:29.827+0000 7f55adcca700 1 --2- 172.21.15.134:0/1404378431 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55a80f5410 0x7f55a80f5830 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:43:29.829 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:29.827+0000 7f55adcca700 1 -- 172.21.15.134:0/1404378431 >> 172.21.15.134:0/1404378431 conn(0x7f55a80f0b70 msgr2=0x7f55a80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:43:29.829 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:29.827+0000 7f55adcca700 1 -- 172.21.15.134:0/1404378431 shutdown_connections 2024-01-21T11:43:29.829 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:29.827+0000 7f55adcca700 1 -- 172.21.15.134:0/1404378431 wait complete. 2024-01-21T11:43:29.829 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:29.827+0000 7f55adcca700 1 Processor -- start 2024-01-21T11:43:29.829 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:29.827+0000 7f55adcca700 1 -- start start 2024-01-21T11:43:29.829 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:29.827+0000 7f55adcca700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55a80f5410 0x7f55a8106710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:43:29.829 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:29.827+0000 7f55adcca700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55a8106c00 con 0x7f55a80f5410 2024-01-21T11:43:29.829 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:29.828+0000 7f55a77fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55a80f5410 0x7f55a8106710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:43:29.829 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:29.828+0000 7f55a77fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55a80f5410 0x7f55a8106710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:58240/0 (socket says 172.21.15.134:58240) 2024-01-21T11:43:29.830 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:29.828+0000 7f55a77fe700 1 -- 172.21.15.134:0/2708053609 learned_addr learned my addr 172.21.15.134:0/2708053609 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:43:29.830 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:29.828+0000 7f55a77fe700 1 -- 172.21.15.134:0/2708053609 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5598005040 con 0x7f55a80f5410 2024-01-21T11:43:29.830 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:29.828+0000 7f55a77fe700 1 --2- 172.21.15.134:0/2708053609 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55a80f5410 0x7f55a8106710 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f5598013510 tx=0x7f559800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:43:29.830 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:29.828+0000 7f559ffff700 1 -- 172.21.15.134:0/2708053609 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5598009320 con 0x7f55a80f5410 2024-01-21T11:43:29.830 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:29.828+0000 7f55adcca700 1 -- 172.21.15.134:0/2708053609 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55a8106e00 con 0x7f55a80f5410 2024-01-21T11:43:29.830 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:29.828+0000 7f559ffff700 1 -- 172.21.15.134:0/2708053609 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5598013f00 con 0x7f55a80f5410 2024-01-21T11:43:29.830 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:29.828+0000 7f559ffff700 1 -- 172.21.15.134:0/2708053609 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5598014d90 con 0x7f55a80f5410 2024-01-21T11:43:29.830 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:29.828+0000 7f55adcca700 1 -- 172.21.15.134:0/2708053609 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55a8107290 con 0x7f55a80f5410 2024-01-21T11:43:29.831 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:29.829+0000 7f559ffff700 1 -- 172.21.15.134:0/2708053609 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f55980095f0 con 0x7f55a80f5410 2024-01-21T11:43:29.831 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:29.829+0000 7f55adcca700 1 -- 172.21.15.134:0/2708053609 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55a8045ad0 con 0x7f55a80f5410 2024-01-21T11:43:29.831 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:29.829+0000 7f559ffff700 1 --2- 172.21.15.134:0/2708053609 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f559402ff60 0x7f5594032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:43:29.831 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:29.829+0000 7f559ffff700 1 -- 172.21.15.134:0/2708053609 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f55980410c0 con 0x7f55a80f5410 2024-01-21T11:43:29.831 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:29.830+0000 7f55a6ffd700 1 --2- 172.21.15.134:0/2708053609 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f559402ff60 0x7f5594032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:43:29.831 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:29.830+0000 7f55a6ffd700 1 --2- 172.21.15.134:0/2708053609 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f559402ff60 0x7f5594032420 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f55900096a0 tx=0x7f5590006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:43:29.834 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:29.834+0000 7f559ffff700 1 -- 172.21.15.134:0/2708053609 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5598014420 con 0x7f55a80f5410 2024-01-21T11:43:29.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:43:29 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:43:27.814727+0000 mgr.smithi029.blhxxk (mgr.14164) 44 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:43:30.039 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:30.039+0000 7f55adcca700 1 -- 172.21.15.134:0/2708053609 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f55a80fae60 con 0x7f55a80f5410 2024-01-21T11:43:30.039 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:30.039+0000 7f559ffff700 1 -- 172.21.15.134:0/2708053609 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f559801d7f0 con 0x7f55a80f5410 2024-01-21T11:43:30.040 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:43:30.040 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:43:30.041 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:30.041+0000 7f55adcca700 1 -- 172.21.15.134:0/2708053609 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f559402ff60 msgr2=0x7f5594032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:43:30.041 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:30.041+0000 7f55adcca700 1 --2- 172.21.15.134:0/2708053609 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f559402ff60 0x7f5594032420 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f55900096a0 tx=0x7f5590006b40).stop 2024-01-21T11:43:30.041 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:30.041+0000 7f55adcca700 1 -- 172.21.15.134:0/2708053609 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55a80f5410 msgr2=0x7f55a8106710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:43:30.041 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:30.041+0000 7f55adcca700 1 --2- 172.21.15.134:0/2708053609 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55a80f5410 0x7f55a8106710 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f5598013510 tx=0x7f559800b040).stop 2024-01-21T11:43:30.041 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:30.041+0000 7f55adcca700 1 -- 172.21.15.134:0/2708053609 shutdown_connections 2024-01-21T11:43:30.041 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:30.041+0000 7f55adcca700 1 --2- 172.21.15.134:0/2708053609 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f559402ff60 0x7f5594032420 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:43:30.041 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:30.041+0000 7f55adcca700 1 --2- 172.21.15.134:0/2708053609 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55a80f5410 0x7f55a8106710 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:43:30.042 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:30.041+0000 7f55adcca700 1 -- 172.21.15.134:0/2708053609 >> 172.21.15.134:0/2708053609 conn(0x7f55a80f0b70 msgr2=0x7f55a805fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:43:30.042 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:30.041+0000 7f55adcca700 1 -- 172.21.15.134:0/2708053609 shutdown_connections 2024-01-21T11:43:30.042 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:30.041+0000 7f55adcca700 1 -- 172.21.15.134:0/2708053609 wait complete. 2024-01-21T11:43:30.042 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:43:30.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:43:30 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:43:30.040519+0000 mon.smithi029 (mon.0) 140 : audit [DBG] from='client.? 172.21.15.134:0/2708053609' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:43:31.437 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:43:31.437 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:43:31.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:43:31 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:43:29.815111+0000 mgr.smithi029.blhxxk (mgr.14164) 45 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:43:32.828 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:32.826+0000 7fc3e6731700 1 -- 172.21.15.134:0/399236483 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3e00f51a0 msgr2=0x7fc3e00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:43:32.829 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:32.826+0000 7fc3e6731700 1 --2- 172.21.15.134:0/399236483 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3e00f51a0 0x7fc3e00f55c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7fc3dc009fb0 tx=0x7fc3dc009b30).stop 2024-01-21T11:43:32.829 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:32.827+0000 7fc3e6731700 1 -- 172.21.15.134:0/399236483 shutdown_connections 2024-01-21T11:43:32.829 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:32.827+0000 7fc3e6731700 1 --2- 172.21.15.134:0/399236483 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3e00f51a0 0x7fc3e00f55c0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:43:32.829 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:32.827+0000 7fc3e6731700 1 -- 172.21.15.134:0/399236483 >> 172.21.15.134:0/399236483 conn(0x7fc3e00f09a0 msgr2=0x7fc3e00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:43:32.829 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:32.827+0000 7fc3e6731700 1 -- 172.21.15.134:0/399236483 shutdown_connections 2024-01-21T11:43:32.829 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:32.827+0000 7fc3e6731700 1 -- 172.21.15.134:0/399236483 wait complete. 2024-01-21T11:43:32.829 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:32.827+0000 7fc3e6731700 1 Processor -- start 2024-01-21T11:43:32.829 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:32.827+0000 7fc3e6731700 1 -- start start 2024-01-21T11:43:32.829 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:32.827+0000 7fc3e6731700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3e00f51a0 0x7fc3e01053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:43:32.829 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:32.827+0000 7fc3e6731700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e01058a0 con 0x7fc3e00f51a0 2024-01-21T11:43:32.829 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:32.827+0000 7fc3e572f700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3e00f51a0 0x7fc3e01053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:43:32.829 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:32.828+0000 7fc3e572f700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3e00f51a0 0x7fc3e01053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:58264/0 (socket says 172.21.15.134:58264) 2024-01-21T11:43:32.829 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:32.828+0000 7fc3e572f700 1 -- 172.21.15.134:0/1269592979 learned_addr learned my addr 172.21.15.134:0/1269592979 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:43:32.830 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:32.828+0000 7fc3e572f700 1 -- 172.21.15.134:0/1269592979 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3dc005040 con 0x7fc3e00f51a0 2024-01-21T11:43:32.830 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:32.828+0000 7fc3e572f700 1 --2- 172.21.15.134:0/1269592979 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3e00f51a0 0x7fc3e01053b0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7fc3dc013510 tx=0x7fc3dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:43:32.830 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:32.828+0000 7fc3d5ffb700 1 -- 172.21.15.134:0/1269592979 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc3dc009320 con 0x7fc3e00f51a0 2024-01-21T11:43:32.830 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:32.828+0000 7fc3e6731700 1 -- 172.21.15.134:0/1269592979 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3e0105aa0 con 0x7fc3e00f51a0 2024-01-21T11:43:32.830 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:32.828+0000 7fc3d5ffb700 1 -- 172.21.15.134:0/1269592979 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc3dc013f00 con 0x7fc3e00f51a0 2024-01-21T11:43:32.830 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:32.828+0000 7fc3d5ffb700 1 -- 172.21.15.134:0/1269592979 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc3dc014dc0 con 0x7fc3e00f51a0 2024-01-21T11:43:32.830 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:32.828+0000 7fc3e6731700 1 -- 172.21.15.134:0/1269592979 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3e0105f30 con 0x7fc3e00f51a0 2024-01-21T11:43:32.831 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:32.829+0000 7fc3d5ffb700 1 -- 172.21.15.134:0/1269592979 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc3dc0095f0 con 0x7fc3e00f51a0 2024-01-21T11:43:32.831 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:32.829+0000 7fc3e6731700 1 -- 172.21.15.134:0/1269592979 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3e0045ad0 con 0x7fc3e00f51a0 2024-01-21T11:43:32.831 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:32.829+0000 7fc3d5ffb700 1 --2- 172.21.15.134:0/1269592979 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fc3cc02ff60 0x7fc3cc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:43:32.831 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:32.829+0000 7fc3d5ffb700 1 -- 172.21.15.134:0/1269592979 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc3dc041020 con 0x7fc3e00f51a0 2024-01-21T11:43:32.831 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:32.830+0000 7fc3e4f2e700 1 --2- 172.21.15.134:0/1269592979 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fc3cc02ff60 0x7fc3cc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:43:32.831 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:32.830+0000 7fc3e4f2e700 1 --2- 172.21.15.134:0/1269592979 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fc3cc02ff60 0x7fc3cc032420 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fc3d00096a0 tx=0x7fc3d0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:43:32.834 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:32.834+0000 7fc3d5ffb700 1 -- 172.21.15.134:0/1269592979 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc3dc014420 con 0x7fc3e00f51a0 2024-01-21T11:43:33.039 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:33.039+0000 7fc3e6731700 1 -- 172.21.15.134:0/1269592979 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc3e0106d80 con 0x7fc3e00f51a0 2024-01-21T11:43:33.040 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:33.040+0000 7fc3d5ffb700 1 -- 172.21.15.134:0/1269592979 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc3dc01c920 con 0x7fc3e00f51a0 2024-01-21T11:43:33.040 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:43:33.040 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:43:33.041 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:33.041+0000 7fc3e6731700 1 -- 172.21.15.134:0/1269592979 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fc3cc02ff60 msgr2=0x7fc3cc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:43:33.042 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:33.041+0000 7fc3e6731700 1 --2- 172.21.15.134:0/1269592979 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fc3cc02ff60 0x7fc3cc032420 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fc3d00096a0 tx=0x7fc3d0006b40).stop 2024-01-21T11:43:33.042 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:33.041+0000 7fc3e6731700 1 -- 172.21.15.134:0/1269592979 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3e00f51a0 msgr2=0x7fc3e01053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:43:33.042 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:33.041+0000 7fc3e6731700 1 --2- 172.21.15.134:0/1269592979 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3e00f51a0 0x7fc3e01053b0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7fc3dc013510 tx=0x7fc3dc00b040).stop 2024-01-21T11:43:33.042 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:33.041+0000 7fc3e6731700 1 -- 172.21.15.134:0/1269592979 shutdown_connections 2024-01-21T11:43:33.042 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:33.041+0000 7fc3e6731700 1 --2- 172.21.15.134:0/1269592979 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fc3cc02ff60 0x7fc3cc032420 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:43:33.042 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:33.041+0000 7fc3e6731700 1 --2- 172.21.15.134:0/1269592979 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3e00f51a0 0x7fc3e01053b0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:43:33.042 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:33.041+0000 7fc3e6731700 1 -- 172.21.15.134:0/1269592979 >> 172.21.15.134:0/1269592979 conn(0x7fc3e00f09a0 msgr2=0x7fc3e00f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:43:33.042 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:33.041+0000 7fc3e6731700 1 -- 172.21.15.134:0/1269592979 shutdown_connections 2024-01-21T11:43:33.042 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:33.041+0000 7fc3e6731700 1 -- 172.21.15.134:0/1269592979 wait complete. 2024-01-21T11:43:33.042 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:43:33.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:43:33 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:43:31.815504+0000 mgr.smithi029.blhxxk (mgr.14164) 46 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:43:33.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:43:33 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:43:33.040847+0000 mon.smithi029 (mon.0) 141 : audit [DBG] from='client.? 172.21.15.134:0/1269592979' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:43:34.425 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:43:34.425 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:43:35.552 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:35.549+0000 7f1738b8b700 1 -- 172.21.15.134:0/2134858854 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17340f5410 msgr2=0x7f17340f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:43:35.574 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:35.549+0000 7f1738b8b700 1 --2- 172.21.15.134:0/2134858854 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17340f5410 0x7f17340f5830 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f171c009fb0 tx=0x7f171c009b30).stop 2024-01-21T11:43:35.574 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:35.549+0000 7f1738b8b700 1 -- 172.21.15.134:0/2134858854 shutdown_connections 2024-01-21T11:43:35.574 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:35.549+0000 7f1738b8b700 1 --2- 172.21.15.134:0/2134858854 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17340f5410 0x7f17340f5830 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:43:35.574 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:35.549+0000 7f1738b8b700 1 -- 172.21.15.134:0/2134858854 >> 172.21.15.134:0/2134858854 conn(0x7f17340f0b70 msgr2=0x7f17340f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:43:35.574 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:35.549+0000 7f1738b8b700 1 -- 172.21.15.134:0/2134858854 shutdown_connections 2024-01-21T11:43:35.574 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:35.549+0000 7f1738b8b700 1 -- 172.21.15.134:0/2134858854 wait complete. 2024-01-21T11:43:35.574 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:35.550+0000 7f1738b8b700 1 Processor -- start 2024-01-21T11:43:35.574 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:35.550+0000 7f1738b8b700 1 -- start start 2024-01-21T11:43:35.574 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:35.550+0000 7f1738b8b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17340f5410 0x7f17340fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:43:35.575 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:35.550+0000 7f1738b8b700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17340fb820 con 0x7f17340f5410 2024-01-21T11:43:35.575 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:35.550+0000 7f173259c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17340f5410 0x7f17340fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:43:35.575 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:35.550+0000 7f173259c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17340f5410 0x7f17340fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:40488/0 (socket says 172.21.15.134:40488) 2024-01-21T11:43:35.575 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:35.550+0000 7f173259c700 1 -- 172.21.15.134:0/3616271129 learned_addr learned my addr 172.21.15.134:0/3616271129 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:43:35.575 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:35.550+0000 7f173259c700 1 -- 172.21.15.134:0/3616271129 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f171c005040 con 0x7f17340f5410 2024-01-21T11:43:35.575 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:35.550+0000 7f173259c700 1 --2- 172.21.15.134:0/3616271129 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17340f5410 0x7f17340fd3a0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f171c013510 tx=0x7f171c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:43:35.575 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:35.551+0000 7f172affd700 1 -- 172.21.15.134:0/3616271129 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f171c009320 con 0x7f17340f5410 2024-01-21T11:43:35.575 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:35.551+0000 7f1738b8b700 1 -- 172.21.15.134:0/3616271129 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17340fb9c0 con 0x7f17340f5410 2024-01-21T11:43:35.575 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:35.551+0000 7f172affd700 1 -- 172.21.15.134:0/3616271129 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f171c013f00 con 0x7f17340f5410 2024-01-21T11:43:35.575 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:35.551+0000 7f172affd700 1 -- 172.21.15.134:0/3616271129 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f171c014d90 con 0x7f17340f5410 2024-01-21T11:43:35.575 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:35.551+0000 7f1738b8b700 1 -- 172.21.15.134:0/3616271129 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17340fbe50 con 0x7f17340f5410 2024-01-21T11:43:35.575 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:35.551+0000 7f172affd700 1 -- 172.21.15.134:0/3616271129 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f171c0095f0 con 0x7f17340f5410 2024-01-21T11:43:35.575 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:35.551+0000 7f1738b8b700 1 -- 172.21.15.134:0/3616271129 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1734059100 con 0x7f17340f5410 2024-01-21T11:43:35.575 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:35.552+0000 7f172affd700 1 --2- 172.21.15.134:0/3616271129 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f172002ff60 0x7f1720032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:43:35.575 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:35.552+0000 7f172affd700 1 -- 172.21.15.134:0/3616271129 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f171c0410a0 con 0x7f17340f5410 2024-01-21T11:43:35.576 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:35.552+0000 7f1731d9b700 1 --2- 172.21.15.134:0/3616271129 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f172002ff60 0x7f1720032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:43:35.576 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:35.552+0000 7f1731d9b700 1 --2- 172.21.15.134:0/3616271129 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f172002ff60 0x7f1720032420 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f17240096a0 tx=0x7f1724006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:43:35.576 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:35.557+0000 7f172affd700 1 -- 172.21.15.134:0/3616271129 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f171c01d7f0 con 0x7f17340f5410 2024-01-21T11:43:35.764 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:35.763+0000 7f1738b8b700 1 -- 172.21.15.134:0/3616271129 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f17340fae60 con 0x7f17340f5410 2024-01-21T11:43:35.764 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:35.764+0000 7f172affd700 1 -- 172.21.15.134:0/3616271129 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f171c010230 con 0x7f17340f5410 2024-01-21T11:43:35.764 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:43:35.765 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:43:35.766 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:35.766+0000 7f1738b8b700 1 -- 172.21.15.134:0/3616271129 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f172002ff60 msgr2=0x7f1720032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:43:35.766 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:35.766+0000 7f1738b8b700 1 --2- 172.21.15.134:0/3616271129 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f172002ff60 0x7f1720032420 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f17240096a0 tx=0x7f1724006b40).stop 2024-01-21T11:43:35.766 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:35.766+0000 7f1738b8b700 1 -- 172.21.15.134:0/3616271129 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17340f5410 msgr2=0x7f17340fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:43:35.766 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:35.766+0000 7f1738b8b700 1 --2- 172.21.15.134:0/3616271129 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17340f5410 0x7f17340fd3a0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f171c013510 tx=0x7f171c00b040).stop 2024-01-21T11:43:35.766 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:35.766+0000 7f1738b8b700 1 -- 172.21.15.134:0/3616271129 shutdown_connections 2024-01-21T11:43:35.766 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:35.766+0000 7f1738b8b700 1 --2- 172.21.15.134:0/3616271129 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f172002ff60 0x7f1720032420 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:43:35.767 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:35.766+0000 7f1738b8b700 1 --2- 172.21.15.134:0/3616271129 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17340f5410 0x7f17340fd3a0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:43:35.767 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:35.766+0000 7f1738b8b700 1 -- 172.21.15.134:0/3616271129 >> 172.21.15.134:0/3616271129 conn(0x7f17340f0b70 msgr2=0x7f173405fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:43:35.767 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:35.766+0000 7f1738b8b700 1 -- 172.21.15.134:0/3616271129 shutdown_connections 2024-01-21T11:43:35.767 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:35.766+0000 7f1738b8b700 1 -- 172.21.15.134:0/3616271129 wait complete. 2024-01-21T11:43:35.767 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:43:35.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:43:35 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:43:33.815828+0000 mgr.smithi029.blhxxk (mgr.14164) 47 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:43:36.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:43:36 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:43:35.765304+0000 mon.smithi029 (mon.0) 142 : audit [DBG] from='client.? 172.21.15.134:0/3616271129' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:43:36.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:43:36 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:43:35.816059+0000 mgr.smithi029.blhxxk (mgr.14164) 48 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:43:37.167 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:43:37.167 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:43:38.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:38.330+0000 7f3040170700 1 -- 172.21.15.134:0/453698285 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30380f5410 msgr2=0x7f30380f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:43:38.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:38.330+0000 7f3040170700 1 --2- 172.21.15.134:0/453698285 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30380f5410 0x7f30380f5830 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f3028009fb0 tx=0x7f3028009b30).stop 2024-01-21T11:43:38.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:38.330+0000 7f3040170700 1 -- 172.21.15.134:0/453698285 shutdown_connections 2024-01-21T11:43:38.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:38.330+0000 7f3040170700 1 --2- 172.21.15.134:0/453698285 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30380f5410 0x7f30380f5830 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:43:38.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:38.330+0000 7f3040170700 1 -- 172.21.15.134:0/453698285 >> 172.21.15.134:0/453698285 conn(0x7f30380f0b70 msgr2=0x7f30380f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:43:38.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:38.330+0000 7f3040170700 1 -- 172.21.15.134:0/453698285 shutdown_connections 2024-01-21T11:43:38.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:38.330+0000 7f3040170700 1 -- 172.21.15.134:0/453698285 wait complete. 2024-01-21T11:43:38.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:38.331+0000 7f3040170700 1 Processor -- start 2024-01-21T11:43:38.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:38.331+0000 7f3040170700 1 -- start start 2024-01-21T11:43:38.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:38.331+0000 7f3040170700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30380f5410 0x7f3038105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:43:38.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:38.331+0000 7f3040170700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3038105b20 con 0x7f30380f5410 2024-01-21T11:43:38.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:38.331+0000 7f303df0c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30380f5410 0x7f3038105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:43:38.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:38.331+0000 7f303df0c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30380f5410 0x7f3038105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:40516/0 (socket says 172.21.15.134:40516) 2024-01-21T11:43:38.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:38.331+0000 7f303df0c700 1 -- 172.21.15.134:0/2707345080 learned_addr learned my addr 172.21.15.134:0/2707345080 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:43:38.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:38.331+0000 7f303df0c700 1 -- 172.21.15.134:0/2707345080 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3028005040 con 0x7f30380f5410 2024-01-21T11:43:38.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:38.331+0000 7f303df0c700 1 --2- 172.21.15.134:0/2707345080 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30380f5410 0x7f3038105630 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f3028013510 tx=0x7f302800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:43:38.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:38.332+0000 7f302e7fc700 1 -- 172.21.15.134:0/2707345080 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3028009320 con 0x7f30380f5410 2024-01-21T11:43:38.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:38.332+0000 7f3040170700 1 -- 172.21.15.134:0/2707345080 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3038105d20 con 0x7f30380f5410 2024-01-21T11:43:38.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:38.332+0000 7f302e7fc700 1 -- 172.21.15.134:0/2707345080 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3028013f00 con 0x7f30380f5410 2024-01-21T11:43:38.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:38.332+0000 7f302e7fc700 1 -- 172.21.15.134:0/2707345080 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3028014d90 con 0x7f30380f5410 2024-01-21T11:43:38.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:38.332+0000 7f3040170700 1 -- 172.21.15.134:0/2707345080 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30381061b0 con 0x7f30380f5410 2024-01-21T11:43:38.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:38.332+0000 7f302e7fc700 1 -- 172.21.15.134:0/2707345080 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f30280095f0 con 0x7f30380f5410 2024-01-21T11:43:38.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:38.332+0000 7f3040170700 1 -- 172.21.15.134:0/2707345080 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3038045ad0 con 0x7f30380f5410 2024-01-21T11:43:38.335 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:38.333+0000 7f302e7fc700 1 --2- 172.21.15.134:0/2707345080 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f302402ff60 0x7f3024032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:43:38.335 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:38.333+0000 7f302e7fc700 1 -- 172.21.15.134:0/2707345080 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f30280410c0 con 0x7f30380f5410 2024-01-21T11:43:38.335 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:38.333+0000 7f303d70b700 1 --2- 172.21.15.134:0/2707345080 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f302402ff60 0x7f3024032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:43:38.335 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:38.333+0000 7f303d70b700 1 --2- 172.21.15.134:0/2707345080 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f302402ff60 0x7f3024032420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f30340096a0 tx=0x7f3034006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:43:38.337 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:38.337+0000 7f302e7fc700 1 -- 172.21.15.134:0/2707345080 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3028014420 con 0x7f30380f5410 2024-01-21T11:43:38.546 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:38.546+0000 7f3040170700 1 -- 172.21.15.134:0/2707345080 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f30380fae60 con 0x7f30380f5410 2024-01-21T11:43:38.547 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:38.547+0000 7f302e7fc700 1 -- 172.21.15.134:0/2707345080 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3028010230 con 0x7f30380f5410 2024-01-21T11:43:38.547 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:43:38.547 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:43:38.549 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:38.549+0000 7f3040170700 1 -- 172.21.15.134:0/2707345080 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f302402ff60 msgr2=0x7f3024032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:43:38.549 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:38.549+0000 7f3040170700 1 --2- 172.21.15.134:0/2707345080 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f302402ff60 0x7f3024032420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f30340096a0 tx=0x7f3034006b40).stop 2024-01-21T11:43:38.549 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:38.549+0000 7f3040170700 1 -- 172.21.15.134:0/2707345080 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30380f5410 msgr2=0x7f3038105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:43:38.549 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:38.549+0000 7f3040170700 1 --2- 172.21.15.134:0/2707345080 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30380f5410 0x7f3038105630 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f3028013510 tx=0x7f302800b040).stop 2024-01-21T11:43:38.549 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:38.549+0000 7f3040170700 1 -- 172.21.15.134:0/2707345080 shutdown_connections 2024-01-21T11:43:38.549 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:38.549+0000 7f3040170700 1 --2- 172.21.15.134:0/2707345080 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f302402ff60 0x7f3024032420 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:43:38.549 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:38.549+0000 7f3040170700 1 --2- 172.21.15.134:0/2707345080 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30380f5410 0x7f3038105630 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:43:38.549 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:38.549+0000 7f3040170700 1 -- 172.21.15.134:0/2707345080 >> 172.21.15.134:0/2707345080 conn(0x7f30380f0b70 msgr2=0x7f30380f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:43:38.550 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:38.549+0000 7f3040170700 1 -- 172.21.15.134:0/2707345080 shutdown_connections 2024-01-21T11:43:38.550 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:38.549+0000 7f3040170700 1 -- 172.21.15.134:0/2707345080 wait complete. 2024-01-21T11:43:38.550 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:43:39.168 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:43:38 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:43:37.816483+0000 mgr.smithi029.blhxxk (mgr.14164) 49 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:43:39.168 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:43:38 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:43:38.548164+0000 mon.smithi029 (mon.0) 143 : audit [DBG] from='client.? 172.21.15.134:0/2707345080' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:43:39.950 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:43:39.951 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:43:41.104 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:41.104+0000 7f89286f9700 1 -- 172.21.15.134:0/1944074166 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89200f5410 msgr2=0x7f89200f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:43:41.104 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:41.104+0000 7f89286f9700 1 --2- 172.21.15.134:0/1944074166 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89200f5410 0x7f89200f5830 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f891c009fb0 tx=0x7f891c009b30).stop 2024-01-21T11:43:41.104 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:41.104+0000 7f89286f9700 1 -- 172.21.15.134:0/1944074166 shutdown_connections 2024-01-21T11:43:41.104 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:41.104+0000 7f89286f9700 1 --2- 172.21.15.134:0/1944074166 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89200f5410 0x7f89200f5830 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:43:41.105 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:41.104+0000 7f89286f9700 1 -- 172.21.15.134:0/1944074166 >> 172.21.15.134:0/1944074166 conn(0x7f89200f0b70 msgr2=0x7f89200f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:43:41.105 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:41.104+0000 7f89286f9700 1 -- 172.21.15.134:0/1944074166 shutdown_connections 2024-01-21T11:43:41.105 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:41.104+0000 7f89286f9700 1 -- 172.21.15.134:0/1944074166 wait complete. 2024-01-21T11:43:41.106 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:41.104+0000 7f89286f9700 1 Processor -- start 2024-01-21T11:43:41.106 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:41.105+0000 7f89286f9700 1 -- start start 2024-01-21T11:43:41.106 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:41.105+0000 7f89286f9700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89200f5410 0x7f8920106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:43:41.106 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:41.105+0000 7f89286f9700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8920106c10 con 0x7f89200f5410 2024-01-21T11:43:41.106 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:41.105+0000 7f8926495700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89200f5410 0x7f8920106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:43:41.106 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:41.105+0000 7f8926495700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89200f5410 0x7f8920106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:40522/0 (socket says 172.21.15.134:40522) 2024-01-21T11:43:41.106 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:41.105+0000 7f8926495700 1 -- 172.21.15.134:0/557941963 learned_addr learned my addr 172.21.15.134:0/557941963 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:43:41.106 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:41.105+0000 7f8926495700 1 -- 172.21.15.134:0/557941963 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f891c005040 con 0x7f89200f5410 2024-01-21T11:43:41.107 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:41.105+0000 7f8926495700 1 --2- 172.21.15.134:0/557941963 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89200f5410 0x7f8920106720 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f891c013510 tx=0x7f891c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:43:41.107 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:41.105+0000 7f8916ffd700 1 -- 172.21.15.134:0/557941963 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f891c009320 con 0x7f89200f5410 2024-01-21T11:43:41.107 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:41.105+0000 7f89286f9700 1 -- 172.21.15.134:0/557941963 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8920106e10 con 0x7f89200f5410 2024-01-21T11:43:41.107 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:41.105+0000 7f8916ffd700 1 -- 172.21.15.134:0/557941963 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f891c013f00 con 0x7f89200f5410 2024-01-21T11:43:41.107 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:41.105+0000 7f8916ffd700 1 -- 172.21.15.134:0/557941963 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f891c014d90 con 0x7f89200f5410 2024-01-21T11:43:41.107 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:41.106+0000 7f89286f9700 1 -- 172.21.15.134:0/557941963 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89201072a0 con 0x7f89200f5410 2024-01-21T11:43:41.108 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:41.106+0000 7f8916ffd700 1 -- 172.21.15.134:0/557941963 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f891c0095f0 con 0x7f89200f5410 2024-01-21T11:43:41.108 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:41.106+0000 7f89286f9700 1 -- 172.21.15.134:0/557941963 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8920045ad0 con 0x7f89200f5410 2024-01-21T11:43:41.108 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:41.107+0000 7f8916ffd700 1 --2- 172.21.15.134:0/557941963 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f890c02ff60 0x7f890c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:43:41.108 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:41.107+0000 7f8916ffd700 1 -- 172.21.15.134:0/557941963 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f891c040fe0 con 0x7f89200f5410 2024-01-21T11:43:41.108 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:41.107+0000 7f8925c94700 1 --2- 172.21.15.134:0/557941963 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f890c02ff60 0x7f890c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:43:41.108 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:41.107+0000 7f8925c94700 1 --2- 172.21.15.134:0/557941963 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f890c02ff60 0x7f890c032420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f89100096a0 tx=0x7f8910006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:43:41.111 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:41.111+0000 7f8916ffd700 1 -- 172.21.15.134:0/557941963 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f891c01d7f0 con 0x7f89200f5410 2024-01-21T11:43:41.321 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:41.321+0000 7f89286f9700 1 -- 172.21.15.134:0/557941963 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8920108150 con 0x7f89200f5410 2024-01-21T11:43:41.322 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:41.322+0000 7f8916ffd700 1 -- 172.21.15.134:0/557941963 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f891c010230 con 0x7f89200f5410 2024-01-21T11:43:41.322 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:43:41.322 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:43:41.324 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:41.323+0000 7f89286f9700 1 -- 172.21.15.134:0/557941963 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f890c02ff60 msgr2=0x7f890c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:43:41.324 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:41.323+0000 7f89286f9700 1 --2- 172.21.15.134:0/557941963 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f890c02ff60 0x7f890c032420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f89100096a0 tx=0x7f8910006b40).stop 2024-01-21T11:43:41.324 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:41.323+0000 7f89286f9700 1 -- 172.21.15.134:0/557941963 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89200f5410 msgr2=0x7f8920106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:43:41.324 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:41.323+0000 7f89286f9700 1 --2- 172.21.15.134:0/557941963 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89200f5410 0x7f8920106720 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f891c013510 tx=0x7f891c00b040).stop 2024-01-21T11:43:41.324 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:41.324+0000 7f89286f9700 1 -- 172.21.15.134:0/557941963 shutdown_connections 2024-01-21T11:43:41.324 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:41.324+0000 7f89286f9700 1 --2- 172.21.15.134:0/557941963 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f890c02ff60 0x7f890c032420 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:43:41.325 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:41.324+0000 7f89286f9700 1 --2- 172.21.15.134:0/557941963 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89200f5410 0x7f8920106720 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:43:41.325 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:41.324+0000 7f89286f9700 1 -- 172.21.15.134:0/557941963 >> 172.21.15.134:0/557941963 conn(0x7f89200f0b70 msgr2=0x7f892005fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:43:41.325 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:41.324+0000 7f89286f9700 1 -- 172.21.15.134:0/557941963 shutdown_connections 2024-01-21T11:43:41.325 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:41.324+0000 7f89286f9700 1 -- 172.21.15.134:0/557941963 wait complete. 2024-01-21T11:43:41.325 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:43:41.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:43:41 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:43:39.816817+0000 mgr.smithi029.blhxxk (mgr.14164) 50 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:43:41.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:43:41 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:43:41.323248+0000 mon.smithi029 (mon.0) 144 : audit [DBG] from='client.? 172.21.15.134:0/557941963' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:43:42.696 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:43:42.697 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:43:43.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:43:43 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:43:41.817218+0000 mgr.smithi029.blhxxk (mgr.14164) 51 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:43:43.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:43.940+0000 7f3bb759e700 1 -- 172.21.15.134:0/4040719116 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3bb80f62a0 msgr2=0x7f3bb80f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:43:43.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:43.940+0000 7f3bb759e700 1 --2- 172.21.15.134:0/4040719116 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3bb80f62a0 0x7f3bb80f66c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f3bac009fb0 tx=0x7f3bac009b30).stop 2024-01-21T11:43:43.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:43.940+0000 7f3bb759e700 1 -- 172.21.15.134:0/4040719116 shutdown_connections 2024-01-21T11:43:43.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:43.940+0000 7f3bb759e700 1 --2- 172.21.15.134:0/4040719116 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3bb80f62a0 0x7f3bb80f66c0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:43:43.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:43.940+0000 7f3bb759e700 1 -- 172.21.15.134:0/4040719116 >> 172.21.15.134:0/4040719116 conn(0x7f3bb80f1aa0 msgr2=0x7f3bb80f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:43:43.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:43.941+0000 7f3bb759e700 1 -- 172.21.15.134:0/4040719116 shutdown_connections 2024-01-21T11:43:43.947 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:43.941+0000 7f3bb759e700 1 -- 172.21.15.134:0/4040719116 wait complete. 2024-01-21T11:43:43.947 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:43.941+0000 7f3bb759e700 1 Processor -- start 2024-01-21T11:43:43.947 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:43.941+0000 7f3bb759e700 1 -- start start 2024-01-21T11:43:43.947 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:43.941+0000 7f3bb759e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3bb80f62a0 0x7f3bb81064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:43:43.947 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:43.941+0000 7f3bb759e700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb81069a0 con 0x7f3bb80f62a0 2024-01-21T11:43:43.948 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:43.941+0000 7f3bb659c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3bb80f62a0 0x7f3bb81064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:43:43.948 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:43.941+0000 7f3bb659c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3bb80f62a0 0x7f3bb81064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:40544/0 (socket says 172.21.15.134:40544) 2024-01-21T11:43:43.948 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:43.941+0000 7f3bb659c700 1 -- 172.21.15.134:0/3221155182 learned_addr learned my addr 172.21.15.134:0/3221155182 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:43:43.948 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:43.942+0000 7f3bb659c700 1 -- 172.21.15.134:0/3221155182 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bac005040 con 0x7f3bb80f62a0 2024-01-21T11:43:43.948 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:43.942+0000 7f3bb659c700 1 --2- 172.21.15.134:0/3221155182 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3bb80f62a0 0x7f3bb81064b0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f3bac009830 tx=0x7f3bac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:43:43.949 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:43.942+0000 7f3ba6ffd700 1 -- 172.21.15.134:0/3221155182 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3bac009320 con 0x7f3bb80f62a0 2024-01-21T11:43:43.949 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:43.942+0000 7f3ba6ffd700 1 -- 172.21.15.134:0/3221155182 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3bac0136c0 con 0x7f3bb80f62a0 2024-01-21T11:43:43.949 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:43.942+0000 7f3bb759e700 1 -- 172.21.15.134:0/3221155182 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bb8106ba0 con 0x7f3bb80f62a0 2024-01-21T11:43:43.949 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:43.942+0000 7f3ba6ffd700 1 -- 172.21.15.134:0/3221155182 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3bac01ee90 con 0x7f3bb80f62a0 2024-01-21T11:43:43.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:43.942+0000 7f3bb759e700 1 -- 172.21.15.134:0/3221155182 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bb8107030 con 0x7f3bb80f62a0 2024-01-21T11:43:43.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:43.943+0000 7f3ba6ffd700 1 -- 172.21.15.134:0/3221155182 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3bac01e420 con 0x7f3bb80f62a0 2024-01-21T11:43:43.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:43.943+0000 7f3bb759e700 1 -- 172.21.15.134:0/3221155182 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bb8045ad0 con 0x7f3bb80f62a0 2024-01-21T11:43:43.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:43.943+0000 7f3ba6ffd700 1 --2- 172.21.15.134:0/3221155182 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f3ba002ff10 0x7f3ba00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:43:43.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:43.943+0000 7f3ba6ffd700 1 -- 172.21.15.134:0/3221155182 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3bac040f40 con 0x7f3bb80f62a0 2024-01-21T11:43:43.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:43.943+0000 7f3bb5d9b700 1 --2- 172.21.15.134:0/3221155182 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f3ba002ff10 0x7f3ba00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:43:43.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:43.944+0000 7f3bb5d9b700 1 --2- 172.21.15.134:0/3221155182 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f3ba002ff10 0x7f3ba00323d0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f3ba80096a0 tx=0x7f3ba8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:43:43.952 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:43.948+0000 7f3ba6ffd700 1 -- 172.21.15.134:0/3221155182 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3bac06d2b0 con 0x7f3bb80f62a0 2024-01-21T11:43:44.152 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:44.152+0000 7f3bb759e700 1 -- 172.21.15.134:0/3221155182 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3bb8107ee0 con 0x7f3bb80f62a0 2024-01-21T11:43:44.153 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:44.152+0000 7f3ba6ffd700 1 -- 172.21.15.134:0/3221155182 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3bac0095f0 con 0x7f3bb80f62a0 2024-01-21T11:43:44.153 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:43:44.153 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:43:44.154 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:44.154+0000 7f3bb759e700 1 -- 172.21.15.134:0/3221155182 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f3ba002ff10 msgr2=0x7f3ba00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:43:44.154 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:44.154+0000 7f3bb759e700 1 --2- 172.21.15.134:0/3221155182 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f3ba002ff10 0x7f3ba00323d0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f3ba80096a0 tx=0x7f3ba8006b40).stop 2024-01-21T11:43:44.154 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:44.154+0000 7f3bb759e700 1 -- 172.21.15.134:0/3221155182 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3bb80f62a0 msgr2=0x7f3bb81064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:43:44.155 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:44.154+0000 7f3bb759e700 1 --2- 172.21.15.134:0/3221155182 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3bb80f62a0 0x7f3bb81064b0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f3bac009830 tx=0x7f3bac00b040).stop 2024-01-21T11:43:44.155 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:44.154+0000 7f3bb759e700 1 -- 172.21.15.134:0/3221155182 shutdown_connections 2024-01-21T11:43:44.155 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:44.154+0000 7f3bb759e700 1 --2- 172.21.15.134:0/3221155182 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f3ba002ff10 0x7f3ba00323d0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:43:44.155 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:44.154+0000 7f3bb759e700 1 --2- 172.21.15.134:0/3221155182 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3bb80f62a0 0x7f3bb81064b0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:43:44.155 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:44.154+0000 7f3bb759e700 1 -- 172.21.15.134:0/3221155182 >> 172.21.15.134:0/3221155182 conn(0x7f3bb80f1aa0 msgr2=0x7f3bb80f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:43:44.155 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:44.154+0000 7f3bb759e700 1 -- 172.21.15.134:0/3221155182 shutdown_connections 2024-01-21T11:43:44.155 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:44.154+0000 7f3bb759e700 1 -- 172.21.15.134:0/3221155182 wait complete. 2024-01-21T11:43:44.155 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:43:44.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:43:44 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:43:44.153614+0000 mon.smithi029 (mon.0) 145 : audit [DBG] from='client.? 172.21.15.134:0/3221155182' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:43:45.547 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:43:45.547 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:43:45.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:43:45 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:43:43.817524+0000 mgr.smithi029.blhxxk (mgr.14164) 52 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:43:46.758 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:46.757+0000 7f4f8aa44700 1 -- 172.21.15.134:0/3616194482 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f840f62a0 msgr2=0x7f4f840f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:43:46.759 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:46.757+0000 7f4f8aa44700 1 --2- 172.21.15.134:0/3616194482 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f840f62a0 0x7f4f840f66c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f4f74009fb0 tx=0x7f4f74009b30).stop 2024-01-21T11:43:46.759 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:46.757+0000 7f4f8aa44700 1 -- 172.21.15.134:0/3616194482 shutdown_connections 2024-01-21T11:43:46.759 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:46.757+0000 7f4f8aa44700 1 --2- 172.21.15.134:0/3616194482 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f840f62a0 0x7f4f840f66c0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:43:46.759 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:46.757+0000 7f4f8aa44700 1 -- 172.21.15.134:0/3616194482 >> 172.21.15.134:0/3616194482 conn(0x7f4f840f1aa0 msgr2=0x7f4f840f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:43:46.759 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:46.757+0000 7f4f8aa44700 1 -- 172.21.15.134:0/3616194482 shutdown_connections 2024-01-21T11:43:46.759 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:46.757+0000 7f4f8aa44700 1 -- 172.21.15.134:0/3616194482 wait complete. 2024-01-21T11:43:46.759 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:46.758+0000 7f4f8aa44700 1 Processor -- start 2024-01-21T11:43:46.759 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:46.758+0000 7f4f8aa44700 1 -- start start 2024-01-21T11:43:46.759 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:46.758+0000 7f4f8aa44700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f840f62a0 0x7f4f841064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:43:46.759 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:46.758+0000 7f4f8aa44700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f841069a0 con 0x7f4f840f62a0 2024-01-21T11:43:46.759 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:46.758+0000 7f4f89a42700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f840f62a0 0x7f4f841064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:43:46.759 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:46.758+0000 7f4f89a42700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f840f62a0 0x7f4f841064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:56268/0 (socket says 172.21.15.134:56268) 2024-01-21T11:43:46.760 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:46.758+0000 7f4f89a42700 1 -- 172.21.15.134:0/3059509911 learned_addr learned my addr 172.21.15.134:0/3059509911 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:43:46.761 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:46.759+0000 7f4f89a42700 1 -- 172.21.15.134:0/3059509911 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f74005040 con 0x7f4f840f62a0 2024-01-21T11:43:46.761 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:46.759+0000 7f4f89a42700 1 --2- 172.21.15.134:0/3059509911 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f840f62a0 0x7f4f841064b0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f4f74013510 tx=0x7f4f7400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:43:46.761 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:46.759+0000 7f4f7a7fc700 1 -- 172.21.15.134:0/3059509911 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4f74009320 con 0x7f4f840f62a0 2024-01-21T11:43:46.761 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:46.759+0000 7f4f8aa44700 1 -- 172.21.15.134:0/3059509911 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f84106ba0 con 0x7f4f840f62a0 2024-01-21T11:43:46.761 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:46.759+0000 7f4f7a7fc700 1 -- 172.21.15.134:0/3059509911 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4f74013f00 con 0x7f4f840f62a0 2024-01-21T11:43:46.762 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:46.759+0000 7f4f7a7fc700 1 -- 172.21.15.134:0/3059509911 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4f74014dc0 con 0x7f4f840f62a0 2024-01-21T11:43:46.762 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:46.759+0000 7f4f8aa44700 1 -- 172.21.15.134:0/3059509911 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f84107030 con 0x7f4f840f62a0 2024-01-21T11:43:46.762 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:46.760+0000 7f4f7a7fc700 1 -- 172.21.15.134:0/3059509911 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4f740095f0 con 0x7f4f840f62a0 2024-01-21T11:43:46.762 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:46.760+0000 7f4f8aa44700 1 -- 172.21.15.134:0/3059509911 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f84045ad0 con 0x7f4f840f62a0 2024-01-21T11:43:46.762 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:46.760+0000 7f4f7a7fc700 1 --2- 172.21.15.134:0/3059509911 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f4f7002ff60 0x7f4f70032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:43:46.762 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:46.760+0000 7f4f7a7fc700 1 -- 172.21.15.134:0/3059509911 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4f74041100 con 0x7f4f840f62a0 2024-01-21T11:43:46.762 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:46.760+0000 7f4f89241700 1 --2- 172.21.15.134:0/3059509911 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f4f7002ff60 0x7f4f70032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:43:46.762 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:46.760+0000 7f4f89241700 1 --2- 172.21.15.134:0/3059509911 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f4f7002ff60 0x7f4f70032420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f4f800096a0 tx=0x7f4f80006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:43:46.764 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:46.765+0000 7f4f7a7fc700 1 -- 172.21.15.134:0/3059509911 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4f74014420 con 0x7f4f840f62a0 2024-01-21T11:43:46.970 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:46.970+0000 7f4f8aa44700 1 -- 172.21.15.134:0/3059509911 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4f84107ee0 con 0x7f4f840f62a0 2024-01-21T11:43:46.971 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:46.971+0000 7f4f7a7fc700 1 -- 172.21.15.134:0/3059509911 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4f74010230 con 0x7f4f840f62a0 2024-01-21T11:43:46.971 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:43:46.971 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:43:46.973 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:46.972+0000 7f4f8aa44700 1 -- 172.21.15.134:0/3059509911 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f4f7002ff60 msgr2=0x7f4f70032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:43:46.973 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:46.972+0000 7f4f8aa44700 1 --2- 172.21.15.134:0/3059509911 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f4f7002ff60 0x7f4f70032420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f4f800096a0 tx=0x7f4f80006b40).stop 2024-01-21T11:43:46.973 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:46.972+0000 7f4f8aa44700 1 -- 172.21.15.134:0/3059509911 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f840f62a0 msgr2=0x7f4f841064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:43:46.973 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:46.972+0000 7f4f8aa44700 1 --2- 172.21.15.134:0/3059509911 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f840f62a0 0x7f4f841064b0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f4f74013510 tx=0x7f4f7400b040).stop 2024-01-21T11:43:46.973 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:46.972+0000 7f4f8aa44700 1 -- 172.21.15.134:0/3059509911 shutdown_connections 2024-01-21T11:43:46.973 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:46.972+0000 7f4f8aa44700 1 --2- 172.21.15.134:0/3059509911 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f4f7002ff60 0x7f4f70032420 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:43:46.973 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:46.972+0000 7f4f8aa44700 1 --2- 172.21.15.134:0/3059509911 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f840f62a0 0x7f4f841064b0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:43:46.973 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:46.972+0000 7f4f8aa44700 1 -- 172.21.15.134:0/3059509911 >> 172.21.15.134:0/3059509911 conn(0x7f4f840f1aa0 msgr2=0x7f4f840f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:43:46.973 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:46.972+0000 7f4f8aa44700 1 -- 172.21.15.134:0/3059509911 shutdown_connections 2024-01-21T11:43:46.973 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:46.972+0000 7f4f8aa44700 1 -- 172.21.15.134:0/3059509911 wait complete. 2024-01-21T11:43:46.973 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:43:47.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:43:47 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:43:45.817933+0000 mgr.smithi029.blhxxk (mgr.14164) 53 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:43:47.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:43:47 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:43:46.971886+0000 mon.smithi029 (mon.0) 146 : audit [DBG] from='client.? 172.21.15.134:0/3059509911' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:43:48.361 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:43:48.361 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:43:49.575 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:49.575+0000 7fcdda743700 1 -- 172.21.15.134:0/341309111 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcdd40f5410 msgr2=0x7fcdd40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:43:49.576 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:49.575+0000 7fcdda743700 1 --2- 172.21.15.134:0/341309111 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcdd40f5410 0x7fcdd40f5830 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7fcdc4009fb0 tx=0x7fcdc4009b30).stop 2024-01-21T11:43:49.576 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:49.575+0000 7fcdda743700 1 -- 172.21.15.134:0/341309111 shutdown_connections 2024-01-21T11:43:49.576 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:49.575+0000 7fcdda743700 1 --2- 172.21.15.134:0/341309111 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcdd40f5410 0x7fcdd40f5830 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:43:49.576 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:49.575+0000 7fcdda743700 1 -- 172.21.15.134:0/341309111 >> 172.21.15.134:0/341309111 conn(0x7fcdd40f0b70 msgr2=0x7fcdd40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:43:49.576 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:49.575+0000 7fcdda743700 1 -- 172.21.15.134:0/341309111 shutdown_connections 2024-01-21T11:43:49.576 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:49.575+0000 7fcdda743700 1 -- 172.21.15.134:0/341309111 wait complete. 2024-01-21T11:43:49.576 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:49.575+0000 7fcdda743700 1 Processor -- start 2024-01-21T11:43:49.576 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:49.576+0000 7fcdda743700 1 -- start start 2024-01-21T11:43:49.576 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:49.576+0000 7fcdda743700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcdd40f5410 0x7fcdd41093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:43:49.576 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:49.576+0000 7fcdda743700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd4060280 con 0x7fcdd40f5410 2024-01-21T11:43:49.578 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:49.576+0000 7fcdd3fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcdd40f5410 0x7fcdd41093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:43:49.578 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:49.576+0000 7fcdd3fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcdd40f5410 0x7fcdd41093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:56288/0 (socket says 172.21.15.134:56288) 2024-01-21T11:43:49.578 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:49.576+0000 7fcdd3fff700 1 -- 172.21.15.134:0/226643166 learned_addr learned my addr 172.21.15.134:0/226643166 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:43:49.578 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:49.576+0000 7fcdd3fff700 1 -- 172.21.15.134:0/226643166 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdc4005040 con 0x7fcdd40f5410 2024-01-21T11:43:49.579 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:49.576+0000 7fcdd3fff700 1 --2- 172.21.15.134:0/226643166 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcdd40f5410 0x7fcdd41093f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fcdc4013510 tx=0x7fcdc400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:43:49.579 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:49.577+0000 7fcdd0ff9700 1 -- 172.21.15.134:0/226643166 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcdc4009320 con 0x7fcdd40f5410 2024-01-21T11:43:49.579 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:49.577+0000 7fcdda743700 1 -- 172.21.15.134:0/226643166 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdd4060480 con 0x7fcdd40f5410 2024-01-21T11:43:49.579 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:49.577+0000 7fcdd0ff9700 1 -- 172.21.15.134:0/226643166 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcdc4013f00 con 0x7fcdd40f5410 2024-01-21T11:43:49.579 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:49.577+0000 7fcdd0ff9700 1 -- 172.21.15.134:0/226643166 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcdc4014d90 con 0x7fcdd40f5410 2024-01-21T11:43:49.579 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:49.577+0000 7fcdda743700 1 -- 172.21.15.134:0/226643166 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdd4109b20 con 0x7fcdd40f5410 2024-01-21T11:43:49.579 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:49.577+0000 7fcdd0ff9700 1 -- 172.21.15.134:0/226643166 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcdc40095f0 con 0x7fcdd40f5410 2024-01-21T11:43:49.579 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:49.577+0000 7fcdda743700 1 -- 172.21.15.134:0/226643166 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdd4045ad0 con 0x7fcdd40f5410 2024-01-21T11:43:49.579 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:49.578+0000 7fcdd0ff9700 1 --2- 172.21.15.134:0/226643166 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fcdbc02ff60 0x7fcdbc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:43:49.579 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:49.578+0000 7fcdd0ff9700 1 -- 172.21.15.134:0/226643166 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcdc40410c0 con 0x7fcdd40f5410 2024-01-21T11:43:49.579 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:49.578+0000 7fcdd37fe700 1 --2- 172.21.15.134:0/226643166 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fcdbc02ff60 0x7fcdbc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:43:49.579 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:49.578+0000 7fcdd37fe700 1 --2- 172.21.15.134:0/226643166 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fcdbc02ff60 0x7fcdbc032420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fcdc80096a0 tx=0x7fcdc8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:43:49.582 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:49.582+0000 7fcdd0ff9700 1 -- 172.21.15.134:0/226643166 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcdc4014420 con 0x7fcdd40f5410 2024-01-21T11:43:49.790 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:49.789+0000 7fcdda743700 1 -- 172.21.15.134:0/226643166 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcdd410a340 con 0x7fcdd40f5410 2024-01-21T11:43:49.790 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:49.790+0000 7fcdd0ff9700 1 -- 172.21.15.134:0/226643166 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcdc4010230 con 0x7fcdd40f5410 2024-01-21T11:43:49.790 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:43:49.791 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:43:49.792 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:49.792+0000 7fcdda743700 1 -- 172.21.15.134:0/226643166 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fcdbc02ff60 msgr2=0x7fcdbc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:43:49.792 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:49.792+0000 7fcdda743700 1 --2- 172.21.15.134:0/226643166 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fcdbc02ff60 0x7fcdbc032420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fcdc80096a0 tx=0x7fcdc8006b40).stop 2024-01-21T11:43:49.792 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:49.792+0000 7fcdda743700 1 -- 172.21.15.134:0/226643166 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcdd40f5410 msgr2=0x7fcdd41093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:43:49.792 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:49.792+0000 7fcdda743700 1 --2- 172.21.15.134:0/226643166 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcdd40f5410 0x7fcdd41093f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fcdc4013510 tx=0x7fcdc400b040).stop 2024-01-21T11:43:49.792 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:49.792+0000 7fcdda743700 1 -- 172.21.15.134:0/226643166 shutdown_connections 2024-01-21T11:43:49.792 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:49.792+0000 7fcdda743700 1 --2- 172.21.15.134:0/226643166 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fcdbc02ff60 0x7fcdbc032420 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:43:49.792 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:49.792+0000 7fcdda743700 1 --2- 172.21.15.134:0/226643166 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcdd40f5410 0x7fcdd41093f0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:43:49.793 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:49.792+0000 7fcdda743700 1 -- 172.21.15.134:0/226643166 >> 172.21.15.134:0/226643166 conn(0x7fcdd40f0b70 msgr2=0x7fcdd40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:43:49.793 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:49.792+0000 7fcdda743700 1 -- 172.21.15.134:0/226643166 shutdown_connections 2024-01-21T11:43:49.793 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:49.792+0000 7fcdda743700 1 -- 172.21.15.134:0/226643166 wait complete. 2024-01-21T11:43:49.793 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:43:49.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:43:49 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:43:47.818247+0000 mgr.smithi029.blhxxk (mgr.14164) 54 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:43:50.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:43:50 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:43:49.791423+0000 mon.smithi029 (mon.0) 147 : audit [DBG] from='client.? 172.21.15.134:0/226643166' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:43:51.184 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:43:51.184 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:43:51.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:43:51 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:43:49.818544+0000 mgr.smithi029.blhxxk (mgr.14164) 55 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:43:52.410 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:52.408+0000 7f694b59e700 1 -- 172.21.15.134:0/2586586432 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f694c0f51a0 msgr2=0x7f694c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:43:52.410 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:52.408+0000 7f694b59e700 1 --2- 172.21.15.134:0/2586586432 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f694c0f51a0 0x7f694c0f55c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f6940009fb0 tx=0x7f6940009b30).stop 2024-01-21T11:43:52.410 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:52.408+0000 7f694b59e700 1 -- 172.21.15.134:0/2586586432 shutdown_connections 2024-01-21T11:43:52.410 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:52.408+0000 7f694b59e700 1 --2- 172.21.15.134:0/2586586432 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f694c0f51a0 0x7f694c0f55c0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:43:52.410 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:52.408+0000 7f694b59e700 1 -- 172.21.15.134:0/2586586432 >> 172.21.15.134:0/2586586432 conn(0x7f694c0f09a0 msgr2=0x7f694c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:43:52.410 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:52.408+0000 7f694b59e700 1 -- 172.21.15.134:0/2586586432 shutdown_connections 2024-01-21T11:43:52.410 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:52.408+0000 7f694b59e700 1 -- 172.21.15.134:0/2586586432 wait complete. 2024-01-21T11:43:52.410 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:52.409+0000 7f694b59e700 1 Processor -- start 2024-01-21T11:43:52.411 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:52.409+0000 7f694b59e700 1 -- start start 2024-01-21T11:43:52.411 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:52.409+0000 7f694b59e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f694c0f51a0 0x7f694c1064a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:43:52.411 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:52.409+0000 7f694b59e700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f694c106990 con 0x7f694c0f51a0 2024-01-21T11:43:52.411 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:52.409+0000 7f694a59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f694c0f51a0 0x7f694c1064a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:43:52.411 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:52.409+0000 7f694a59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f694c0f51a0 0x7f694c1064a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:56292/0 (socket says 172.21.15.134:56292) 2024-01-21T11:43:52.411 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:52.409+0000 7f694a59c700 1 -- 172.21.15.134:0/1531007960 learned_addr learned my addr 172.21.15.134:0/1531007960 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:43:52.411 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:52.409+0000 7f694a59c700 1 -- 172.21.15.134:0/1531007960 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6940005040 con 0x7f694c0f51a0 2024-01-21T11:43:52.411 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:52.410+0000 7f694a59c700 1 --2- 172.21.15.134:0/1531007960 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f694c0f51a0 0x7f694c1064a0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f6940013510 tx=0x7f694000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:43:52.411 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:52.410+0000 7f693affd700 1 -- 172.21.15.134:0/1531007960 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6940009320 con 0x7f694c0f51a0 2024-01-21T11:43:52.411 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:52.410+0000 7f694b59e700 1 -- 172.21.15.134:0/1531007960 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f694c106b90 con 0x7f694c0f51a0 2024-01-21T11:43:52.411 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:52.410+0000 7f693affd700 1 -- 172.21.15.134:0/1531007960 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6940013f00 con 0x7f694c0f51a0 2024-01-21T11:43:52.411 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:52.410+0000 7f693affd700 1 -- 172.21.15.134:0/1531007960 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6940014dc0 con 0x7f694c0f51a0 2024-01-21T11:43:52.411 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:52.410+0000 7f694b59e700 1 -- 172.21.15.134:0/1531007960 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f694c107020 con 0x7f694c0f51a0 2024-01-21T11:43:52.412 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:52.411+0000 7f693affd700 1 -- 172.21.15.134:0/1531007960 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f69400095f0 con 0x7f694c0f51a0 2024-01-21T11:43:52.412 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:52.411+0000 7f694b59e700 1 -- 172.21.15.134:0/1531007960 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f694c045ad0 con 0x7f694c0f51a0 2024-01-21T11:43:52.413 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:52.411+0000 7f693affd700 1 --2- 172.21.15.134:0/1531007960 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f693402ff60 0x7f6934032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:43:52.413 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:52.411+0000 7f693affd700 1 -- 172.21.15.134:0/1531007960 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6940041000 con 0x7f694c0f51a0 2024-01-21T11:43:52.413 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:52.411+0000 7f6949d9b700 1 --2- 172.21.15.134:0/1531007960 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f693402ff60 0x7f6934032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:43:52.413 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:52.411+0000 7f6949d9b700 1 --2- 172.21.15.134:0/1531007960 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f693402ff60 0x7f6934032420 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f693c0096a0 tx=0x7f693c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:43:52.416 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:52.416+0000 7f693affd700 1 -- 172.21.15.134:0/1531007960 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6940014420 con 0x7f694c0f51a0 2024-01-21T11:43:52.619 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:52.619+0000 7f694b59e700 1 -- 172.21.15.134:0/1531007960 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f694c107ed0 con 0x7f694c0f51a0 2024-01-21T11:43:52.620 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:52.620+0000 7f693affd700 1 -- 172.21.15.134:0/1531007960 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6940010230 con 0x7f694c0f51a0 2024-01-21T11:43:52.620 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:43:52.620 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:43:52.621 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:52.621+0000 7f694b59e700 1 -- 172.21.15.134:0/1531007960 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f693402ff60 msgr2=0x7f6934032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:43:52.621 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:52.621+0000 7f694b59e700 1 --2- 172.21.15.134:0/1531007960 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f693402ff60 0x7f6934032420 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f693c0096a0 tx=0x7f693c006b40).stop 2024-01-21T11:43:52.622 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:52.621+0000 7f694b59e700 1 -- 172.21.15.134:0/1531007960 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f694c0f51a0 msgr2=0x7f694c1064a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:43:52.622 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:52.621+0000 7f694b59e700 1 --2- 172.21.15.134:0/1531007960 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f694c0f51a0 0x7f694c1064a0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f6940013510 tx=0x7f694000b040).stop 2024-01-21T11:43:52.622 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:52.621+0000 7f694b59e700 1 -- 172.21.15.134:0/1531007960 shutdown_connections 2024-01-21T11:43:52.622 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:52.621+0000 7f694b59e700 1 --2- 172.21.15.134:0/1531007960 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f693402ff60 0x7f6934032420 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:43:52.622 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:52.621+0000 7f694b59e700 1 --2- 172.21.15.134:0/1531007960 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f694c0f51a0 0x7f694c1064a0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:43:52.622 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:52.621+0000 7f694b59e700 1 -- 172.21.15.134:0/1531007960 >> 172.21.15.134:0/1531007960 conn(0x7f694c0f09a0 msgr2=0x7f694c05fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:43:52.622 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:52.621+0000 7f694b59e700 1 -- 172.21.15.134:0/1531007960 shutdown_connections 2024-01-21T11:43:52.622 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:52.621+0000 7f694b59e700 1 -- 172.21.15.134:0/1531007960 wait complete. 2024-01-21T11:43:52.622 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:43:53.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:43:53 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:43:51.818904+0000 mgr.smithi029.blhxxk (mgr.14164) 56 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:43:53.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:43:53 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:43:52.620880+0000 mon.smithi029 (mon.0) 148 : audit [DBG] from='client.? 172.21.15.134:0/1531007960' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:43:54.029 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:43:54.030 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:43:55.224 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:55.222+0000 7fe8ac13a700 1 -- 172.21.15.134:0/1927866405 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8a40f5410 msgr2=0x7fe8a40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:43:55.224 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:55.222+0000 7fe8ac13a700 1 --2- 172.21.15.134:0/1927866405 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8a40f5410 0x7fe8a40f5830 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7fe894009fb0 tx=0x7fe894009b30).stop 2024-01-21T11:43:55.225 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:55.222+0000 7fe8ac13a700 1 -- 172.21.15.134:0/1927866405 shutdown_connections 2024-01-21T11:43:55.225 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:55.222+0000 7fe8ac13a700 1 --2- 172.21.15.134:0/1927866405 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8a40f5410 0x7fe8a40f5830 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:43:55.225 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:55.222+0000 7fe8ac13a700 1 -- 172.21.15.134:0/1927866405 >> 172.21.15.134:0/1927866405 conn(0x7fe8a40f0b70 msgr2=0x7fe8a40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:43:55.225 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:55.222+0000 7fe8ac13a700 1 -- 172.21.15.134:0/1927866405 shutdown_connections 2024-01-21T11:43:55.225 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:55.222+0000 7fe8ac13a700 1 -- 172.21.15.134:0/1927866405 wait complete. 2024-01-21T11:43:55.225 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:55.223+0000 7fe8ac13a700 1 Processor -- start 2024-01-21T11:43:55.225 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:55.223+0000 7fe8ac13a700 1 -- start start 2024-01-21T11:43:55.225 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:55.223+0000 7fe8ac13a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8a40f5410 0x7fe8a4109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:43:55.225 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:55.223+0000 7fe8ac13a700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a4060220 con 0x7fe8a40f5410 2024-01-21T11:43:55.225 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:55.223+0000 7fe8a9ed6700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8a40f5410 0x7fe8a4109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:43:55.226 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:55.223+0000 7fe8a9ed6700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8a40f5410 0x7fe8a4109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:32896/0 (socket says 172.21.15.134:32896) 2024-01-21T11:43:55.226 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:55.223+0000 7fe8a9ed6700 1 -- 172.21.15.134:0/3313464617 learned_addr learned my addr 172.21.15.134:0/3313464617 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:43:55.226 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:55.224+0000 7fe8a9ed6700 1 -- 172.21.15.134:0/3313464617 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe894005040 con 0x7fe8a40f5410 2024-01-21T11:43:55.226 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:55.224+0000 7fe8a9ed6700 1 --2- 172.21.15.134:0/3313464617 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8a40f5410 0x7fe8a4109450 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7fe894013510 tx=0x7fe89400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:43:55.226 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:55.224+0000 7fe89a7fc700 1 -- 172.21.15.134:0/3313464617 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe894009320 con 0x7fe8a40f5410 2024-01-21T11:43:55.226 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:55.224+0000 7fe8ac13a700 1 -- 172.21.15.134:0/3313464617 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8a4060420 con 0x7fe8a40f5410 2024-01-21T11:43:55.226 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:55.224+0000 7fe89a7fc700 1 -- 172.21.15.134:0/3313464617 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe894013f00 con 0x7fe8a40f5410 2024-01-21T11:43:55.226 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:55.224+0000 7fe89a7fc700 1 -- 172.21.15.134:0/3313464617 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe894014d90 con 0x7fe8a40f5410 2024-01-21T11:43:55.226 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:55.224+0000 7fe8ac13a700 1 -- 172.21.15.134:0/3313464617 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8a4109b00 con 0x7fe8a40f5410 2024-01-21T11:43:55.227 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:55.225+0000 7fe89a7fc700 1 -- 172.21.15.134:0/3313464617 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe8940095f0 con 0x7fe8a40f5410 2024-01-21T11:43:55.227 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:55.225+0000 7fe8ac13a700 1 -- 172.21.15.134:0/3313464617 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8a4059100 con 0x7fe8a40f5410 2024-01-21T11:43:55.228 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:55.225+0000 7fe89a7fc700 1 --2- 172.21.15.134:0/3313464617 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fe89002ff60 0x7fe890032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:43:55.228 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:55.225+0000 7fe89a7fc700 1 -- 172.21.15.134:0/3313464617 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe894040fe0 con 0x7fe8a40f5410 2024-01-21T11:43:55.228 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:55.225+0000 7fe8a96d5700 1 --2- 172.21.15.134:0/3313464617 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fe89002ff60 0x7fe890032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:43:55.228 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:55.226+0000 7fe8a96d5700 1 --2- 172.21.15.134:0/3313464617 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fe89002ff60 0x7fe890032420 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7fe8a00096a0 tx=0x7fe8a0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:43:55.229 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:55.229+0000 7fe89a7fc700 1 -- 172.21.15.134:0/3313464617 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe89401d7f0 con 0x7fe8a40f5410 2024-01-21T11:43:55.433 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:55.433+0000 7fe8ac13a700 1 -- 172.21.15.134:0/3313464617 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe8a4045ad0 con 0x7fe8a40f5410 2024-01-21T11:43:55.434 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:55.434+0000 7fe89a7fc700 1 -- 172.21.15.134:0/3313464617 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe894010230 con 0x7fe8a40f5410 2024-01-21T11:43:55.434 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:43:55.434 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:43:55.436 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:55.435+0000 7fe8ac13a700 1 -- 172.21.15.134:0/3313464617 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fe89002ff60 msgr2=0x7fe890032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:43:55.436 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:55.435+0000 7fe8ac13a700 1 --2- 172.21.15.134:0/3313464617 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fe89002ff60 0x7fe890032420 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7fe8a00096a0 tx=0x7fe8a0006b40).stop 2024-01-21T11:43:55.436 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:55.435+0000 7fe8ac13a700 1 -- 172.21.15.134:0/3313464617 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8a40f5410 msgr2=0x7fe8a4109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:43:55.436 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:55.435+0000 7fe8ac13a700 1 --2- 172.21.15.134:0/3313464617 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8a40f5410 0x7fe8a4109450 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7fe894013510 tx=0x7fe89400b040).stop 2024-01-21T11:43:55.436 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:55.435+0000 7fe8ac13a700 1 -- 172.21.15.134:0/3313464617 shutdown_connections 2024-01-21T11:43:55.436 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:55.435+0000 7fe8ac13a700 1 --2- 172.21.15.134:0/3313464617 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fe89002ff60 0x7fe890032420 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:43:55.436 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:55.435+0000 7fe8ac13a700 1 --2- 172.21.15.134:0/3313464617 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8a40f5410 0x7fe8a4109450 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:43:55.436 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:55.435+0000 7fe8ac13a700 1 -- 172.21.15.134:0/3313464617 >> 172.21.15.134:0/3313464617 conn(0x7fe8a40f0b70 msgr2=0x7fe8a40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:43:55.436 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:55.436+0000 7fe8ac13a700 1 -- 172.21.15.134:0/3313464617 shutdown_connections 2024-01-21T11:43:55.436 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:55.436+0000 7fe8ac13a700 1 -- 172.21.15.134:0/3313464617 wait complete. 2024-01-21T11:43:55.437 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:43:55.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:43:55 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:43:53.819182+0000 mgr.smithi029.blhxxk (mgr.14164) 57 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:43:55.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:43:55 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:43:55.435144+0000 mon.smithi029 (mon.0) 149 : audit [DBG] from='client.? 172.21.15.134:0/3313464617' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:43:56.826 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:43:56.827 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:43:57.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:43:57 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:43:55.819468+0000 mgr.smithi029.blhxxk (mgr.14164) 58 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:43:58.039 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:58.040+0000 7f20862aa700 1 -- 172.21.15.134:0/1072312842 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2080060280 msgr2=0x7f20800f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:43:58.039 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:58.040+0000 7f20862aa700 1 --2- 172.21.15.134:0/1072312842 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2080060280 0x7f20800f90f0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f2068009fb0 tx=0x7f2068009b30).stop 2024-01-21T11:43:58.039 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:58.040+0000 7f20862aa700 1 -- 172.21.15.134:0/1072312842 shutdown_connections 2024-01-21T11:43:58.039 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:58.040+0000 7f20862aa700 1 --2- 172.21.15.134:0/1072312842 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2080060280 0x7f20800f90f0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:43:58.039 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:58.040+0000 7f20862aa700 1 -- 172.21.15.134:0/1072312842 >> 172.21.15.134:0/1072312842 conn(0x7f20800f0b70 msgr2=0x7f20800f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:43:58.040 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:58.040+0000 7f20862aa700 1 -- 172.21.15.134:0/1072312842 shutdown_connections 2024-01-21T11:43:58.040 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:58.040+0000 7f20862aa700 1 -- 172.21.15.134:0/1072312842 wait complete. 2024-01-21T11:43:58.041 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:58.040+0000 7f20862aa700 1 Processor -- start 2024-01-21T11:43:58.041 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:58.041+0000 7f20862aa700 1 -- start start 2024-01-21T11:43:58.041 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:58.041+0000 7f20862aa700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2080060280 0x7f20801066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:43:58.041 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:58.041+0000 7f20862aa700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2080106bd0 con 0x7f2080060280 2024-01-21T11:43:58.041 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:58.041+0000 7f207ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2080060280 0x7f20801066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:43:58.041 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:58.041+0000 7f207ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2080060280 0x7f20801066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:32912/0 (socket says 172.21.15.134:32912) 2024-01-21T11:43:58.041 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:58.041+0000 7f207ffff700 1 -- 172.21.15.134:0/4165366284 learned_addr learned my addr 172.21.15.134:0/4165366284 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:43:58.041 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:58.041+0000 7f207ffff700 1 -- 172.21.15.134:0/4165366284 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2068005040 con 0x7f2080060280 2024-01-21T11:43:58.041 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:58.042+0000 7f207ffff700 1 --2- 172.21.15.134:0/4165366284 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2080060280 0x7f20801066e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f2068013510 tx=0x7f206800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:43:58.041 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:58.042+0000 7f207cff9700 1 -- 172.21.15.134:0/4165366284 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2068009320 con 0x7f2080060280 2024-01-21T11:43:58.042 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:58.042+0000 7f20862aa700 1 -- 172.21.15.134:0/4165366284 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2080106dd0 con 0x7f2080060280 2024-01-21T11:43:58.042 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:58.042+0000 7f207cff9700 1 -- 172.21.15.134:0/4165366284 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2068013f00 con 0x7f2080060280 2024-01-21T11:43:58.042 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:58.042+0000 7f207cff9700 1 -- 172.21.15.134:0/4165366284 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2068014dc0 con 0x7f2080060280 2024-01-21T11:43:58.042 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:58.042+0000 7f20862aa700 1 -- 172.21.15.134:0/4165366284 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2080107260 con 0x7f2080060280 2024-01-21T11:43:58.042 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:58.043+0000 7f207cff9700 1 -- 172.21.15.134:0/4165366284 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f20680095f0 con 0x7f2080060280 2024-01-21T11:43:58.043 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:58.043+0000 7f20862aa700 1 -- 172.21.15.134:0/4165366284 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2080045ad0 con 0x7f2080060280 2024-01-21T11:43:58.043 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:58.043+0000 7f207cff9700 1 --2- 172.21.15.134:0/4165366284 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f206c02ff60 0x7f206c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:43:58.043 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:58.043+0000 7f207cff9700 1 -- 172.21.15.134:0/4165366284 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f20680410e0 con 0x7f2080060280 2024-01-21T11:43:58.043 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:58.043+0000 7f207f7fe700 1 --2- 172.21.15.134:0/4165366284 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f206c02ff60 0x7f206c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:43:58.043 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:58.043+0000 7f207f7fe700 1 --2- 172.21.15.134:0/4165366284 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f206c02ff60 0x7f206c032420 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f20700096a0 tx=0x7f2070006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:43:58.047 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:58.048+0000 7f207cff9700 1 -- 172.21.15.134:0/4165366284 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2068014420 con 0x7f2080060280 2024-01-21T11:43:58.252 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:58.252+0000 7f20862aa700 1 -- 172.21.15.134:0/4165366284 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2080108110 con 0x7f2080060280 2024-01-21T11:43:58.252 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:58.253+0000 7f207cff9700 1 -- 172.21.15.134:0/4165366284 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f206801c920 con 0x7f2080060280 2024-01-21T11:43:58.252 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:43:58.253 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:43:58.254 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:58.254+0000 7f20862aa700 1 -- 172.21.15.134:0/4165366284 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f206c02ff60 msgr2=0x7f206c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:43:58.254 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:58.254+0000 7f20862aa700 1 --2- 172.21.15.134:0/4165366284 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f206c02ff60 0x7f206c032420 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f20700096a0 tx=0x7f2070006b40).stop 2024-01-21T11:43:58.254 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:58.254+0000 7f20862aa700 1 -- 172.21.15.134:0/4165366284 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2080060280 msgr2=0x7f20801066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:43:58.254 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:58.254+0000 7f20862aa700 1 --2- 172.21.15.134:0/4165366284 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2080060280 0x7f20801066e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f2068013510 tx=0x7f206800b040).stop 2024-01-21T11:43:58.254 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:58.255+0000 7f20862aa700 1 -- 172.21.15.134:0/4165366284 shutdown_connections 2024-01-21T11:43:58.254 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:58.255+0000 7f20862aa700 1 --2- 172.21.15.134:0/4165366284 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f206c02ff60 0x7f206c032420 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:43:58.254 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:58.255+0000 7f20862aa700 1 --2- 172.21.15.134:0/4165366284 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2080060280 0x7f20801066e0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:43:58.254 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:58.255+0000 7f20862aa700 1 -- 172.21.15.134:0/4165366284 >> 172.21.15.134:0/4165366284 conn(0x7f20800f0b70 msgr2=0x7f20800f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:43:58.255 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:58.255+0000 7f20862aa700 1 -- 172.21.15.134:0/4165366284 shutdown_connections 2024-01-21T11:43:58.255 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:43:58.255+0000 7f20862aa700 1 -- 172.21.15.134:0/4165366284 wait complete. 2024-01-21T11:43:58.255 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:43:58.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:43:58 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:43:58.252813+0000 mon.smithi029 (mon.0) 150 : audit [DBG] from='client.? 172.21.15.134:0/4165366284' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:43:59.636 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:43:59.637 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:43:59.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:43:59 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:43:57.819781+0000 mgr.smithi029.blhxxk (mgr.14164) 59 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:44:00.831 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:00.830+0000 7f11f6231700 1 -- 172.21.15.134:0/3829463139 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11f00f5410 msgr2=0x7f11f00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:44:00.831 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:00.830+0000 7f11f6231700 1 --2- 172.21.15.134:0/3829463139 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11f00f5410 0x7f11f00f5830 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f11e4009fb0 tx=0x7f11e4009b30).stop 2024-01-21T11:44:00.831 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:00.830+0000 7f11f6231700 1 -- 172.21.15.134:0/3829463139 shutdown_connections 2024-01-21T11:44:00.831 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:00.830+0000 7f11f6231700 1 --2- 172.21.15.134:0/3829463139 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11f00f5410 0x7f11f00f5830 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:44:00.831 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:00.830+0000 7f11f6231700 1 -- 172.21.15.134:0/3829463139 >> 172.21.15.134:0/3829463139 conn(0x7f11f00f0b70 msgr2=0x7f11f00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:44:00.831 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:00.830+0000 7f11f6231700 1 -- 172.21.15.134:0/3829463139 shutdown_connections 2024-01-21T11:44:00.831 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:00.830+0000 7f11f6231700 1 -- 172.21.15.134:0/3829463139 wait complete. 2024-01-21T11:44:00.831 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:00.831+0000 7f11f6231700 1 Processor -- start 2024-01-21T11:44:00.832 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:00.831+0000 7f11f6231700 1 -- start start 2024-01-21T11:44:00.833 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:00.831+0000 7f11f6231700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11f00f5410 0x7f11f0108920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:44:00.833 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:00.831+0000 7f11f6231700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11f0108e10 con 0x7f11f00f5410 2024-01-21T11:44:00.833 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:00.831+0000 7f11ef7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11f00f5410 0x7f11f0108920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:44:00.833 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:00.831+0000 7f11ef7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11f00f5410 0x7f11f0108920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:32928/0 (socket says 172.21.15.134:32928) 2024-01-21T11:44:00.833 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:00.831+0000 7f11ef7fe700 1 -- 172.21.15.134:0/3578652167 learned_addr learned my addr 172.21.15.134:0/3578652167 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:44:00.834 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:00.832+0000 7f11ef7fe700 1 -- 172.21.15.134:0/3578652167 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11e4005040 con 0x7f11f00f5410 2024-01-21T11:44:00.834 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:00.832+0000 7f11ef7fe700 1 --2- 172.21.15.134:0/3578652167 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11f00f5410 0x7f11f0108920 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f11e4013510 tx=0x7f11e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:44:00.834 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:00.832+0000 7f11d7fff700 1 -- 172.21.15.134:0/3578652167 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f11e4009320 con 0x7f11f00f5410 2024-01-21T11:44:00.834 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:00.832+0000 7f11f6231700 1 -- 172.21.15.134:0/3578652167 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11f0109010 con 0x7f11f00f5410 2024-01-21T11:44:00.834 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:00.832+0000 7f11d7fff700 1 -- 172.21.15.134:0/3578652167 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f11e4013f00 con 0x7f11f00f5410 2024-01-21T11:44:00.834 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:00.832+0000 7f11d7fff700 1 -- 172.21.15.134:0/3578652167 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f11e4014d90 con 0x7f11f00f5410 2024-01-21T11:44:00.834 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:00.832+0000 7f11f6231700 1 -- 172.21.15.134:0/3578652167 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11f01094a0 con 0x7f11f00f5410 2024-01-21T11:44:00.834 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:00.833+0000 7f11d7fff700 1 -- 172.21.15.134:0/3578652167 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f11e40263a0 con 0x7f11f00f5410 2024-01-21T11:44:00.834 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:00.833+0000 7f11d7fff700 1 --2- 172.21.15.134:0/3578652167 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f11d802ff60 0x7f11d8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:44:00.834 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:00.833+0000 7f11f6231700 1 -- 172.21.15.134:0/3578652167 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11f0045ad0 con 0x7f11f00f5410 2024-01-21T11:44:00.834 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:00.833+0000 7f11d7fff700 1 -- 172.21.15.134:0/3578652167 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f11e40410b0 con 0x7f11f00f5410 2024-01-21T11:44:00.834 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:00.833+0000 7f11eeffd700 1 --2- 172.21.15.134:0/3578652167 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f11d802ff60 0x7f11d8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:44:00.835 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:00.834+0000 7f11eeffd700 1 --2- 172.21.15.134:0/3578652167 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f11d802ff60 0x7f11d8032420 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f11e00096a0 tx=0x7f11e0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:44:00.838 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:00.838+0000 7f11d7fff700 1 -- 172.21.15.134:0/3578652167 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f11e406c2b0 con 0x7f11f00f5410 2024-01-21T11:44:01.040 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:01.040+0000 7f11f6231700 1 -- 172.21.15.134:0/3578652167 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f11f010a350 con 0x7f11f00f5410 2024-01-21T11:44:01.041 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:01.041+0000 7f11d7fff700 1 -- 172.21.15.134:0/3578652167 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f11e40126f0 con 0x7f11f00f5410 2024-01-21T11:44:01.041 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:44:01.041 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:44:01.042 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:01.042+0000 7f11f6231700 1 -- 172.21.15.134:0/3578652167 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f11d802ff60 msgr2=0x7f11d8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:44:01.042 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:01.042+0000 7f11f6231700 1 --2- 172.21.15.134:0/3578652167 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f11d802ff60 0x7f11d8032420 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f11e00096a0 tx=0x7f11e0006b40).stop 2024-01-21T11:44:01.043 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:01.042+0000 7f11f6231700 1 -- 172.21.15.134:0/3578652167 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11f00f5410 msgr2=0x7f11f0108920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:44:01.043 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:01.042+0000 7f11f6231700 1 --2- 172.21.15.134:0/3578652167 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11f00f5410 0x7f11f0108920 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f11e4013510 tx=0x7f11e400b040).stop 2024-01-21T11:44:01.043 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:01.042+0000 7f11f6231700 1 -- 172.21.15.134:0/3578652167 shutdown_connections 2024-01-21T11:44:01.043 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:01.042+0000 7f11f6231700 1 --2- 172.21.15.134:0/3578652167 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f11d802ff60 0x7f11d8032420 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:44:01.043 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:01.042+0000 7f11f6231700 1 --2- 172.21.15.134:0/3578652167 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11f00f5410 0x7f11f0108920 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:44:01.043 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:01.042+0000 7f11f6231700 1 -- 172.21.15.134:0/3578652167 >> 172.21.15.134:0/3578652167 conn(0x7f11f00f0b70 msgr2=0x7f11f005fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:44:01.043 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:01.042+0000 7f11f6231700 1 -- 172.21.15.134:0/3578652167 shutdown_connections 2024-01-21T11:44:01.043 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:01.042+0000 7f11f6231700 1 -- 172.21.15.134:0/3578652167 wait complete. 2024-01-21T11:44:01.043 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:44:01.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:44:01 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:43:59.820077+0000 mgr.smithi029.blhxxk (mgr.14164) 60 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:44:01.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:44:01 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:44:01.042052+0000 mon.smithi029 (mon.0) 151 : audit [DBG] from='client.? 172.21.15.134:0/3578652167' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:44:02.425 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:44:02.426 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:44:02.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:44:02 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:44:01.938653+0000 mon.smithi029 (mon.0) 152 : audit [INF] from='mgr.14164 172.21.15.29:0/3252518221' entity='mgr.smithi029.blhxxk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi029.blhxxk/mirror_snapshot_schedule"}]: dispatch 2024-01-21T11:44:02.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:44:02 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:44:01.941036+0000 mon.smithi029 (mon.0) 153 : audit [INF] from='mgr.14164 172.21.15.29:0/3252518221' entity='mgr.smithi029.blhxxk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi029.blhxxk/trash_purge_schedule"}]: dispatch 2024-01-21T11:44:03.614 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:03.613+0000 7fd752a28700 1 -- 172.21.15.134:0/3350247859 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd74c0f6510 msgr2=0x7fd74c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:44:03.614 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:03.613+0000 7fd752a28700 1 --2- 172.21.15.134:0/3350247859 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd74c0f6510 0x7fd74c0f6930 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7fd734009fb0 tx=0x7fd734009b30).stop 2024-01-21T11:44:03.614 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:03.613+0000 7fd752a28700 1 -- 172.21.15.134:0/3350247859 shutdown_connections 2024-01-21T11:44:03.614 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:03.613+0000 7fd752a28700 1 --2- 172.21.15.134:0/3350247859 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd74c0f6510 0x7fd74c0f6930 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:44:03.615 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:03.613+0000 7fd752a28700 1 -- 172.21.15.134:0/3350247859 >> 172.21.15.134:0/3350247859 conn(0x7fd74c0f1c70 msgr2=0x7fd74c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:44:03.615 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:03.613+0000 7fd752a28700 1 -- 172.21.15.134:0/3350247859 shutdown_connections 2024-01-21T11:44:03.615 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:03.613+0000 7fd752a28700 1 -- 172.21.15.134:0/3350247859 wait complete. 2024-01-21T11:44:03.615 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:03.613+0000 7fd752a28700 1 Processor -- start 2024-01-21T11:44:03.615 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:03.614+0000 7fd752a28700 1 -- start start 2024-01-21T11:44:03.615 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:03.614+0000 7fd752a28700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd74c0f6510 0x7fd74c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:44:03.615 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:03.614+0000 7fd752a28700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd74c106c20 con 0x7fd74c0f6510 2024-01-21T11:44:03.615 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:03.614+0000 7fd74bfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd74c0f6510 0x7fd74c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:44:03.615 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:03.614+0000 7fd74bfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd74c0f6510 0x7fd74c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:32948/0 (socket says 172.21.15.134:32948) 2024-01-21T11:44:03.615 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:03.614+0000 7fd74bfff700 1 -- 172.21.15.134:0/1285989279 learned_addr learned my addr 172.21.15.134:0/1285989279 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:44:03.617 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:03.614+0000 7fd74bfff700 1 -- 172.21.15.134:0/1285989279 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd734005040 con 0x7fd74c0f6510 2024-01-21T11:44:03.617 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:03.614+0000 7fd74bfff700 1 --2- 172.21.15.134:0/1285989279 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd74c0f6510 0x7fd74c106730 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fd734013510 tx=0x7fd73400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:44:03.617 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:03.615+0000 7fd748ff9700 1 -- 172.21.15.134:0/1285989279 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd734009320 con 0x7fd74c0f6510 2024-01-21T11:44:03.617 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:03.615+0000 7fd752a28700 1 -- 172.21.15.134:0/1285989279 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd74c106e20 con 0x7fd74c0f6510 2024-01-21T11:44:03.617 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:03.615+0000 7fd752a28700 1 -- 172.21.15.134:0/1285989279 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd74c1072b0 con 0x7fd74c0f6510 2024-01-21T11:44:03.617 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:03.615+0000 7fd748ff9700 1 -- 172.21.15.134:0/1285989279 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd734013f00 con 0x7fd74c0f6510 2024-01-21T11:44:03.617 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:03.615+0000 7fd748ff9700 1 -- 172.21.15.134:0/1285989279 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd734014d60 con 0x7fd74c0f6510 2024-01-21T11:44:03.617 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:03.616+0000 7fd7427fc700 1 -- 172.21.15.134:0/1285989279 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd74c045ad0 con 0x7fd74c0f6510 2024-01-21T11:44:03.617 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:03.616+0000 7fd748ff9700 1 -- 172.21.15.134:0/1285989279 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd7340095f0 con 0x7fd74c0f6510 2024-01-21T11:44:03.617 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:03.616+0000 7fd748ff9700 1 --2- 172.21.15.134:0/1285989279 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fd73802ff60 0x7fd738032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:44:03.617 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:03.616+0000 7fd748ff9700 1 -- 172.21.15.134:0/1285989279 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd734040fc0 con 0x7fd74c0f6510 2024-01-21T11:44:03.617 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:03.616+0000 7fd74b7fe700 1 --2- 172.21.15.134:0/1285989279 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fd73802ff60 0x7fd738032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:44:03.618 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:03.617+0000 7fd74b7fe700 1 --2- 172.21.15.134:0/1285989279 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fd73802ff60 0x7fd738032420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7fd73c0096a0 tx=0x7fd73c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:44:03.621 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:03.621+0000 7fd748ff9700 1 -- 172.21.15.134:0/1285989279 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd734014420 con 0x7fd74c0f6510 2024-01-21T11:44:03.831 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:03.831+0000 7fd7427fc700 1 -- 172.21.15.134:0/1285989279 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd74c0fbf60 con 0x7fd74c0f6510 2024-01-21T11:44:03.831 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:03.832+0000 7fd748ff9700 1 -- 172.21.15.134:0/1285989279 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd734012980 con 0x7fd74c0f6510 2024-01-21T11:44:03.832 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:44:03.832 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:44:03.833 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:03.833+0000 7fd7427fc700 1 -- 172.21.15.134:0/1285989279 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fd73802ff60 msgr2=0x7fd738032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:44:03.834 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:03.833+0000 7fd7427fc700 1 --2- 172.21.15.134:0/1285989279 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fd73802ff60 0x7fd738032420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7fd73c0096a0 tx=0x7fd73c006b40).stop 2024-01-21T11:44:03.834 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:03.833+0000 7fd7427fc700 1 -- 172.21.15.134:0/1285989279 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd74c0f6510 msgr2=0x7fd74c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:44:03.834 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:03.833+0000 7fd7427fc700 1 --2- 172.21.15.134:0/1285989279 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd74c0f6510 0x7fd74c106730 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fd734013510 tx=0x7fd73400b040).stop 2024-01-21T11:44:03.834 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:03.833+0000 7fd7427fc700 1 -- 172.21.15.134:0/1285989279 shutdown_connections 2024-01-21T11:44:03.834 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:03.833+0000 7fd7427fc700 1 --2- 172.21.15.134:0/1285989279 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fd73802ff60 0x7fd738032420 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:44:03.834 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:03.833+0000 7fd7427fc700 1 --2- 172.21.15.134:0/1285989279 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd74c0f6510 0x7fd74c106730 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:44:03.834 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:03.833+0000 7fd7427fc700 1 -- 172.21.15.134:0/1285989279 >> 172.21.15.134:0/1285989279 conn(0x7fd74c0f1c70 msgr2=0x7fd74c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:44:03.834 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:03.833+0000 7fd7427fc700 1 -- 172.21.15.134:0/1285989279 shutdown_connections 2024-01-21T11:44:03.834 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:03.833+0000 7fd7427fc700 1 -- 172.21.15.134:0/1285989279 wait complete. 2024-01-21T11:44:03.834 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:44:03.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:44:03 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:44:01.820326+0000 mgr.smithi029.blhxxk (mgr.14164) 61 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:44:04.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:44:04 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:44:03.832679+0000 mon.smithi029 (mon.0) 154 : audit [DBG] from='client.? 172.21.15.134:0/1285989279' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:44:05.222 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:44:05.222 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:44:05.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:44:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:44:03.820681+0000 mgr.smithi029.blhxxk (mgr.14164) 62 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:44:06.396 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:06.395+0000 7fc85f738700 1 -- 172.21.15.134:0/1412663978 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8580f4860 msgr2=0x7fc8580f4c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:44:06.396 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:06.396+0000 7fc85f738700 1 --2- 172.21.15.134:0/1412663978 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8580f4860 0x7fc8580f4c80 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7fc848009fb0 tx=0x7fc848009b30).stop 2024-01-21T11:44:06.396 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:06.396+0000 7fc85f738700 1 -- 172.21.15.134:0/1412663978 shutdown_connections 2024-01-21T11:44:06.396 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:06.396+0000 7fc85f738700 1 --2- 172.21.15.134:0/1412663978 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8580f4860 0x7fc8580f4c80 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:44:06.396 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:06.396+0000 7fc85f738700 1 -- 172.21.15.134:0/1412663978 >> 172.21.15.134:0/1412663978 conn(0x7fc8580f02e0 msgr2=0x7fc8580f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:44:06.396 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:06.396+0000 7fc85f738700 1 -- 172.21.15.134:0/1412663978 shutdown_connections 2024-01-21T11:44:06.396 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:06.396+0000 7fc85f738700 1 -- 172.21.15.134:0/1412663978 wait complete. 2024-01-21T11:44:06.398 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:06.396+0000 7fc85f738700 1 Processor -- start 2024-01-21T11:44:06.398 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:06.396+0000 7fc85f738700 1 -- start start 2024-01-21T11:44:06.398 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:06.396+0000 7fc85f738700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8580f4860 0x7fc8581068e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:44:06.398 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:06.396+0000 7fc85f738700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc858106dd0 con 0x7fc8580f4860 2024-01-21T11:44:06.398 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:06.396+0000 7fc85d4d4700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8580f4860 0x7fc8581068e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:44:06.398 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:06.396+0000 7fc85d4d4700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8580f4860 0x7fc8581068e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:34932/0 (socket says 172.21.15.134:34932) 2024-01-21T11:44:06.398 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:06.397+0000 7fc85d4d4700 1 -- 172.21.15.134:0/2507291311 learned_addr learned my addr 172.21.15.134:0/2507291311 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:44:06.398 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:06.397+0000 7fc85d4d4700 1 -- 172.21.15.134:0/2507291311 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc848005040 con 0x7fc8580f4860 2024-01-21T11:44:06.398 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:06.397+0000 7fc85d4d4700 1 --2- 172.21.15.134:0/2507291311 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8580f4860 0x7fc8581068e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fc848013510 tx=0x7fc84800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:44:06.399 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:06.397+0000 7fc84dffb700 1 -- 172.21.15.134:0/2507291311 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc848009320 con 0x7fc8580f4860 2024-01-21T11:44:06.399 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:06.397+0000 7fc85f738700 1 -- 172.21.15.134:0/2507291311 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc858106fd0 con 0x7fc8580f4860 2024-01-21T11:44:06.399 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:06.397+0000 7fc84dffb700 1 -- 172.21.15.134:0/2507291311 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc848013f00 con 0x7fc8580f4860 2024-01-21T11:44:06.399 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:06.397+0000 7fc84dffb700 1 -- 172.21.15.134:0/2507291311 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc848014dc0 con 0x7fc8580f4860 2024-01-21T11:44:06.399 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:06.397+0000 7fc85f738700 1 -- 172.21.15.134:0/2507291311 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc858107460 con 0x7fc8580f4860 2024-01-21T11:44:06.400 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:06.398+0000 7fc84dffb700 1 -- 172.21.15.134:0/2507291311 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc8480095f0 con 0x7fc8580f4860 2024-01-21T11:44:06.400 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:06.398+0000 7fc85f738700 1 -- 172.21.15.134:0/2507291311 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc858045ad0 con 0x7fc8580f4860 2024-01-21T11:44:06.400 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:06.398+0000 7fc84dffb700 1 --2- 172.21.15.134:0/2507291311 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fc84402ff60 0x7fc844032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:44:06.400 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:06.398+0000 7fc84dffb700 1 -- 172.21.15.134:0/2507291311 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc8480410e0 con 0x7fc8580f4860 2024-01-21T11:44:06.400 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:06.398+0000 7fc85ccd3700 1 --2- 172.21.15.134:0/2507291311 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fc84402ff60 0x7fc844032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:44:06.400 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:06.399+0000 7fc85ccd3700 1 --2- 172.21.15.134:0/2507291311 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fc84402ff60 0x7fc844032420 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fc8540096a0 tx=0x7fc854006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:44:06.403 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:06.403+0000 7fc84dffb700 1 -- 172.21.15.134:0/2507291311 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc848014420 con 0x7fc8580f4860 2024-01-21T11:44:06.610 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:06.609+0000 7fc85f738700 1 -- 172.21.15.134:0/2507291311 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc85805f370 con 0x7fc8580f4860 2024-01-21T11:44:06.610 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:06.610+0000 7fc84dffb700 1 -- 172.21.15.134:0/2507291311 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc84801c920 con 0x7fc8580f4860 2024-01-21T11:44:06.610 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:44:06.610 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:44:06.612 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:06.611+0000 7fc85f738700 1 -- 172.21.15.134:0/2507291311 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fc84402ff60 msgr2=0x7fc844032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:44:06.612 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:06.612+0000 7fc85f738700 1 --2- 172.21.15.134:0/2507291311 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fc84402ff60 0x7fc844032420 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fc8540096a0 tx=0x7fc854006b40).stop 2024-01-21T11:44:06.612 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:06.612+0000 7fc85f738700 1 -- 172.21.15.134:0/2507291311 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8580f4860 msgr2=0x7fc8581068e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:44:06.612 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:06.612+0000 7fc85f738700 1 --2- 172.21.15.134:0/2507291311 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8580f4860 0x7fc8581068e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fc848013510 tx=0x7fc84800b040).stop 2024-01-21T11:44:06.612 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:06.612+0000 7fc85f738700 1 -- 172.21.15.134:0/2507291311 shutdown_connections 2024-01-21T11:44:06.612 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:06.612+0000 7fc85f738700 1 --2- 172.21.15.134:0/2507291311 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fc84402ff60 0x7fc844032420 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:44:06.612 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:06.612+0000 7fc85f738700 1 --2- 172.21.15.134:0/2507291311 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8580f4860 0x7fc8581068e0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:44:06.613 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:06.612+0000 7fc85f738700 1 -- 172.21.15.134:0/2507291311 >> 172.21.15.134:0/2507291311 conn(0x7fc8580f02e0 msgr2=0x7fc8580f0e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:44:06.613 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:06.612+0000 7fc85f738700 1 -- 172.21.15.134:0/2507291311 shutdown_connections 2024-01-21T11:44:06.613 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:06.612+0000 7fc85f738700 1 -- 172.21.15.134:0/2507291311 wait complete. 2024-01-21T11:44:06.613 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:44:07.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:44:07 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:44:05.820958+0000 mgr.smithi029.blhxxk (mgr.14164) 63 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:44:07.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:44:07 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:44:06.611077+0000 mon.smithi029 (mon.0) 155 : audit [DBG] from='client.? 172.21.15.134:0/2507291311' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:44:07.992 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:44:07.992 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:44:09.120 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:09.120+0000 7f464f59e700 1 -- 172.21.15.134:0/2111309058 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4650060140 msgr2=0x7f46500f8fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:44:09.120 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:09.120+0000 7f464f59e700 1 --2- 172.21.15.134:0/2111309058 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4650060140 0x7f46500f8fc0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f4644009fb0 tx=0x7f4644009b30).stop 2024-01-21T11:44:09.120 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:09.120+0000 7f464f59e700 1 -- 172.21.15.134:0/2111309058 shutdown_connections 2024-01-21T11:44:09.120 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:09.120+0000 7f464f59e700 1 --2- 172.21.15.134:0/2111309058 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4650060140 0x7f46500f8fc0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:44:09.121 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:09.120+0000 7f464f59e700 1 -- 172.21.15.134:0/2111309058 >> 172.21.15.134:0/2111309058 conn(0x7f46500f09a0 msgr2=0x7f46500f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:44:09.121 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:09.120+0000 7f464f59e700 1 -- 172.21.15.134:0/2111309058 shutdown_connections 2024-01-21T11:44:09.121 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:09.120+0000 7f464f59e700 1 -- 172.21.15.134:0/2111309058 wait complete. 2024-01-21T11:44:09.122 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:09.120+0000 7f464f59e700 1 Processor -- start 2024-01-21T11:44:09.122 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:09.120+0000 7f464f59e700 1 -- start start 2024-01-21T11:44:09.122 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:09.121+0000 7f464f59e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4650060140 0x7f4650106520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:44:09.122 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:09.121+0000 7f464f59e700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4650106a10 con 0x7f4650060140 2024-01-21T11:44:09.122 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:09.121+0000 7f464e59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4650060140 0x7f4650106520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:44:09.122 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:09.121+0000 7f464e59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4650060140 0x7f4650106520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:34948/0 (socket says 172.21.15.134:34948) 2024-01-21T11:44:09.122 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:09.121+0000 7f464e59c700 1 -- 172.21.15.134:0/3724047700 learned_addr learned my addr 172.21.15.134:0/3724047700 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:44:09.123 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:09.121+0000 7f464e59c700 1 -- 172.21.15.134:0/3724047700 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4644005040 con 0x7f4650060140 2024-01-21T11:44:09.123 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:09.121+0000 7f464e59c700 1 --2- 172.21.15.134:0/3724047700 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4650060140 0x7f4650106520 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f4644014000 tx=0x7f464400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:44:09.123 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:09.122+0000 7f463effd700 1 -- 172.21.15.134:0/3724047700 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f46440095f0 con 0x7f4650060140 2024-01-21T11:44:09.123 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:09.122+0000 7f463effd700 1 -- 172.21.15.134:0/3724047700 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4644007e70 con 0x7f4650060140 2024-01-21T11:44:09.123 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:09.122+0000 7f464f59e700 1 -- 172.21.15.134:0/3724047700 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4650106c10 con 0x7f4650060140 2024-01-21T11:44:09.123 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:09.122+0000 7f463effd700 1 -- 172.21.15.134:0/3724047700 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4644014ea0 con 0x7f4650060140 2024-01-21T11:44:09.123 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:09.122+0000 7f464f59e700 1 -- 172.21.15.134:0/3724047700 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46501070a0 con 0x7f4650060140 2024-01-21T11:44:09.124 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:09.122+0000 7f463effd700 1 -- 172.21.15.134:0/3724047700 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4644025070 con 0x7f4650060140 2024-01-21T11:44:09.124 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:09.122+0000 7f464f59e700 1 -- 172.21.15.134:0/3724047700 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4650045ad0 con 0x7f4650060140 2024-01-21T11:44:09.124 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:09.123+0000 7f463effd700 1 --2- 172.21.15.134:0/3724047700 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f463802ffb0 0x7f4638032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:44:09.124 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:09.123+0000 7f463effd700 1 -- 172.21.15.134:0/3724047700 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4644040930 con 0x7f4650060140 2024-01-21T11:44:09.124 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:09.123+0000 7f464dd9b700 1 --2- 172.21.15.134:0/3724047700 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f463802ffb0 0x7f4638032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:44:09.124 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:09.123+0000 7f464dd9b700 1 --2- 172.21.15.134:0/3724047700 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f463802ffb0 0x7f4638032470 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f46400096a0 tx=0x7f4640006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:44:09.128 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:09.128+0000 7f463effd700 1 -- 172.21.15.134:0/3724047700 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f46440079c0 con 0x7f4650060140 2024-01-21T11:44:09.335 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:09.335+0000 7f464f59e700 1 -- 172.21.15.134:0/3724047700 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4650107f50 con 0x7f4650060140 2024-01-21T11:44:09.336 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:09.336+0000 7f463effd700 1 -- 172.21.15.134:0/3724047700 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f464400f300 con 0x7f4650060140 2024-01-21T11:44:09.336 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:44:09.336 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:44:09.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:09.337+0000 7f464f59e700 1 -- 172.21.15.134:0/3724047700 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f463802ffb0 msgr2=0x7f4638032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:44:09.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:09.337+0000 7f464f59e700 1 --2- 172.21.15.134:0/3724047700 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f463802ffb0 0x7f4638032470 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f46400096a0 tx=0x7f4640006b40).stop 2024-01-21T11:44:09.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:09.337+0000 7f464f59e700 1 -- 172.21.15.134:0/3724047700 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4650060140 msgr2=0x7f4650106520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:44:09.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:09.337+0000 7f464f59e700 1 --2- 172.21.15.134:0/3724047700 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4650060140 0x7f4650106520 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f4644014000 tx=0x7f464400b040).stop 2024-01-21T11:44:09.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:09.337+0000 7f464f59e700 1 -- 172.21.15.134:0/3724047700 shutdown_connections 2024-01-21T11:44:09.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:09.337+0000 7f464f59e700 1 --2- 172.21.15.134:0/3724047700 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f463802ffb0 0x7f4638032470 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:44:09.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:09.337+0000 7f464f59e700 1 --2- 172.21.15.134:0/3724047700 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4650060140 0x7f4650106520 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:44:09.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:09.337+0000 7f464f59e700 1 -- 172.21.15.134:0/3724047700 >> 172.21.15.134:0/3724047700 conn(0x7f46500f09a0 msgr2=0x7f46500f14d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:44:09.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:09.337+0000 7f464f59e700 1 -- 172.21.15.134:0/3724047700 shutdown_connections 2024-01-21T11:44:09.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:09.337+0000 7f464f59e700 1 -- 172.21.15.134:0/3724047700 wait complete. 2024-01-21T11:44:09.339 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:44:09.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:44:09 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:44:07.821312+0000 mgr.smithi029.blhxxk (mgr.14164) 64 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:44:09.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:44:09 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:44:09.336881+0000 mon.smithi029 (mon.0) 156 : audit [DBG] from='client.? 172.21.15.134:0/3724047700' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:44:10.719 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:44:10.719 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:44:11.863 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:11.862+0000 7fd346bb3700 1 -- 172.21.15.134:0/335429551 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3400f5410 msgr2=0x7fd3400f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:44:11.863 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:11.862+0000 7fd346bb3700 1 --2- 172.21.15.134:0/335429551 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3400f5410 0x7fd3400f5830 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7fd334009fb0 tx=0x7fd334009b30).stop 2024-01-21T11:44:11.863 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:11.862+0000 7fd346bb3700 1 -- 172.21.15.134:0/335429551 shutdown_connections 2024-01-21T11:44:11.863 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:11.862+0000 7fd346bb3700 1 --2- 172.21.15.134:0/335429551 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3400f5410 0x7fd3400f5830 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:44:11.863 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:11.862+0000 7fd346bb3700 1 -- 172.21.15.134:0/335429551 >> 172.21.15.134:0/335429551 conn(0x7fd3400f0b70 msgr2=0x7fd3400f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:44:11.863 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:11.862+0000 7fd346bb3700 1 -- 172.21.15.134:0/335429551 shutdown_connections 2024-01-21T11:44:11.863 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:11.862+0000 7fd346bb3700 1 -- 172.21.15.134:0/335429551 wait complete. 2024-01-21T11:44:11.863 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:11.862+0000 7fd346bb3700 1 Processor -- start 2024-01-21T11:44:11.863 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:11.862+0000 7fd346bb3700 1 -- start start 2024-01-21T11:44:11.863 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:11.862+0000 7fd346bb3700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3400f5410 0x7fd3400fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:44:11.864 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:11.863+0000 7fd346bb3700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3400fb820 con 0x7fd3400f5410 2024-01-21T11:44:11.864 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:11.863+0000 7fd34494f700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3400f5410 0x7fd3400fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:44:11.864 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:11.863+0000 7fd34494f700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3400f5410 0x7fd3400fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:34976/0 (socket says 172.21.15.134:34976) 2024-01-21T11:44:11.864 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:11.863+0000 7fd34494f700 1 -- 172.21.15.134:0/3056853632 learned_addr learned my addr 172.21.15.134:0/3056853632 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:44:11.865 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:11.863+0000 7fd34494f700 1 -- 172.21.15.134:0/3056853632 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd334005040 con 0x7fd3400f5410 2024-01-21T11:44:11.865 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:11.863+0000 7fd34494f700 1 --2- 172.21.15.134:0/3056853632 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3400f5410 0x7fd3400fd3a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7fd334013510 tx=0x7fd33400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:44:11.865 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:11.863+0000 7fd33d7fa700 1 -- 172.21.15.134:0/3056853632 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd334009320 con 0x7fd3400f5410 2024-01-21T11:44:11.865 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:11.863+0000 7fd346bb3700 1 -- 172.21.15.134:0/3056853632 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3400fb9c0 con 0x7fd3400f5410 2024-01-21T11:44:11.865 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:11.863+0000 7fd33d7fa700 1 -- 172.21.15.134:0/3056853632 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd334013f00 con 0x7fd3400f5410 2024-01-21T11:44:11.865 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:11.863+0000 7fd33d7fa700 1 -- 172.21.15.134:0/3056853632 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd334014d90 con 0x7fd3400f5410 2024-01-21T11:44:11.866 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:11.863+0000 7fd346bb3700 1 -- 172.21.15.134:0/3056853632 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3400fbe50 con 0x7fd3400f5410 2024-01-21T11:44:11.866 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:11.864+0000 7fd346bb3700 1 -- 172.21.15.134:0/3056853632 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd340059100 con 0x7fd3400f5410 2024-01-21T11:44:11.866 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:11.864+0000 7fd33d7fa700 1 -- 172.21.15.134:0/3056853632 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd3340095f0 con 0x7fd3400f5410 2024-01-21T11:44:11.866 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:11.864+0000 7fd33d7fa700 1 --2- 172.21.15.134:0/3056853632 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fd33002ff60 0x7fd330032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:44:11.866 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:11.864+0000 7fd33d7fa700 1 -- 172.21.15.134:0/3056853632 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd334041000 con 0x7fd3400f5410 2024-01-21T11:44:11.866 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:11.865+0000 7fd33ffff700 1 --2- 172.21.15.134:0/3056853632 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fd33002ff60 0x7fd330032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:44:11.866 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:11.865+0000 7fd33ffff700 1 --2- 172.21.15.134:0/3056853632 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fd33002ff60 0x7fd330032420 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7fd32c0096a0 tx=0x7fd32c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:44:11.869 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:11.869+0000 7fd33d7fa700 1 -- 172.21.15.134:0/3056853632 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd334014420 con 0x7fd3400f5410 2024-01-21T11:44:11.920 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:44:11 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:44:09.821756+0000 mgr.smithi029.blhxxk (mgr.14164) 65 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:44:12.074 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:12.074+0000 7fd346bb3700 1 -- 172.21.15.134:0/3056853632 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd340045ad0 con 0x7fd3400f5410 2024-01-21T11:44:12.075 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:12.075+0000 7fd33d7fa700 1 -- 172.21.15.134:0/3056853632 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd334010230 con 0x7fd3400f5410 2024-01-21T11:44:12.075 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:44:12.075 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:44:12.076 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:12.076+0000 7fd346bb3700 1 -- 172.21.15.134:0/3056853632 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fd33002ff60 msgr2=0x7fd330032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:44:12.076 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:12.076+0000 7fd346bb3700 1 --2- 172.21.15.134:0/3056853632 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fd33002ff60 0x7fd330032420 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7fd32c0096a0 tx=0x7fd32c006b40).stop 2024-01-21T11:44:12.076 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:12.076+0000 7fd346bb3700 1 -- 172.21.15.134:0/3056853632 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3400f5410 msgr2=0x7fd3400fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:44:12.077 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:12.076+0000 7fd346bb3700 1 --2- 172.21.15.134:0/3056853632 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3400f5410 0x7fd3400fd3a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7fd334013510 tx=0x7fd33400b040).stop 2024-01-21T11:44:12.077 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:12.076+0000 7fd346bb3700 1 -- 172.21.15.134:0/3056853632 shutdown_connections 2024-01-21T11:44:12.077 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:12.076+0000 7fd346bb3700 1 --2- 172.21.15.134:0/3056853632 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fd33002ff60 0x7fd330032420 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:44:12.077 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:12.076+0000 7fd346bb3700 1 --2- 172.21.15.134:0/3056853632 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3400f5410 0x7fd3400fd3a0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:44:12.077 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:12.076+0000 7fd346bb3700 1 -- 172.21.15.134:0/3056853632 >> 172.21.15.134:0/3056853632 conn(0x7fd3400f0b70 msgr2=0x7fd34005fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:44:12.077 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:12.076+0000 7fd346bb3700 1 -- 172.21.15.134:0/3056853632 shutdown_connections 2024-01-21T11:44:12.077 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:12.076+0000 7fd346bb3700 1 -- 172.21.15.134:0/3056853632 wait complete. 2024-01-21T11:44:12.077 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:44:12.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:44:12 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:44:12.075796+0000 mon.smithi029 (mon.0) 157 : audit [DBG] from='client.? 172.21.15.134:0/3056853632' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:44:13.495 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:44:13.495 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:44:13.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:44:13 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:44:11.822162+0000 mgr.smithi029.blhxxk (mgr.14164) 66 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:44:14.670 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:14.670+0000 7fc859808700 1 -- 172.21.15.134:0/2724844444 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc854060280 msgr2=0x7fc8540f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:44:14.671 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:14.670+0000 7fc859808700 1 --2- 172.21.15.134:0/2724844444 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc854060280 0x7fc8540f90f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7fc848009fb0 tx=0x7fc848009b30).stop 2024-01-21T11:44:14.671 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:14.670+0000 7fc859808700 1 -- 172.21.15.134:0/2724844444 shutdown_connections 2024-01-21T11:44:14.671 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:14.670+0000 7fc859808700 1 --2- 172.21.15.134:0/2724844444 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc854060280 0x7fc8540f90f0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:44:14.671 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:14.670+0000 7fc859808700 1 -- 172.21.15.134:0/2724844444 >> 172.21.15.134:0/2724844444 conn(0x7fc8540f0b70 msgr2=0x7fc8540f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:44:14.671 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:14.670+0000 7fc859808700 1 -- 172.21.15.134:0/2724844444 shutdown_connections 2024-01-21T11:44:14.671 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:14.670+0000 7fc859808700 1 -- 172.21.15.134:0/2724844444 wait complete. 2024-01-21T11:44:14.672 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:14.671+0000 7fc859808700 1 Processor -- start 2024-01-21T11:44:14.673 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:14.671+0000 7fc859808700 1 -- start start 2024-01-21T11:44:14.673 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:14.671+0000 7fc859808700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc854060280 0x7fc8541066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:44:14.673 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:14.671+0000 7fc859808700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc854106bd0 con 0x7fc854060280 2024-01-21T11:44:14.673 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:14.671+0000 7fc852ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc854060280 0x7fc8541066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:44:14.673 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:14.671+0000 7fc852ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc854060280 0x7fc8541066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:49376/0 (socket says 172.21.15.134:49376) 2024-01-21T11:44:14.673 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:14.671+0000 7fc852ffd700 1 -- 172.21.15.134:0/1135493863 learned_addr learned my addr 172.21.15.134:0/1135493863 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:44:14.673 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:14.672+0000 7fc852ffd700 1 -- 172.21.15.134:0/1135493863 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc848005040 con 0x7fc854060280 2024-01-21T11:44:14.673 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:14.672+0000 7fc852ffd700 1 --2- 172.21.15.134:0/1135493863 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc854060280 0x7fc8541066e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fc848013510 tx=0x7fc84800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:44:14.673 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:14.672+0000 7fc83b7fe700 1 -- 172.21.15.134:0/1135493863 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc848009320 con 0x7fc854060280 2024-01-21T11:44:14.673 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:14.672+0000 7fc859808700 1 -- 172.21.15.134:0/1135493863 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc854106dd0 con 0x7fc854060280 2024-01-21T11:44:14.673 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:14.672+0000 7fc83b7fe700 1 -- 172.21.15.134:0/1135493863 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc848013f00 con 0x7fc854060280 2024-01-21T11:44:14.673 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:14.672+0000 7fc83b7fe700 1 -- 172.21.15.134:0/1135493863 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc848014dc0 con 0x7fc854060280 2024-01-21T11:44:14.673 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:14.672+0000 7fc859808700 1 -- 172.21.15.134:0/1135493863 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc854107260 con 0x7fc854060280 2024-01-21T11:44:14.674 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:14.673+0000 7fc83b7fe700 1 -- 172.21.15.134:0/1135493863 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc8480095f0 con 0x7fc854060280 2024-01-21T11:44:14.674 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:14.673+0000 7fc859808700 1 -- 172.21.15.134:0/1135493863 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc854045ad0 con 0x7fc854060280 2024-01-21T11:44:14.675 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:14.673+0000 7fc83b7fe700 1 --2- 172.21.15.134:0/1135493863 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fc83c02ffb0 0x7fc83c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:44:14.675 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:14.673+0000 7fc83b7fe700 1 -- 172.21.15.134:0/1135493863 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc8480410c0 con 0x7fc854060280 2024-01-21T11:44:14.675 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:14.673+0000 7fc8527fc700 1 --2- 172.21.15.134:0/1135493863 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fc83c02ffb0 0x7fc83c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:44:14.675 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:14.674+0000 7fc8527fc700 1 --2- 172.21.15.134:0/1135493863 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fc83c02ffb0 0x7fc83c032470 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7fc8440096a0 tx=0x7fc844006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:44:14.678 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:14.678+0000 7fc83b7fe700 1 -- 172.21.15.134:0/1135493863 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc84801c790 con 0x7fc854060280 2024-01-21T11:44:14.886 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:14.885+0000 7fc859808700 1 -- 172.21.15.134:0/1135493863 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc854108110 con 0x7fc854060280 2024-01-21T11:44:14.886 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:14.886+0000 7fc83b7fe700 1 -- 172.21.15.134:0/1135493863 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc848010230 con 0x7fc854060280 2024-01-21T11:44:14.886 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:44:14.887 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:44:14.888 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:14.888+0000 7fc859808700 1 -- 172.21.15.134:0/1135493863 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fc83c02ffb0 msgr2=0x7fc83c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:44:14.888 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:14.888+0000 7fc859808700 1 --2- 172.21.15.134:0/1135493863 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fc83c02ffb0 0x7fc83c032470 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7fc8440096a0 tx=0x7fc844006b40).stop 2024-01-21T11:44:14.888 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:14.888+0000 7fc859808700 1 -- 172.21.15.134:0/1135493863 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc854060280 msgr2=0x7fc8541066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:44:14.888 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:14.888+0000 7fc859808700 1 --2- 172.21.15.134:0/1135493863 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc854060280 0x7fc8541066e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fc848013510 tx=0x7fc84800b040).stop 2024-01-21T11:44:14.888 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:14.888+0000 7fc859808700 1 -- 172.21.15.134:0/1135493863 shutdown_connections 2024-01-21T11:44:14.888 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:14.888+0000 7fc859808700 1 --2- 172.21.15.134:0/1135493863 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fc83c02ffb0 0x7fc83c032470 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:44:14.888 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:14.888+0000 7fc859808700 1 --2- 172.21.15.134:0/1135493863 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc854060280 0x7fc8541066e0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:44:14.888 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:14.888+0000 7fc859808700 1 -- 172.21.15.134:0/1135493863 >> 172.21.15.134:0/1135493863 conn(0x7fc8540f0b70 msgr2=0x7fc8540f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:44:14.888 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:14.888+0000 7fc859808700 1 -- 172.21.15.134:0/1135493863 shutdown_connections 2024-01-21T11:44:14.889 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:14.888+0000 7fc859808700 1 -- 172.21.15.134:0/1135493863 wait complete. 2024-01-21T11:44:14.889 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:44:15.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:44:15 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:44:13.822432+0000 mgr.smithi029.blhxxk (mgr.14164) 67 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:44:15.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:44:15 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:44:14.887300+0000 mon.smithi029 (mon.0) 158 : audit [DBG] from='client.? 172.21.15.134:0/1135493863' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:44:16.300 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:44:16.301 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:44:17.475 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:17.475+0000 7f2d1d030700 1 -- 172.21.15.134:0/2925416317 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d180f5330 msgr2=0x7f2d180f5750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:44:17.475 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:17.475+0000 7f2d1d030700 1 --2- 172.21.15.134:0/2925416317 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d180f5330 0x7f2d180f5750 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f2d0c009fb0 tx=0x7f2d0c009b30).stop 2024-01-21T11:44:17.476 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:17.475+0000 7f2d1d030700 1 -- 172.21.15.134:0/2925416317 shutdown_connections 2024-01-21T11:44:17.476 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:17.475+0000 7f2d1d030700 1 --2- 172.21.15.134:0/2925416317 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d180f5330 0x7f2d180f5750 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:44:17.476 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:17.475+0000 7f2d1d030700 1 -- 172.21.15.134:0/2925416317 >> 172.21.15.134:0/2925416317 conn(0x7f2d180f0a90 msgr2=0x7f2d180f2eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:44:17.476 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:17.475+0000 7f2d1d030700 1 -- 172.21.15.134:0/2925416317 shutdown_connections 2024-01-21T11:44:17.476 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:17.475+0000 7f2d1d030700 1 -- 172.21.15.134:0/2925416317 wait complete. 2024-01-21T11:44:17.477 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:17.475+0000 7f2d1d030700 1 Processor -- start 2024-01-21T11:44:17.477 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:17.476+0000 7f2d1d030700 1 -- start start 2024-01-21T11:44:17.477 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:17.476+0000 7f2d1d030700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d180f5330 0x7f2d181069d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:44:17.477 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:17.476+0000 7f2d1d030700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d18106ec0 con 0x7f2d180f5330 2024-01-21T11:44:17.478 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:17.476+0000 7f2d16d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d180f5330 0x7f2d181069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:44:17.478 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:17.476+0000 7f2d16d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d180f5330 0x7f2d181069d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:49392/0 (socket says 172.21.15.134:49392) 2024-01-21T11:44:17.478 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:17.476+0000 7f2d16d9d700 1 -- 172.21.15.134:0/1630373960 learned_addr learned my addr 172.21.15.134:0/1630373960 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:44:17.478 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:17.476+0000 7f2d16d9d700 1 -- 172.21.15.134:0/1630373960 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d0c005040 con 0x7f2d180f5330 2024-01-21T11:44:17.478 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:17.477+0000 7f2d16d9d700 1 --2- 172.21.15.134:0/1630373960 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d180f5330 0x7f2d181069d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f2d0c013510 tx=0x7f2d0c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:44:17.478 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:17.477+0000 7f2cff7fe700 1 -- 172.21.15.134:0/1630373960 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d0c009320 con 0x7f2d180f5330 2024-01-21T11:44:17.478 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:17.477+0000 7f2d1d030700 1 -- 172.21.15.134:0/1630373960 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d181070c0 con 0x7f2d180f5330 2024-01-21T11:44:17.478 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:17.477+0000 7f2cff7fe700 1 -- 172.21.15.134:0/1630373960 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2d0c013f00 con 0x7f2d180f5330 2024-01-21T11:44:17.478 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:17.477+0000 7f2cff7fe700 1 -- 172.21.15.134:0/1630373960 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d0c014d90 con 0x7f2d180f5330 2024-01-21T11:44:17.478 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:17.477+0000 7f2d1d030700 1 -- 172.21.15.134:0/1630373960 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d18107550 con 0x7f2d180f5330 2024-01-21T11:44:17.479 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:17.478+0000 7f2cff7fe700 1 -- 172.21.15.134:0/1630373960 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2d0c0095f0 con 0x7f2d180f5330 2024-01-21T11:44:17.479 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:17.478+0000 7f2d1d030700 1 -- 172.21.15.134:0/1630373960 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d18045ad0 con 0x7f2d180f5330 2024-01-21T11:44:17.479 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:17.478+0000 7f2cff7fe700 1 --2- 172.21.15.134:0/1630373960 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f2d0002ff60 0x7f2d00032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:44:17.479 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:17.478+0000 7f2cff7fe700 1 -- 172.21.15.134:0/1630373960 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2d0c040fe0 con 0x7f2d180f5330 2024-01-21T11:44:17.479 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:17.478+0000 7f2d1659c700 1 --2- 172.21.15.134:0/1630373960 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f2d0002ff60 0x7f2d00032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:44:17.479 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:17.478+0000 7f2d1659c700 1 --2- 172.21.15.134:0/1630373960 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f2d0002ff60 0x7f2d00032420 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f2d080096a0 tx=0x7f2d08006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:44:17.482 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:17.483+0000 7f2cff7fe700 1 -- 172.21.15.134:0/1630373960 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2d0c01d7f0 con 0x7f2d180f5330 2024-01-21T11:44:17.688 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:17.687+0000 7f2d1d030700 1 -- 172.21.15.134:0/1630373960 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2d1805ea50 con 0x7f2d180f5330 2024-01-21T11:44:17.688 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:17.688+0000 7f2cff7fe700 1 -- 172.21.15.134:0/1630373960 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2d0c010230 con 0x7f2d180f5330 2024-01-21T11:44:17.688 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:44:17.689 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:44:17.690 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:17.690+0000 7f2d1d030700 1 -- 172.21.15.134:0/1630373960 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f2d0002ff60 msgr2=0x7f2d00032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:44:17.690 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:17.690+0000 7f2d1d030700 1 --2- 172.21.15.134:0/1630373960 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f2d0002ff60 0x7f2d00032420 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f2d080096a0 tx=0x7f2d08006b40).stop 2024-01-21T11:44:17.690 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:17.690+0000 7f2d1d030700 1 -- 172.21.15.134:0/1630373960 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d180f5330 msgr2=0x7f2d181069d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:44:17.690 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:17.690+0000 7f2d1d030700 1 --2- 172.21.15.134:0/1630373960 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d180f5330 0x7f2d181069d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f2d0c013510 tx=0x7f2d0c00b040).stop 2024-01-21T11:44:17.690 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:17.690+0000 7f2d1d030700 1 -- 172.21.15.134:0/1630373960 shutdown_connections 2024-01-21T11:44:17.690 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:17.690+0000 7f2d1d030700 1 --2- 172.21.15.134:0/1630373960 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f2d0002ff60 0x7f2d00032420 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:44:17.690 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:17.690+0000 7f2d1d030700 1 --2- 172.21.15.134:0/1630373960 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d180f5330 0x7f2d181069d0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:44:17.691 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:17.690+0000 7f2d1d030700 1 -- 172.21.15.134:0/1630373960 >> 172.21.15.134:0/1630373960 conn(0x7f2d180f0a90 msgr2=0x7f2d180f1530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:44:17.691 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:17.690+0000 7f2d1d030700 1 -- 172.21.15.134:0/1630373960 shutdown_connections 2024-01-21T11:44:17.691 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:17.690+0000 7f2d1d030700 1 -- 172.21.15.134:0/1630373960 wait complete. 2024-01-21T11:44:17.691 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:44:17.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:44:17 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:44:15.822728+0000 mgr.smithi029.blhxxk (mgr.14164) 68 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:44:18.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:44:18 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:44:17.689172+0000 mon.smithi029 (mon.0) 159 : audit [DBG] from='client.? 172.21.15.134:0/1630373960' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:44:19.121 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:44:19.121 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:44:19.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:44:19 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:44:17.823074+0000 mgr.smithi029.blhxxk (mgr.14164) 69 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:44:20.225 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:20.223+0000 7f98d041a700 1 -- 172.21.15.134:0/4168940157 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98c80f6510 msgr2=0x7f98c80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:44:20.226 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:20.223+0000 7f98d041a700 1 --2- 172.21.15.134:0/4168940157 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98c80f6510 0x7f98c80f6930 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f98b8009fb0 tx=0x7f98b8009b30).stop 2024-01-21T11:44:20.226 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:20.223+0000 7f98d041a700 1 -- 172.21.15.134:0/4168940157 shutdown_connections 2024-01-21T11:44:20.226 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:20.223+0000 7f98d041a700 1 --2- 172.21.15.134:0/4168940157 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98c80f6510 0x7f98c80f6930 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:44:20.226 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:20.223+0000 7f98d041a700 1 -- 172.21.15.134:0/4168940157 >> 172.21.15.134:0/4168940157 conn(0x7f98c80f1c70 msgr2=0x7f98c80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:44:20.226 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:20.223+0000 7f98d041a700 1 -- 172.21.15.134:0/4168940157 shutdown_connections 2024-01-21T11:44:20.226 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:20.223+0000 7f98d041a700 1 -- 172.21.15.134:0/4168940157 wait complete. 2024-01-21T11:44:20.226 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:20.224+0000 7f98d041a700 1 Processor -- start 2024-01-21T11:44:20.226 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:20.224+0000 7f98d041a700 1 -- start start 2024-01-21T11:44:20.226 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:20.224+0000 7f98d041a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98c80f6510 0x7f98c8106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:44:20.226 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:20.224+0000 7f98d041a700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98c8106c20 con 0x7f98c80f6510 2024-01-21T11:44:20.226 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:20.224+0000 7f98ce1b6700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98c80f6510 0x7f98c8106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:44:20.227 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:20.224+0000 7f98ce1b6700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98c80f6510 0x7f98c8106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:49412/0 (socket says 172.21.15.134:49412) 2024-01-21T11:44:20.227 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:20.224+0000 7f98ce1b6700 1 -- 172.21.15.134:0/1400533784 learned_addr learned my addr 172.21.15.134:0/1400533784 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:44:20.227 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:20.225+0000 7f98ce1b6700 1 -- 172.21.15.134:0/1400533784 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98b8005040 con 0x7f98c80f6510 2024-01-21T11:44:20.227 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:20.225+0000 7f98ce1b6700 1 --2- 172.21.15.134:0/1400533784 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98c80f6510 0x7f98c8106730 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f98b8013510 tx=0x7f98b800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:44:20.227 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:20.225+0000 7f98beffd700 1 -- 172.21.15.134:0/1400533784 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f98b8009320 con 0x7f98c80f6510 2024-01-21T11:44:20.227 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:20.225+0000 7f98beffd700 1 -- 172.21.15.134:0/1400533784 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f98b8013f00 con 0x7f98c80f6510 2024-01-21T11:44:20.227 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:20.225+0000 7f98d041a700 1 -- 172.21.15.134:0/1400533784 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98c8106e20 con 0x7f98c80f6510 2024-01-21T11:44:20.227 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:20.225+0000 7f98beffd700 1 -- 172.21.15.134:0/1400533784 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f98b8014d90 con 0x7f98c80f6510 2024-01-21T11:44:20.227 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:20.225+0000 7f98d041a700 1 -- 172.21.15.134:0/1400533784 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98c81072b0 con 0x7f98c80f6510 2024-01-21T11:44:20.228 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:20.226+0000 7f98beffd700 1 -- 172.21.15.134:0/1400533784 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f98b80095f0 con 0x7f98c80f6510 2024-01-21T11:44:20.228 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:20.226+0000 7f98d041a700 1 -- 172.21.15.134:0/1400533784 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98c8045ad0 con 0x7f98c80f6510 2024-01-21T11:44:20.228 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:20.226+0000 7f98beffd700 1 --2- 172.21.15.134:0/1400533784 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f98b402ffb0 0x7f98b4032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:44:20.228 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:20.226+0000 7f98beffd700 1 -- 172.21.15.134:0/1400533784 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f98b8040ff0 con 0x7f98c80f6510 2024-01-21T11:44:20.228 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:20.226+0000 7f98cd9b5700 1 --2- 172.21.15.134:0/1400533784 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f98b402ffb0 0x7f98b4032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:44:20.228 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:20.227+0000 7f98cd9b5700 1 --2- 172.21.15.134:0/1400533784 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f98b402ffb0 0x7f98b4032470 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f98c40096a0 tx=0x7f98c4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:44:20.231 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:20.231+0000 7f98beffd700 1 -- 172.21.15.134:0/1400533784 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f98b806d2b0 con 0x7f98c80f6510 2024-01-21T11:44:20.442 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:20.442+0000 7f98d041a700 1 -- 172.21.15.134:0/1400533784 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f98c80fbf60 con 0x7f98c80f6510 2024-01-21T11:44:20.442 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:20.442+0000 7f98beffd700 1 -- 172.21.15.134:0/1400533784 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f98b8014420 con 0x7f98c80f6510 2024-01-21T11:44:20.443 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:44:20.443 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:44:20.444 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:20.444+0000 7f98d041a700 1 -- 172.21.15.134:0/1400533784 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f98b402ffb0 msgr2=0x7f98b4032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:44:20.445 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:20.444+0000 7f98d041a700 1 --2- 172.21.15.134:0/1400533784 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f98b402ffb0 0x7f98b4032470 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f98c40096a0 tx=0x7f98c4006b40).stop 2024-01-21T11:44:20.445 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:20.444+0000 7f98d041a700 1 -- 172.21.15.134:0/1400533784 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98c80f6510 msgr2=0x7f98c8106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:44:20.445 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:20.444+0000 7f98d041a700 1 --2- 172.21.15.134:0/1400533784 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98c80f6510 0x7f98c8106730 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f98b8013510 tx=0x7f98b800b040).stop 2024-01-21T11:44:20.445 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:20.444+0000 7f98d041a700 1 -- 172.21.15.134:0/1400533784 shutdown_connections 2024-01-21T11:44:20.445 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:20.444+0000 7f98d041a700 1 --2- 172.21.15.134:0/1400533784 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f98b402ffb0 0x7f98b4032470 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:44:20.445 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:20.444+0000 7f98d041a700 1 --2- 172.21.15.134:0/1400533784 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98c80f6510 0x7f98c8106730 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:44:20.445 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:20.444+0000 7f98d041a700 1 -- 172.21.15.134:0/1400533784 >> 172.21.15.134:0/1400533784 conn(0x7f98c80f1c70 msgr2=0x7f98c80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:44:20.445 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:20.444+0000 7f98d041a700 1 -- 172.21.15.134:0/1400533784 shutdown_connections 2024-01-21T11:44:20.445 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:20.444+0000 7f98d041a700 1 -- 172.21.15.134:0/1400533784 wait complete. 2024-01-21T11:44:20.446 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:44:20.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:44:20 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:44:20.443760+0000 mon.smithi029 (mon.0) 160 : audit [DBG] from='client.? 172.21.15.134:0/1400533784' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:44:21.836 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:44:21.836 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:44:21.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:44:21 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:44:19.823417+0000 mgr.smithi029.blhxxk (mgr.14164) 70 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:44:22.970 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:22.968+0000 7fd932b49700 1 -- 172.21.15.134:0/2222808502 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd92c0f6510 msgr2=0x7fd92c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:44:22.970 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:22.968+0000 7fd932b49700 1 --2- 172.21.15.134:0/2222808502 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd92c0f6510 0x7fd92c0f6930 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7fd920009fb0 tx=0x7fd920009b30).stop 2024-01-21T11:44:22.970 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:22.968+0000 7fd932b49700 1 -- 172.21.15.134:0/2222808502 shutdown_connections 2024-01-21T11:44:22.971 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:22.968+0000 7fd932b49700 1 --2- 172.21.15.134:0/2222808502 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd92c0f6510 0x7fd92c0f6930 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:44:22.971 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:22.968+0000 7fd932b49700 1 -- 172.21.15.134:0/2222808502 >> 172.21.15.134:0/2222808502 conn(0x7fd92c0f1c70 msgr2=0x7fd92c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:44:22.971 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:22.968+0000 7fd932b49700 1 -- 172.21.15.134:0/2222808502 shutdown_connections 2024-01-21T11:44:22.971 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:22.968+0000 7fd932b49700 1 -- 172.21.15.134:0/2222808502 wait complete. 2024-01-21T11:44:22.971 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:22.968+0000 7fd932b49700 1 Processor -- start 2024-01-21T11:44:22.971 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:22.968+0000 7fd932b49700 1 -- start start 2024-01-21T11:44:22.971 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:22.969+0000 7fd932b49700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd92c0f6510 0x7fd92c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:44:22.971 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:22.969+0000 7fd932b49700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd92c106c20 con 0x7fd92c0f6510 2024-01-21T11:44:22.971 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:22.969+0000 7fd9308e5700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd92c0f6510 0x7fd92c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:44:22.971 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:22.969+0000 7fd9308e5700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd92c0f6510 0x7fd92c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:49426/0 (socket says 172.21.15.134:49426) 2024-01-21T11:44:22.971 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:22.969+0000 7fd9308e5700 1 -- 172.21.15.134:0/2190241374 learned_addr learned my addr 172.21.15.134:0/2190241374 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:44:22.971 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:22.969+0000 7fd9308e5700 1 -- 172.21.15.134:0/2190241374 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd920005040 con 0x7fd92c0f6510 2024-01-21T11:44:22.972 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:22.970+0000 7fd9308e5700 1 --2- 172.21.15.134:0/2190241374 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd92c0f6510 0x7fd92c106730 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fd920013510 tx=0x7fd92000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:44:22.972 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:22.970+0000 7fd9297fa700 1 -- 172.21.15.134:0/2190241374 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd920009320 con 0x7fd92c0f6510 2024-01-21T11:44:22.972 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:22.970+0000 7fd932b49700 1 -- 172.21.15.134:0/2190241374 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd92c106e20 con 0x7fd92c0f6510 2024-01-21T11:44:22.972 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:22.970+0000 7fd9297fa700 1 -- 172.21.15.134:0/2190241374 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd920013f00 con 0x7fd92c0f6510 2024-01-21T11:44:22.972 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:22.970+0000 7fd9297fa700 1 -- 172.21.15.134:0/2190241374 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd920014d90 con 0x7fd92c0f6510 2024-01-21T11:44:22.972 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:22.970+0000 7fd932b49700 1 -- 172.21.15.134:0/2190241374 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd92c1072b0 con 0x7fd92c0f6510 2024-01-21T11:44:22.973 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:22.971+0000 7fd9297fa700 1 -- 172.21.15.134:0/2190241374 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd9200095f0 con 0x7fd92c0f6510 2024-01-21T11:44:22.973 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:22.971+0000 7fd932b49700 1 -- 172.21.15.134:0/2190241374 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd92c045ad0 con 0x7fd92c0f6510 2024-01-21T11:44:22.973 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:22.971+0000 7fd9297fa700 1 --2- 172.21.15.134:0/2190241374 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fd91c02ff60 0x7fd91c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:44:22.973 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:22.971+0000 7fd9297fa700 1 -- 172.21.15.134:0/2190241374 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd920041000 con 0x7fd92c0f6510 2024-01-21T11:44:22.973 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:22.971+0000 7fd92bfff700 1 --2- 172.21.15.134:0/2190241374 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fd91c02ff60 0x7fd91c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:44:22.973 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:22.972+0000 7fd92bfff700 1 --2- 172.21.15.134:0/2190241374 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fd91c02ff60 0x7fd91c032420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7fd9180096a0 tx=0x7fd918006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:44:22.976 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:22.976+0000 7fd9297fa700 1 -- 172.21.15.134:0/2190241374 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd920014420 con 0x7fd92c0f6510 2024-01-21T11:44:23.185 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:23.185+0000 7fd932b49700 1 -- 172.21.15.134:0/2190241374 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd92c0fbf60 con 0x7fd92c0f6510 2024-01-21T11:44:23.185 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:23.185+0000 7fd9297fa700 1 -- 172.21.15.134:0/2190241374 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd92001d7f0 con 0x7fd92c0f6510 2024-01-21T11:44:23.186 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:44:23.186 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:44:23.187 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:23.187+0000 7fd932b49700 1 -- 172.21.15.134:0/2190241374 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fd91c02ff60 msgr2=0x7fd91c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:44:23.187 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:23.187+0000 7fd932b49700 1 --2- 172.21.15.134:0/2190241374 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fd91c02ff60 0x7fd91c032420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7fd9180096a0 tx=0x7fd918006b40).stop 2024-01-21T11:44:23.187 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:23.187+0000 7fd932b49700 1 -- 172.21.15.134:0/2190241374 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd92c0f6510 msgr2=0x7fd92c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:44:23.187 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:23.187+0000 7fd932b49700 1 --2- 172.21.15.134:0/2190241374 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd92c0f6510 0x7fd92c106730 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fd920013510 tx=0x7fd92000b040).stop 2024-01-21T11:44:23.187 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:23.187+0000 7fd932b49700 1 -- 172.21.15.134:0/2190241374 shutdown_connections 2024-01-21T11:44:23.187 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:23.187+0000 7fd932b49700 1 --2- 172.21.15.134:0/2190241374 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fd91c02ff60 0x7fd91c032420 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:44:23.188 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:23.187+0000 7fd932b49700 1 --2- 172.21.15.134:0/2190241374 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd92c0f6510 0x7fd92c106730 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:44:23.188 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:23.187+0000 7fd932b49700 1 -- 172.21.15.134:0/2190241374 >> 172.21.15.134:0/2190241374 conn(0x7fd92c0f1c70 msgr2=0x7fd92c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:44:23.188 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:23.187+0000 7fd932b49700 1 -- 172.21.15.134:0/2190241374 shutdown_connections 2024-01-21T11:44:23.188 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:23.187+0000 7fd932b49700 1 -- 172.21.15.134:0/2190241374 wait complete. 2024-01-21T11:44:23.188 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:44:23.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:44:23 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:44:21.823723+0000 mgr.smithi029.blhxxk (mgr.14164) 71 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:44:23.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:44:23 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:44:23.186481+0000 mon.smithi029 (mon.0) 161 : audit [DBG] from='client.? 172.21.15.134:0/2190241374' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:44:24.581 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:44:24.582 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:44:25.746 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:25.744+0000 7fe2ac466700 1 -- 172.21.15.134:0/3048507454 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2a40f6510 msgr2=0x7fe2a40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:44:25.746 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:25.744+0000 7fe2ac466700 1 --2- 172.21.15.134:0/3048507454 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2a40f6510 0x7fe2a40f6930 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7fe298009fb0 tx=0x7fe298009b30).stop 2024-01-21T11:44:25.746 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:25.744+0000 7fe2ac466700 1 -- 172.21.15.134:0/3048507454 shutdown_connections 2024-01-21T11:44:25.746 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:25.744+0000 7fe2ac466700 1 --2- 172.21.15.134:0/3048507454 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2a40f6510 0x7fe2a40f6930 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:44:25.746 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:25.744+0000 7fe2ac466700 1 -- 172.21.15.134:0/3048507454 >> 172.21.15.134:0/3048507454 conn(0x7fe2a40f1c70 msgr2=0x7fe2a40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:44:25.746 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:25.744+0000 7fe2ac466700 1 -- 172.21.15.134:0/3048507454 shutdown_connections 2024-01-21T11:44:25.746 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:25.744+0000 7fe2ac466700 1 -- 172.21.15.134:0/3048507454 wait complete. 2024-01-21T11:44:25.747 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:25.744+0000 7fe2ac466700 1 Processor -- start 2024-01-21T11:44:25.747 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:25.744+0000 7fe2ac466700 1 -- start start 2024-01-21T11:44:25.747 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:25.745+0000 7fe2ac466700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2a40f6510 0x7fe2a4106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:44:25.747 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:25.745+0000 7fe2ac466700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2a4106c20 con 0x7fe2a40f6510 2024-01-21T11:44:25.747 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:25.745+0000 7fe2aa202700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2a40f6510 0x7fe2a4106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:44:25.747 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:25.745+0000 7fe2aa202700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2a40f6510 0x7fe2a4106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:49810/0 (socket says 172.21.15.134:49810) 2024-01-21T11:44:25.747 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:25.745+0000 7fe2aa202700 1 -- 172.21.15.134:0/3612007133 learned_addr learned my addr 172.21.15.134:0/3612007133 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:44:25.747 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:25.745+0000 7fe2aa202700 1 -- 172.21.15.134:0/3612007133 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe298005040 con 0x7fe2a40f6510 2024-01-21T11:44:25.747 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:25.745+0000 7fe2aa202700 1 --2- 172.21.15.134:0/3612007133 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2a40f6510 0x7fe2a4106730 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7fe298013510 tx=0x7fe29800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:44:25.747 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:25.746+0000 7fe296ffd700 1 -- 172.21.15.134:0/3612007133 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe298009320 con 0x7fe2a40f6510 2024-01-21T11:44:25.747 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:25.746+0000 7fe2ac466700 1 -- 172.21.15.134:0/3612007133 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2a4106e20 con 0x7fe2a40f6510 2024-01-21T11:44:25.748 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:25.746+0000 7fe296ffd700 1 -- 172.21.15.134:0/3612007133 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe298013f00 con 0x7fe2a40f6510 2024-01-21T11:44:25.748 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:25.746+0000 7fe296ffd700 1 -- 172.21.15.134:0/3612007133 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe298014d90 con 0x7fe2a40f6510 2024-01-21T11:44:25.748 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:25.746+0000 7fe2ac466700 1 -- 172.21.15.134:0/3612007133 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2a41072b0 con 0x7fe2a40f6510 2024-01-21T11:44:25.749 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:25.746+0000 7fe296ffd700 1 -- 172.21.15.134:0/3612007133 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe298014420 con 0x7fe2a40f6510 2024-01-21T11:44:25.749 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:25.746+0000 7fe2ac466700 1 -- 172.21.15.134:0/3612007133 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2a4045ad0 con 0x7fe2a40f6510 2024-01-21T11:44:25.749 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:25.747+0000 7fe296ffd700 1 --2- 172.21.15.134:0/3612007133 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fe29002ff60 0x7fe290032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:44:25.749 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:25.747+0000 7fe296ffd700 1 -- 172.21.15.134:0/3612007133 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe298041000 con 0x7fe2a40f6510 2024-01-21T11:44:25.749 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:25.747+0000 7fe2a9a01700 1 --2- 172.21.15.134:0/3612007133 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fe29002ff60 0x7fe290032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:44:25.749 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:25.747+0000 7fe2a9a01700 1 --2- 172.21.15.134:0/3612007133 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fe29002ff60 0x7fe290032420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fe2a00096a0 tx=0x7fe2a0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:44:25.751 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:25.751+0000 7fe296ffd700 1 -- 172.21.15.134:0/3612007133 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe2980095f0 con 0x7fe2a40f6510 2024-01-21T11:44:25.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:44:25 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:44:23.823977+0000 mgr.smithi029.blhxxk (mgr.14164) 72 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:44:25.957 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:25.957+0000 7fe2ac466700 1 -- 172.21.15.134:0/3612007133 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe2a40fbf60 con 0x7fe2a40f6510 2024-01-21T11:44:25.958 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:25.958+0000 7fe296ffd700 1 -- 172.21.15.134:0/3612007133 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe29801db70 con 0x7fe2a40f6510 2024-01-21T11:44:25.958 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:44:25.958 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:44:25.960 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:25.960+0000 7fe2ac466700 1 -- 172.21.15.134:0/3612007133 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fe29002ff60 msgr2=0x7fe290032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:44:25.960 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:25.960+0000 7fe2ac466700 1 --2- 172.21.15.134:0/3612007133 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fe29002ff60 0x7fe290032420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fe2a00096a0 tx=0x7fe2a0006b40).stop 2024-01-21T11:44:25.960 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:25.960+0000 7fe2ac466700 1 -- 172.21.15.134:0/3612007133 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2a40f6510 msgr2=0x7fe2a4106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:44:25.960 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:25.960+0000 7fe2ac466700 1 --2- 172.21.15.134:0/3612007133 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2a40f6510 0x7fe2a4106730 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7fe298013510 tx=0x7fe29800b040).stop 2024-01-21T11:44:25.960 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:25.960+0000 7fe2ac466700 1 -- 172.21.15.134:0/3612007133 shutdown_connections 2024-01-21T11:44:25.960 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:25.960+0000 7fe2ac466700 1 --2- 172.21.15.134:0/3612007133 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fe29002ff60 0x7fe290032420 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:44:25.960 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:25.960+0000 7fe2ac466700 1 --2- 172.21.15.134:0/3612007133 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2a40f6510 0x7fe2a4106730 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:44:25.960 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:25.960+0000 7fe2ac466700 1 -- 172.21.15.134:0/3612007133 >> 172.21.15.134:0/3612007133 conn(0x7fe2a40f1c70 msgr2=0x7fe2a40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:44:25.961 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:25.960+0000 7fe2ac466700 1 -- 172.21.15.134:0/3612007133 shutdown_connections 2024-01-21T11:44:25.961 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:25.960+0000 7fe2ac466700 1 -- 172.21.15.134:0/3612007133 wait complete. 2024-01-21T11:44:25.961 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:44:26.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:44:26 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:44:25.959027+0000 mon.smithi029 (mon.0) 162 : audit [DBG] from='client.? 172.21.15.134:0/3612007133' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:44:27.355 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:44:27.355 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:44:27.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:44:27 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:44:25.824216+0000 mgr.smithi029.blhxxk (mgr.14164) 73 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:44:28.553 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:28.551+0000 7fbcd3c10700 1 -- 172.21.15.134:0/251425885 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbccc0f6510 msgr2=0x7fbccc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:44:28.553 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:28.551+0000 7fbcd3c10700 1 --2- 172.21.15.134:0/251425885 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbccc0f6510 0x7fbccc0f6930 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7fbcbc009fb0 tx=0x7fbcbc009b30).stop 2024-01-21T11:44:28.553 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:28.551+0000 7fbcd3c10700 1 -- 172.21.15.134:0/251425885 shutdown_connections 2024-01-21T11:44:28.553 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:28.551+0000 7fbcd3c10700 1 --2- 172.21.15.134:0/251425885 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbccc0f6510 0x7fbccc0f6930 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:44:28.553 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:28.551+0000 7fbcd3c10700 1 -- 172.21.15.134:0/251425885 >> 172.21.15.134:0/251425885 conn(0x7fbccc0f1c70 msgr2=0x7fbccc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:44:28.553 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:28.551+0000 7fbcd3c10700 1 -- 172.21.15.134:0/251425885 shutdown_connections 2024-01-21T11:44:28.553 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:28.551+0000 7fbcd3c10700 1 -- 172.21.15.134:0/251425885 wait complete. 2024-01-21T11:44:28.553 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:28.552+0000 7fbcd3c10700 1 Processor -- start 2024-01-21T11:44:28.554 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:28.552+0000 7fbcd3c10700 1 -- start start 2024-01-21T11:44:28.554 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:28.552+0000 7fbcd3c10700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbccc0f6510 0x7fbccc106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:44:28.554 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:28.552+0000 7fbcd3c10700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbccc106c20 con 0x7fbccc0f6510 2024-01-21T11:44:28.554 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:28.552+0000 7fbcd19ac700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbccc0f6510 0x7fbccc106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:44:28.554 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:28.552+0000 7fbcd19ac700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbccc0f6510 0x7fbccc106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:49820/0 (socket says 172.21.15.134:49820) 2024-01-21T11:44:28.554 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:28.552+0000 7fbcd19ac700 1 -- 172.21.15.134:0/4227872031 learned_addr learned my addr 172.21.15.134:0/4227872031 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:44:28.554 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:28.552+0000 7fbcd19ac700 1 -- 172.21.15.134:0/4227872031 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcbc005040 con 0x7fbccc0f6510 2024-01-21T11:44:28.554 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:28.552+0000 7fbcd19ac700 1 --2- 172.21.15.134:0/4227872031 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbccc0f6510 0x7fbccc106730 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7fbcbc014000 tx=0x7fbcbc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:44:28.554 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:28.553+0000 7fbcc27fc700 1 -- 172.21.15.134:0/4227872031 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbcbc0095f0 con 0x7fbccc0f6510 2024-01-21T11:44:28.554 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:28.553+0000 7fbcd3c10700 1 -- 172.21.15.134:0/4227872031 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbccc106e20 con 0x7fbccc0f6510 2024-01-21T11:44:28.554 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:28.553+0000 7fbcc27fc700 1 -- 172.21.15.134:0/4227872031 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbcbc007e70 con 0x7fbccc0f6510 2024-01-21T11:44:28.554 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:28.553+0000 7fbcc27fc700 1 -- 172.21.15.134:0/4227872031 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbcbc014ea0 con 0x7fbccc0f6510 2024-01-21T11:44:28.554 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:28.553+0000 7fbcd3c10700 1 -- 172.21.15.134:0/4227872031 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbccc1072b0 con 0x7fbccc0f6510 2024-01-21T11:44:28.555 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:28.553+0000 7fbcd3c10700 1 -- 172.21.15.134:0/4227872031 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbccc045ad0 con 0x7fbccc0f6510 2024-01-21T11:44:28.555 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:28.553+0000 7fbcc27fc700 1 -- 172.21.15.134:0/4227872031 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbcbc025070 con 0x7fbccc0f6510 2024-01-21T11:44:28.555 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:28.554+0000 7fbcc27fc700 1 --2- 172.21.15.134:0/4227872031 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fbcb802ff60 0x7fbcb8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:44:28.555 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:28.554+0000 7fbcc27fc700 1 -- 172.21.15.134:0/4227872031 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbcbc040940 con 0x7fbccc0f6510 2024-01-21T11:44:28.555 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:28.554+0000 7fbcd11ab700 1 --2- 172.21.15.134:0/4227872031 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fbcb802ff60 0x7fbcb8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:44:28.555 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:28.554+0000 7fbcd11ab700 1 --2- 172.21.15.134:0/4227872031 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fbcb802ff60 0x7fbcb8032420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7fbcc80096a0 tx=0x7fbcc8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:44:28.558 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:28.558+0000 7fbcc27fc700 1 -- 172.21.15.134:0/4227872031 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbcbc0079c0 con 0x7fbccc0f6510 2024-01-21T11:44:28.763 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:28.763+0000 7fbcd3c10700 1 -- 172.21.15.134:0/4227872031 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbccc0fbf60 con 0x7fbccc0f6510 2024-01-21T11:44:28.764 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:28.764+0000 7fbcc27fc700 1 -- 172.21.15.134:0/4227872031 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbcbc00f300 con 0x7fbccc0f6510 2024-01-21T11:44:28.765 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:44:28.765 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:44:28.766 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:28.766+0000 7fbcd3c10700 1 -- 172.21.15.134:0/4227872031 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fbcb802ff60 msgr2=0x7fbcb8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:44:28.766 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:28.766+0000 7fbcd3c10700 1 --2- 172.21.15.134:0/4227872031 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fbcb802ff60 0x7fbcb8032420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7fbcc80096a0 tx=0x7fbcc8006b40).stop 2024-01-21T11:44:28.766 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:28.766+0000 7fbcd3c10700 1 -- 172.21.15.134:0/4227872031 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbccc0f6510 msgr2=0x7fbccc106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:44:28.766 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:28.766+0000 7fbcd3c10700 1 --2- 172.21.15.134:0/4227872031 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbccc0f6510 0x7fbccc106730 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7fbcbc014000 tx=0x7fbcbc00b040).stop 2024-01-21T11:44:28.766 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:28.766+0000 7fbcd3c10700 1 -- 172.21.15.134:0/4227872031 shutdown_connections 2024-01-21T11:44:28.767 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:28.766+0000 7fbcd3c10700 1 --2- 172.21.15.134:0/4227872031 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fbcb802ff60 0x7fbcb8032420 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:44:28.767 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:28.766+0000 7fbcd3c10700 1 --2- 172.21.15.134:0/4227872031 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbccc0f6510 0x7fbccc106730 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:44:28.767 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:28.766+0000 7fbcd3c10700 1 -- 172.21.15.134:0/4227872031 >> 172.21.15.134:0/4227872031 conn(0x7fbccc0f1c70 msgr2=0x7fbccc0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:44:28.767 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:28.766+0000 7fbcd3c10700 1 -- 172.21.15.134:0/4227872031 shutdown_connections 2024-01-21T11:44:28.767 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:28.766+0000 7fbcd3c10700 1 -- 172.21.15.134:0/4227872031 wait complete. 2024-01-21T11:44:28.767 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:44:29.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:44:29 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:44:27.824563+0000 mgr.smithi029.blhxxk (mgr.14164) 74 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:44:29.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:44:29 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:44:28.765347+0000 mon.smithi029 (mon.0) 163 : audit [DBG] from='client.? 172.21.15.134:0/4227872031' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:44:30.147 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:44:30.147 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:44:31.301 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:31.299+0000 7f7e8741a700 1 -- 172.21.15.134:0/554976999 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e800f51a0 msgr2=0x7f7e800f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:44:31.301 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:31.299+0000 7f7e8741a700 1 --2- 172.21.15.134:0/554976999 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e800f51a0 0x7f7e800f55c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f7e7c009fb0 tx=0x7f7e7c009b30).stop 2024-01-21T11:44:31.301 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:31.299+0000 7f7e8741a700 1 -- 172.21.15.134:0/554976999 shutdown_connections 2024-01-21T11:44:31.301 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:31.299+0000 7f7e8741a700 1 --2- 172.21.15.134:0/554976999 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e800f51a0 0x7f7e800f55c0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:44:31.301 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:31.299+0000 7f7e8741a700 1 -- 172.21.15.134:0/554976999 >> 172.21.15.134:0/554976999 conn(0x7f7e800f09a0 msgr2=0x7f7e800f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:44:31.301 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:31.299+0000 7f7e8741a700 1 -- 172.21.15.134:0/554976999 shutdown_connections 2024-01-21T11:44:31.301 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:31.299+0000 7f7e8741a700 1 -- 172.21.15.134:0/554976999 wait complete. 2024-01-21T11:44:31.301 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:31.299+0000 7f7e8741a700 1 Processor -- start 2024-01-21T11:44:31.301 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:31.299+0000 7f7e8741a700 1 -- start start 2024-01-21T11:44:31.302 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:31.300+0000 7f7e8741a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e800f51a0 0x7f7e801091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:44:31.302 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:31.300+0000 7f7e8741a700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e80060220 con 0x7f7e800f51a0 2024-01-21T11:44:31.302 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:31.300+0000 7f7e86418700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e800f51a0 0x7f7e801091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:44:31.302 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:31.300+0000 7f7e86418700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e800f51a0 0x7f7e801091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:49836/0 (socket says 172.21.15.134:49836) 2024-01-21T11:44:31.302 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:31.300+0000 7f7e86418700 1 -- 172.21.15.134:0/2829005724 learned_addr learned my addr 172.21.15.134:0/2829005724 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:44:31.302 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:31.300+0000 7f7e86418700 1 -- 172.21.15.134:0/2829005724 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e7c005040 con 0x7f7e800f51a0 2024-01-21T11:44:31.302 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:31.300+0000 7f7e86418700 1 --2- 172.21.15.134:0/2829005724 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e800f51a0 0x7f7e801091d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f7e7c013510 tx=0x7f7e7c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:44:31.302 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:31.300+0000 7f7e76ffd700 1 -- 172.21.15.134:0/2829005724 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e7c009320 con 0x7f7e800f51a0 2024-01-21T11:44:31.302 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:31.301+0000 7f7e8741a700 1 -- 172.21.15.134:0/2829005724 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e80060420 con 0x7f7e800f51a0 2024-01-21T11:44:31.302 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:31.301+0000 7f7e76ffd700 1 -- 172.21.15.134:0/2829005724 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7e7c013f00 con 0x7f7e800f51a0 2024-01-21T11:44:31.302 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:31.301+0000 7f7e76ffd700 1 -- 172.21.15.134:0/2829005724 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e7c014dc0 con 0x7f7e800f51a0 2024-01-21T11:44:31.302 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:31.301+0000 7f7e8741a700 1 -- 172.21.15.134:0/2829005724 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e80109880 con 0x7f7e800f51a0 2024-01-21T11:44:31.303 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:31.301+0000 7f7e76ffd700 1 -- 172.21.15.134:0/2829005724 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7e7c014420 con 0x7f7e800f51a0 2024-01-21T11:44:31.303 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:31.301+0000 7f7e8741a700 1 -- 172.21.15.134:0/2829005724 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e80059100 con 0x7f7e800f51a0 2024-01-21T11:44:31.303 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:31.302+0000 7f7e76ffd700 1 --2- 172.21.15.134:0/2829005724 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f7e6c02ff60 0x7f7e6c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:44:31.303 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:31.302+0000 7f7e76ffd700 1 -- 172.21.15.134:0/2829005724 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7e7c041050 con 0x7f7e800f51a0 2024-01-21T11:44:31.304 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:31.302+0000 7f7e85c17700 1 --2- 172.21.15.134:0/2829005724 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f7e6c02ff60 0x7f7e6c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:44:31.304 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:31.302+0000 7f7e85c17700 1 --2- 172.21.15.134:0/2829005724 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f7e6c02ff60 0x7f7e6c032420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f7e700096a0 tx=0x7f7e70006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:44:31.306 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:31.306+0000 7f7e76ffd700 1 -- 172.21.15.134:0/2829005724 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7e7c0095f0 con 0x7f7e800f51a0 2024-01-21T11:44:31.514 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:31.514+0000 7f7e8741a700 1 -- 172.21.15.134:0/2829005724 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7e80045ad0 con 0x7f7e800f51a0 2024-01-21T11:44:31.515 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:31.515+0000 7f7e76ffd700 1 -- 172.21.15.134:0/2829005724 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7e7c01cb10 con 0x7f7e800f51a0 2024-01-21T11:44:31.515 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:44:31.516 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:44:31.517 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:31.517+0000 7f7e8741a700 1 -- 172.21.15.134:0/2829005724 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f7e6c02ff60 msgr2=0x7f7e6c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:44:31.517 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:31.517+0000 7f7e8741a700 1 --2- 172.21.15.134:0/2829005724 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f7e6c02ff60 0x7f7e6c032420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f7e700096a0 tx=0x7f7e70006b40).stop 2024-01-21T11:44:31.517 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:31.517+0000 7f7e8741a700 1 -- 172.21.15.134:0/2829005724 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e800f51a0 msgr2=0x7f7e801091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:44:31.517 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:31.517+0000 7f7e8741a700 1 --2- 172.21.15.134:0/2829005724 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e800f51a0 0x7f7e801091d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f7e7c013510 tx=0x7f7e7c00b040).stop 2024-01-21T11:44:31.517 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:31.517+0000 7f7e8741a700 1 -- 172.21.15.134:0/2829005724 shutdown_connections 2024-01-21T11:44:31.517 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:31.517+0000 7f7e8741a700 1 --2- 172.21.15.134:0/2829005724 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f7e6c02ff60 0x7f7e6c032420 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:44:31.517 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:31.517+0000 7f7e8741a700 1 --2- 172.21.15.134:0/2829005724 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e800f51a0 0x7f7e801091d0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:44:31.517 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:31.517+0000 7f7e8741a700 1 -- 172.21.15.134:0/2829005724 >> 172.21.15.134:0/2829005724 conn(0x7f7e800f09a0 msgr2=0x7f7e800f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:44:31.518 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:31.517+0000 7f7e8741a700 1 -- 172.21.15.134:0/2829005724 shutdown_connections 2024-01-21T11:44:31.518 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:31.517+0000 7f7e8741a700 1 -- 172.21.15.134:0/2829005724 wait complete. 2024-01-21T11:44:31.518 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:44:31.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:44:31 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:44:29.824974+0000 mgr.smithi029.blhxxk (mgr.14164) 75 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:44:31.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:44:31 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:44:31.516174+0000 mon.smithi029 (mon.0) 164 : audit [DBG] from='client.? 172.21.15.134:0/2829005724' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:44:32.903 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:44:32.903 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:44:33.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:44:33 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:44:31.825267+0000 mgr.smithi029.blhxxk (mgr.14164) 76 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:44:34.082 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:34.080+0000 7f8083aae700 1 -- 172.21.15.134:0/2964138595 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f807c0f6510 msgr2=0x7f807c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:44:34.082 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:34.080+0000 7f8083aae700 1 --2- 172.21.15.134:0/2964138595 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f807c0f6510 0x7f807c0f6930 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f806c009fb0 tx=0x7f806c009b30).stop 2024-01-21T11:44:34.082 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:34.080+0000 7f8083aae700 1 -- 172.21.15.134:0/2964138595 shutdown_connections 2024-01-21T11:44:34.083 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:34.080+0000 7f8083aae700 1 --2- 172.21.15.134:0/2964138595 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f807c0f6510 0x7f807c0f6930 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:44:34.083 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:34.080+0000 7f8083aae700 1 -- 172.21.15.134:0/2964138595 >> 172.21.15.134:0/2964138595 conn(0x7f807c0f1c70 msgr2=0x7f807c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:44:34.083 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:34.081+0000 7f8083aae700 1 -- 172.21.15.134:0/2964138595 shutdown_connections 2024-01-21T11:44:34.083 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:34.081+0000 7f8083aae700 1 -- 172.21.15.134:0/2964138595 wait complete. 2024-01-21T11:44:34.083 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:34.081+0000 7f8083aae700 1 Processor -- start 2024-01-21T11:44:34.083 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:34.081+0000 7f8083aae700 1 -- start start 2024-01-21T11:44:34.083 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:34.081+0000 7f8083aae700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f807c0f6510 0x7f807c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:44:34.083 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:34.081+0000 7f8083aae700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f807c069e20 con 0x7f807c0f6510 2024-01-21T11:44:34.083 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:34.081+0000 7f808184a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f807c0f6510 0x7f807c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:44:34.083 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:34.081+0000 7f808184a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f807c0f6510 0x7f807c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:49842/0 (socket says 172.21.15.134:49842) 2024-01-21T11:44:34.083 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:34.081+0000 7f808184a700 1 -- 172.21.15.134:0/3219946085 learned_addr learned my addr 172.21.15.134:0/3219946085 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:44:34.083 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:34.082+0000 7f808184a700 1 -- 172.21.15.134:0/3219946085 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f806c005040 con 0x7f807c0f6510 2024-01-21T11:44:34.083 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:34.082+0000 7f808184a700 1 --2- 172.21.15.134:0/3219946085 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f807c0f6510 0x7f807c06b9a0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f806c013510 tx=0x7f806c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:44:34.083 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:34.082+0000 7f80727fc700 1 -- 172.21.15.134:0/3219946085 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f806c009320 con 0x7f807c0f6510 2024-01-21T11:44:34.084 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:34.082+0000 7f8083aae700 1 -- 172.21.15.134:0/3219946085 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f807c069fc0 con 0x7f807c0f6510 2024-01-21T11:44:34.084 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:34.082+0000 7f80727fc700 1 -- 172.21.15.134:0/3219946085 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f806c013f00 con 0x7f807c0f6510 2024-01-21T11:44:34.084 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:34.082+0000 7f80727fc700 1 -- 172.21.15.134:0/3219946085 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f806c014d90 con 0x7f807c0f6510 2024-01-21T11:44:34.084 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:34.082+0000 7f8083aae700 1 -- 172.21.15.134:0/3219946085 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f807c06a450 con 0x7f807c0f6510 2024-01-21T11:44:34.085 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:34.083+0000 7f80727fc700 1 -- 172.21.15.134:0/3219946085 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f806c0095f0 con 0x7f807c0f6510 2024-01-21T11:44:34.085 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:34.083+0000 7f8083aae700 1 -- 172.21.15.134:0/3219946085 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f807c059100 con 0x7f807c0f6510 2024-01-21T11:44:34.085 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:34.083+0000 7f80727fc700 1 --2- 172.21.15.134:0/3219946085 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f806802ff60 0x7f8068032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:44:34.085 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:34.083+0000 7f80727fc700 1 -- 172.21.15.134:0/3219946085 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f806c0410c0 con 0x7f807c0f6510 2024-01-21T11:44:34.085 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:34.083+0000 7f8081049700 1 --2- 172.21.15.134:0/3219946085 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f806802ff60 0x7f8068032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:44:34.085 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:34.084+0000 7f8081049700 1 --2- 172.21.15.134:0/3219946085 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f806802ff60 0x7f8068032420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f80780096a0 tx=0x7f8078006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:44:34.088 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:34.088+0000 7f80727fc700 1 -- 172.21.15.134:0/3219946085 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f806c014420 con 0x7f807c0f6510 2024-01-21T11:44:34.296 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:34.296+0000 7f8083aae700 1 -- 172.21.15.134:0/3219946085 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f807c0fbf60 con 0x7f807c0f6510 2024-01-21T11:44:34.297 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:34.297+0000 7f80727fc700 1 -- 172.21.15.134:0/3219946085 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f806c010230 con 0x7f807c0f6510 2024-01-21T11:44:34.297 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:44:34.297 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:44:34.299 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:34.299+0000 7f8083aae700 1 -- 172.21.15.134:0/3219946085 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f806802ff60 msgr2=0x7f8068032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:44:34.299 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:34.299+0000 7f8083aae700 1 --2- 172.21.15.134:0/3219946085 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f806802ff60 0x7f8068032420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f80780096a0 tx=0x7f8078006b40).stop 2024-01-21T11:44:34.299 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:34.299+0000 7f8083aae700 1 -- 172.21.15.134:0/3219946085 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f807c0f6510 msgr2=0x7f807c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:44:34.299 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:34.299+0000 7f8083aae700 1 --2- 172.21.15.134:0/3219946085 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f807c0f6510 0x7f807c06b9a0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f806c013510 tx=0x7f806c00b040).stop 2024-01-21T11:44:34.299 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:34.299+0000 7f8083aae700 1 -- 172.21.15.134:0/3219946085 shutdown_connections 2024-01-21T11:44:34.300 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:34.299+0000 7f8083aae700 1 --2- 172.21.15.134:0/3219946085 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f806802ff60 0x7f8068032420 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:44:34.300 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:34.299+0000 7f8083aae700 1 --2- 172.21.15.134:0/3219946085 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f807c0f6510 0x7f807c06b9a0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:44:34.300 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:34.299+0000 7f8083aae700 1 -- 172.21.15.134:0/3219946085 >> 172.21.15.134:0/3219946085 conn(0x7f807c0f1c70 msgr2=0x7f807c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:44:34.300 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:34.299+0000 7f8083aae700 1 -- 172.21.15.134:0/3219946085 shutdown_connections 2024-01-21T11:44:34.300 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:34.299+0000 7f8083aae700 1 -- 172.21.15.134:0/3219946085 wait complete. 2024-01-21T11:44:34.300 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:44:34.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:44:34 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:44:34.297965+0000 mon.smithi029 (mon.0) 165 : audit [DBG] from='client.? 172.21.15.134:0/3219946085' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:44:35.699 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:44:35.699 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:44:35.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:44:35 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:44:33.825536+0000 mgr.smithi029.blhxxk (mgr.14164) 77 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:44:36.798 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:36.796+0000 7fd532eb6700 1 -- 172.21.15.134:0/1268966047 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd52c060280 msgr2=0x7fd52c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:44:36.798 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:36.796+0000 7fd532eb6700 1 --2- 172.21.15.134:0/1268966047 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd52c060280 0x7fd52c0f90f0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7fd520009fb0 tx=0x7fd520009b30).stop 2024-01-21T11:44:36.798 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:36.796+0000 7fd532eb6700 1 -- 172.21.15.134:0/1268966047 shutdown_connections 2024-01-21T11:44:36.798 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:36.796+0000 7fd532eb6700 1 --2- 172.21.15.134:0/1268966047 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd52c060280 0x7fd52c0f90f0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:44:36.798 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:36.796+0000 7fd532eb6700 1 -- 172.21.15.134:0/1268966047 >> 172.21.15.134:0/1268966047 conn(0x7fd52c0f0b70 msgr2=0x7fd52c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:44:36.798 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:36.796+0000 7fd532eb6700 1 -- 172.21.15.134:0/1268966047 shutdown_connections 2024-01-21T11:44:36.798 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:36.796+0000 7fd532eb6700 1 -- 172.21.15.134:0/1268966047 wait complete. 2024-01-21T11:44:36.798 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:36.797+0000 7fd532eb6700 1 Processor -- start 2024-01-21T11:44:36.798 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:36.797+0000 7fd532eb6700 1 -- start start 2024-01-21T11:44:36.798 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:36.797+0000 7fd532eb6700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd52c060280 0x7fd52c0f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:44:36.798 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:36.797+0000 7fd532eb6700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd52c0f4ff0 con 0x7fd52c060280 2024-01-21T11:44:36.799 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:36.797+0000 7fd530c52700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd52c060280 0x7fd52c0f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:44:36.799 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:36.797+0000 7fd530c52700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd52c060280 0x7fd52c0f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:53088/0 (socket says 172.21.15.134:53088) 2024-01-21T11:44:36.799 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:36.797+0000 7fd530c52700 1 -- 172.21.15.134:0/471411682 learned_addr learned my addr 172.21.15.134:0/471411682 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:44:36.799 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:36.798+0000 7fd530c52700 1 -- 172.21.15.134:0/471411682 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd520005040 con 0x7fd52c060280 2024-01-21T11:44:36.799 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:36.798+0000 7fd530c52700 1 --2- 172.21.15.134:0/471411682 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd52c060280 0x7fd52c0f6b70 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7fd520013510 tx=0x7fd52000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:44:36.800 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:36.798+0000 7fd5297fa700 1 -- 172.21.15.134:0/471411682 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd520009320 con 0x7fd52c060280 2024-01-21T11:44:36.800 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:36.798+0000 7fd532eb6700 1 -- 172.21.15.134:0/471411682 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd52c0f5190 con 0x7fd52c060280 2024-01-21T11:44:36.800 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:36.798+0000 7fd5297fa700 1 -- 172.21.15.134:0/471411682 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd520013f00 con 0x7fd52c060280 2024-01-21T11:44:36.801 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:36.798+0000 7fd5297fa700 1 -- 172.21.15.134:0/471411682 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd520014dc0 con 0x7fd52c060280 2024-01-21T11:44:36.801 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:36.798+0000 7fd532eb6700 1 -- 172.21.15.134:0/471411682 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd52c0f5620 con 0x7fd52c060280 2024-01-21T11:44:36.801 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:36.799+0000 7fd5297fa700 1 -- 172.21.15.134:0/471411682 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd52001d3d0 con 0x7fd52c060280 2024-01-21T11:44:36.801 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:36.799+0000 7fd5297fa700 1 --2- 172.21.15.134:0/471411682 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fd51c02ff60 0x7fd51c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:44:36.801 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:36.799+0000 7fd5297fa700 1 -- 172.21.15.134:0/471411682 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd520041100 con 0x7fd52c060280 2024-01-21T11:44:36.801 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:36.799+0000 7fd532eb6700 1 -- 172.21.15.134:0/471411682 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd52c059100 con 0x7fd52c060280 2024-01-21T11:44:36.801 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:36.799+0000 7fd52bfff700 1 --2- 172.21.15.134:0/471411682 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fd51c02ff60 0x7fd51c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:44:36.801 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:36.800+0000 7fd52bfff700 1 --2- 172.21.15.134:0/471411682 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fd51c02ff60 0x7fd51c032420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7fd5180096a0 tx=0x7fd518006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:44:36.804 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:36.804+0000 7fd5297fa700 1 -- 172.21.15.134:0/471411682 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd520012920 con 0x7fd52c060280 2024-01-21T11:44:37.009 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:37.009+0000 7fd532eb6700 1 -- 172.21.15.134:0/471411682 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd52c045ad0 con 0x7fd52c060280 2024-01-21T11:44:37.009 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:37.009+0000 7fd5297fa700 1 -- 172.21.15.134:0/471411682 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd52002b020 con 0x7fd52c060280 2024-01-21T11:44:37.010 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:44:37.010 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:44:37.011 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:37.011+0000 7fd532eb6700 1 -- 172.21.15.134:0/471411682 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fd51c02ff60 msgr2=0x7fd51c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:44:37.011 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:37.011+0000 7fd532eb6700 1 --2- 172.21.15.134:0/471411682 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fd51c02ff60 0x7fd51c032420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7fd5180096a0 tx=0x7fd518006b40).stop 2024-01-21T11:44:37.012 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:37.011+0000 7fd532eb6700 1 -- 172.21.15.134:0/471411682 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd52c060280 msgr2=0x7fd52c0f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:44:37.012 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:37.011+0000 7fd532eb6700 1 --2- 172.21.15.134:0/471411682 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd52c060280 0x7fd52c0f6b70 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7fd520013510 tx=0x7fd52000b040).stop 2024-01-21T11:44:37.012 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:37.011+0000 7fd532eb6700 1 -- 172.21.15.134:0/471411682 shutdown_connections 2024-01-21T11:44:37.012 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:37.011+0000 7fd532eb6700 1 --2- 172.21.15.134:0/471411682 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fd51c02ff60 0x7fd51c032420 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:44:37.012 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:37.011+0000 7fd532eb6700 1 --2- 172.21.15.134:0/471411682 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd52c060280 0x7fd52c0f6b70 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:44:37.012 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:37.011+0000 7fd532eb6700 1 -- 172.21.15.134:0/471411682 >> 172.21.15.134:0/471411682 conn(0x7fd52c0f0b70 msgr2=0x7fd52c0f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:44:37.012 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:37.011+0000 7fd532eb6700 1 -- 172.21.15.134:0/471411682 shutdown_connections 2024-01-21T11:44:37.012 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:37.011+0000 7fd532eb6700 1 -- 172.21.15.134:0/471411682 wait complete. 2024-01-21T11:44:37.012 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:44:37.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:44:37 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:44:35.825799+0000 mgr.smithi029.blhxxk (mgr.14164) 78 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:44:37.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:44:37 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:44:37.010630+0000 mon.smithi029 (mon.0) 166 : audit [DBG] from='client.? 172.21.15.134:0/471411682' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:44:38.396 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:44:38.396 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:44:38.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:44:38 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:44:37.826175+0000 mgr.smithi029.blhxxk (mgr.14164) 79 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:44:39.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:39.555+0000 7f3f43348700 1 -- 172.21.15.134:0/503168689 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f3c0f5410 msgr2=0x7f3f3c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:44:39.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:39.555+0000 7f3f43348700 1 --2- 172.21.15.134:0/503168689 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f3c0f5410 0x7f3f3c0f5830 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f3f30009fb0 tx=0x7f3f30009b30).stop 2024-01-21T11:44:39.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:39.556+0000 7f3f43348700 1 -- 172.21.15.134:0/503168689 shutdown_connections 2024-01-21T11:44:39.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:39.556+0000 7f3f43348700 1 --2- 172.21.15.134:0/503168689 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f3c0f5410 0x7f3f3c0f5830 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:44:39.557 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:39.556+0000 7f3f43348700 1 -- 172.21.15.134:0/503168689 >> 172.21.15.134:0/503168689 conn(0x7f3f3c0f0b70 msgr2=0x7f3f3c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:44:39.557 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:39.556+0000 7f3f43348700 1 -- 172.21.15.134:0/503168689 shutdown_connections 2024-01-21T11:44:39.557 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:39.556+0000 7f3f43348700 1 -- 172.21.15.134:0/503168689 wait complete. 2024-01-21T11:44:39.557 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:39.556+0000 7f3f43348700 1 Processor -- start 2024-01-21T11:44:39.557 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:39.556+0000 7f3f43348700 1 -- start start 2024-01-21T11:44:39.557 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:39.556+0000 7f3f43348700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f3c0f5410 0x7f3f3c105600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:44:39.557 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:39.556+0000 7f3f43348700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f3c105af0 con 0x7f3f3c0f5410 2024-01-21T11:44:39.557 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:39.556+0000 7f3f410e4700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f3c0f5410 0x7f3f3c105600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:44:39.557 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:39.556+0000 7f3f410e4700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f3c0f5410 0x7f3f3c105600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:53106/0 (socket says 172.21.15.134:53106) 2024-01-21T11:44:39.557 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:39.556+0000 7f3f410e4700 1 -- 172.21.15.134:0/2321559138 learned_addr learned my addr 172.21.15.134:0/2321559138 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:44:39.557 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:39.557+0000 7f3f410e4700 1 -- 172.21.15.134:0/2321559138 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f30005040 con 0x7f3f3c0f5410 2024-01-21T11:44:39.557 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:39.557+0000 7f3f410e4700 1 --2- 172.21.15.134:0/2321559138 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f3c0f5410 0x7f3f3c105600 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f3f30013510 tx=0x7f3f3000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:44:39.557 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:39.557+0000 7f3f2dffb700 1 -- 172.21.15.134:0/2321559138 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3f30009320 con 0x7f3f3c0f5410 2024-01-21T11:44:39.558 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:39.557+0000 7f3f43348700 1 -- 172.21.15.134:0/2321559138 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f3c105cf0 con 0x7f3f3c0f5410 2024-01-21T11:44:39.558 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:39.557+0000 7f3f2dffb700 1 -- 172.21.15.134:0/2321559138 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3f30013f00 con 0x7f3f3c0f5410 2024-01-21T11:44:39.558 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:39.557+0000 7f3f2dffb700 1 -- 172.21.15.134:0/2321559138 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3f30014d90 con 0x7f3f3c0f5410 2024-01-21T11:44:39.559 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:39.557+0000 7f3f43348700 1 -- 172.21.15.134:0/2321559138 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f3c106180 con 0x7f3f3c0f5410 2024-01-21T11:44:39.559 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:39.558+0000 7f3f43348700 1 -- 172.21.15.134:0/2321559138 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f3c045ad0 con 0x7f3f3c0f5410 2024-01-21T11:44:39.559 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:39.558+0000 7f3f2dffb700 1 -- 172.21.15.134:0/2321559138 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3f300095f0 con 0x7f3f3c0f5410 2024-01-21T11:44:39.559 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:39.558+0000 7f3f2dffb700 1 --2- 172.21.15.134:0/2321559138 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f3f2802ff60 0x7f3f28032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:44:39.559 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:39.558+0000 7f3f2dffb700 1 -- 172.21.15.134:0/2321559138 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3f30041000 con 0x7f3f3c0f5410 2024-01-21T11:44:39.559 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:39.559+0000 7f3f408e3700 1 --2- 172.21.15.134:0/2321559138 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f3f2802ff60 0x7f3f28032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:44:39.559 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:39.559+0000 7f3f408e3700 1 --2- 172.21.15.134:0/2321559138 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f3f2802ff60 0x7f3f28032420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f3f380096a0 tx=0x7f3f38006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:44:39.562 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:39.563+0000 7f3f2dffb700 1 -- 172.21.15.134:0/2321559138 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3f30014420 con 0x7f3f3c0f5410 2024-01-21T11:44:39.770 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:39.770+0000 7f3f43348700 1 -- 172.21.15.134:0/2321559138 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3f3c0fae60 con 0x7f3f3c0f5410 2024-01-21T11:44:39.770 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:39.771+0000 7f3f2dffb700 1 -- 172.21.15.134:0/2321559138 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3f3001d7f0 con 0x7f3f3c0f5410 2024-01-21T11:44:39.770 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:44:39.770 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:44:39.772 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:39.773+0000 7f3f43348700 1 -- 172.21.15.134:0/2321559138 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f3f2802ff60 msgr2=0x7f3f28032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:44:39.772 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:39.773+0000 7f3f43348700 1 --2- 172.21.15.134:0/2321559138 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f3f2802ff60 0x7f3f28032420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f3f380096a0 tx=0x7f3f38006b40).stop 2024-01-21T11:44:39.772 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:39.773+0000 7f3f43348700 1 -- 172.21.15.134:0/2321559138 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f3c0f5410 msgr2=0x7f3f3c105600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:44:39.772 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:39.773+0000 7f3f43348700 1 --2- 172.21.15.134:0/2321559138 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f3c0f5410 0x7f3f3c105600 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f3f30013510 tx=0x7f3f3000b040).stop 2024-01-21T11:44:39.772 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:39.773+0000 7f3f43348700 1 -- 172.21.15.134:0/2321559138 shutdown_connections 2024-01-21T11:44:39.772 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:39.773+0000 7f3f43348700 1 --2- 172.21.15.134:0/2321559138 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f3f2802ff60 0x7f3f28032420 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:44:39.772 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:39.773+0000 7f3f43348700 1 --2- 172.21.15.134:0/2321559138 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f3c0f5410 0x7f3f3c105600 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:44:39.772 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:39.773+0000 7f3f43348700 1 -- 172.21.15.134:0/2321559138 >> 172.21.15.134:0/2321559138 conn(0x7f3f3c0f0b70 msgr2=0x7f3f3c0f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:44:39.773 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:39.773+0000 7f3f43348700 1 -- 172.21.15.134:0/2321559138 shutdown_connections 2024-01-21T11:44:39.773 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:39.773+0000 7f3f43348700 1 -- 172.21.15.134:0/2321559138 wait complete. 2024-01-21T11:44:39.773 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:44:40.168 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:44:39 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:44:39.771266+0000 mon.smithi029 (mon.0) 167 : audit [DBG] from='client.? 172.21.15.134:0/2321559138' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:44:41.146 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:44:41.146 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:44:41.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:44:41 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:44:39.826426+0000 mgr.smithi029.blhxxk (mgr.14164) 80 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:44:42.273 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:42.270+0000 7f8cf15b0700 1 -- 172.21.15.134:0/2016130849 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8cec0f6a90 msgr2=0x7f8cec0f6eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:44:42.273 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:42.270+0000 7f8cf15b0700 1 --2- 172.21.15.134:0/2016130849 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8cec0f6a90 0x7f8cec0f6eb0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f8ce0009fb0 tx=0x7f8ce0009b30).stop 2024-01-21T11:44:42.273 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:42.270+0000 7f8cf15b0700 1 -- 172.21.15.134:0/2016130849 shutdown_connections 2024-01-21T11:44:42.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:42.270+0000 7f8cf15b0700 1 --2- 172.21.15.134:0/2016130849 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8cec0f6a90 0x7f8cec0f6eb0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:44:42.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:42.270+0000 7f8cf15b0700 1 -- 172.21.15.134:0/2016130849 >> 172.21.15.134:0/2016130849 conn(0x7f8cec0f02e0 msgr2=0x7f8cec0f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:44:42.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:42.270+0000 7f8cf15b0700 1 -- 172.21.15.134:0/2016130849 shutdown_connections 2024-01-21T11:44:42.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:42.270+0000 7f8cf15b0700 1 -- 172.21.15.134:0/2016130849 wait complete. 2024-01-21T11:44:42.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:42.271+0000 7f8cf15b0700 1 Processor -- start 2024-01-21T11:44:42.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:42.271+0000 7f8cf15b0700 1 -- start start 2024-01-21T11:44:42.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:42.271+0000 7f8cf15b0700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8cec0f6a90 0x7f8cec1088a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:44:42.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:42.271+0000 7f8cf15b0700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cec108d90 con 0x7f8cec0f6a90 2024-01-21T11:44:42.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:42.271+0000 7f8ceaffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8cec0f6a90 0x7f8cec1088a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:44:42.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:42.271+0000 7f8ceaffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8cec0f6a90 0x7f8cec1088a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:53126/0 (socket says 172.21.15.134:53126) 2024-01-21T11:44:42.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:42.271+0000 7f8ceaffd700 1 -- 172.21.15.134:0/2651670375 learned_addr learned my addr 172.21.15.134:0/2651670375 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:44:42.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:42.272+0000 7f8ceaffd700 1 -- 172.21.15.134:0/2651670375 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ce0005040 con 0x7f8cec0f6a90 2024-01-21T11:44:42.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:42.272+0000 7f8ceaffd700 1 --2- 172.21.15.134:0/2651670375 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8cec0f6a90 0x7f8cec1088a0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f8ce0013510 tx=0x7f8ce000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:44:42.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:42.272+0000 7f8cd37fe700 1 -- 172.21.15.134:0/2651670375 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ce0009320 con 0x7f8cec0f6a90 2024-01-21T11:44:42.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:42.272+0000 7f8cf15b0700 1 -- 172.21.15.134:0/2651670375 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cec108f90 con 0x7f8cec0f6a90 2024-01-21T11:44:42.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:42.272+0000 7f8cd37fe700 1 -- 172.21.15.134:0/2651670375 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8ce0013f00 con 0x7f8cec0f6a90 2024-01-21T11:44:42.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:42.272+0000 7f8cd37fe700 1 -- 172.21.15.134:0/2651670375 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ce0014dc0 con 0x7f8cec0f6a90 2024-01-21T11:44:42.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:42.272+0000 7f8cf15b0700 1 -- 172.21.15.134:0/2651670375 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cec109420 con 0x7f8cec0f6a90 2024-01-21T11:44:42.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:42.273+0000 7f8cd37fe700 1 -- 172.21.15.134:0/2651670375 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8ce00095f0 con 0x7f8cec0f6a90 2024-01-21T11:44:42.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:42.273+0000 7f8cf15b0700 1 -- 172.21.15.134:0/2651670375 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cec045ad0 con 0x7f8cec0f6a90 2024-01-21T11:44:42.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:42.273+0000 7f8cd37fe700 1 --2- 172.21.15.134:0/2651670375 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f8cd402ff60 0x7f8cd4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:44:42.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:42.273+0000 7f8cd37fe700 1 -- 172.21.15.134:0/2651670375 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8ce0041000 con 0x7f8cec0f6a90 2024-01-21T11:44:42.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:42.273+0000 7f8cea7fc700 1 --2- 172.21.15.134:0/2651670375 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f8cd402ff60 0x7f8cd4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:44:42.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:42.274+0000 7f8cea7fc700 1 --2- 172.21.15.134:0/2651670375 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f8cd402ff60 0x7f8cd4032420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f8cdc0096a0 tx=0x7f8cdc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:44:42.278 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:42.278+0000 7f8cd37fe700 1 -- 172.21.15.134:0/2651670375 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8ce001c790 con 0x7f8cec0f6a90 2024-01-21T11:44:42.487 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:42.487+0000 7f8cf15b0700 1 -- 172.21.15.134:0/2651670375 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8cec10a2d0 con 0x7f8cec0f6a90 2024-01-21T11:44:42.488 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:42.488+0000 7f8cd37fe700 1 -- 172.21.15.134:0/2651670375 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8ce0010230 con 0x7f8cec0f6a90 2024-01-21T11:44:42.488 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:44:42.489 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:44:42.490 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:42.490+0000 7f8cf15b0700 1 -- 172.21.15.134:0/2651670375 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f8cd402ff60 msgr2=0x7f8cd4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:44:42.490 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:42.490+0000 7f8cf15b0700 1 --2- 172.21.15.134:0/2651670375 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f8cd402ff60 0x7f8cd4032420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f8cdc0096a0 tx=0x7f8cdc006b40).stop 2024-01-21T11:44:42.490 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:42.490+0000 7f8cf15b0700 1 -- 172.21.15.134:0/2651670375 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8cec0f6a90 msgr2=0x7f8cec1088a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:44:42.490 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:42.490+0000 7f8cf15b0700 1 --2- 172.21.15.134:0/2651670375 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8cec0f6a90 0x7f8cec1088a0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f8ce0013510 tx=0x7f8ce000b040).stop 2024-01-21T11:44:42.490 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:42.490+0000 7f8cf15b0700 1 -- 172.21.15.134:0/2651670375 shutdown_connections 2024-01-21T11:44:42.490 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:42.490+0000 7f8cf15b0700 1 --2- 172.21.15.134:0/2651670375 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f8cd402ff60 0x7f8cd4032420 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:44:42.491 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:42.490+0000 7f8cf15b0700 1 --2- 172.21.15.134:0/2651670375 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8cec0f6a90 0x7f8cec1088a0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:44:42.491 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:42.490+0000 7f8cf15b0700 1 -- 172.21.15.134:0/2651670375 >> 172.21.15.134:0/2651670375 conn(0x7f8cec0f02e0 msgr2=0x7f8cec0f0dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:44:42.491 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:42.490+0000 7f8cf15b0700 1 -- 172.21.15.134:0/2651670375 shutdown_connections 2024-01-21T11:44:42.491 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:42.490+0000 7f8cf15b0700 1 -- 172.21.15.134:0/2651670375 wait complete. 2024-01-21T11:44:42.491 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:44:42.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:44:42 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:44:42.489158+0000 mon.smithi029 (mon.0) 168 : audit [DBG] from='client.? 172.21.15.134:0/2651670375' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:44:43.882 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:44:43.882 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:44:43.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:44:43 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:44:41.826745+0000 mgr.smithi029.blhxxk (mgr.14164) 81 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:44:44.999 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:44.998+0000 7f0b9bfff700 1 -- 172.21.15.134:0/827135918 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b9c0f51a0 msgr2=0x7f0b9c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:44:45.000 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:44.998+0000 7f0b9bfff700 1 --2- 172.21.15.134:0/827135918 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b9c0f51a0 0x7f0b9c0f55c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f0b90009fb0 tx=0x7f0b90009b30).stop 2024-01-21T11:44:45.000 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:44.998+0000 7f0b9bfff700 1 -- 172.21.15.134:0/827135918 shutdown_connections 2024-01-21T11:44:45.000 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:44.998+0000 7f0b9bfff700 1 --2- 172.21.15.134:0/827135918 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b9c0f51a0 0x7f0b9c0f55c0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:44:45.000 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:44.998+0000 7f0b9bfff700 1 -- 172.21.15.134:0/827135918 >> 172.21.15.134:0/827135918 conn(0x7f0b9c0f09a0 msgr2=0x7f0b9c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:44:45.000 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:44.998+0000 7f0b9bfff700 1 -- 172.21.15.134:0/827135918 shutdown_connections 2024-01-21T11:44:45.000 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:44.998+0000 7f0b9bfff700 1 -- 172.21.15.134:0/827135918 wait complete. 2024-01-21T11:44:45.000 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:44.998+0000 7f0b9bfff700 1 Processor -- start 2024-01-21T11:44:45.000 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:44.998+0000 7f0b9bfff700 1 -- start start 2024-01-21T11:44:45.000 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:44.999+0000 7f0b9bfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b9c0f51a0 0x7f0b9c109200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:44:45.000 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:44.999+0000 7f0b9bfff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b9c060220 con 0x7f0b9c0f51a0 2024-01-21T11:44:45.000 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:44.999+0000 7f0b9affd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b9c0f51a0 0x7f0b9c109200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:44:45.001 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:44.999+0000 7f0b9affd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b9c0f51a0 0x7f0b9c109200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:34472/0 (socket says 172.21.15.134:34472) 2024-01-21T11:44:45.001 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:44.999+0000 7f0b9affd700 1 -- 172.21.15.134:0/3238010291 learned_addr learned my addr 172.21.15.134:0/3238010291 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:44:45.001 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:44.999+0000 7f0b9affd700 1 -- 172.21.15.134:0/3238010291 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b90005040 con 0x7f0b9c0f51a0 2024-01-21T11:44:45.001 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:44.999+0000 7f0b9affd700 1 --2- 172.21.15.134:0/3238010291 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b9c0f51a0 0x7f0b9c109200 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f0b90009830 tx=0x7f0b9000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:44:45.001 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:44.999+0000 7f0b837fe700 1 -- 172.21.15.134:0/3238010291 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b90009320 con 0x7f0b9c0f51a0 2024-01-21T11:44:45.001 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:44.999+0000 7f0b9bfff700 1 -- 172.21.15.134:0/3238010291 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b9c060420 con 0x7f0b9c0f51a0 2024-01-21T11:44:45.001 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:44.999+0000 7f0b837fe700 1 -- 172.21.15.134:0/3238010291 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0b900136c0 con 0x7f0b9c0f51a0 2024-01-21T11:44:45.001 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:44.999+0000 7f0b837fe700 1 -- 172.21.15.134:0/3238010291 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b9001ee90 con 0x7f0b9c0f51a0 2024-01-21T11:44:45.001 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:45.000+0000 7f0b9bfff700 1 -- 172.21.15.134:0/3238010291 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b9c1098b0 con 0x7f0b9c0f51a0 2024-01-21T11:44:45.002 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:45.000+0000 7f0b9bfff700 1 -- 172.21.15.134:0/3238010291 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b9c059100 con 0x7f0b9c0f51a0 2024-01-21T11:44:45.002 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:45.000+0000 7f0b837fe700 1 -- 172.21.15.134:0/3238010291 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0b90014440 con 0x7f0b9c0f51a0 2024-01-21T11:44:45.002 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:45.000+0000 7f0b837fe700 1 --2- 172.21.15.134:0/3238010291 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f0b8402ff10 0x7f0b840323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:44:45.002 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:45.001+0000 7f0b837fe700 1 -- 172.21.15.134:0/3238010291 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0b90041220 con 0x7f0b9c0f51a0 2024-01-21T11:44:45.002 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:45.001+0000 7f0b9a7fc700 1 --2- 172.21.15.134:0/3238010291 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f0b8402ff10 0x7f0b840323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:44:45.002 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:45.001+0000 7f0b9a7fc700 1 --2- 172.21.15.134:0/3238010291 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f0b8402ff10 0x7f0b840323d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f0b8c0096a0 tx=0x7f0b8c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:44:45.005 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:45.005+0000 7f0b837fe700 1 -- 172.21.15.134:0/3238010291 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0b9001c320 con 0x7f0b9c0f51a0 2024-01-21T11:44:45.210 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:45.210+0000 7f0b9bfff700 1 -- 172.21.15.134:0/3238010291 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0b9c0fabe0 con 0x7f0b9c0f51a0 2024-01-21T11:44:45.210 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:45.210+0000 7f0b837fe700 1 -- 172.21.15.134:0/3238010291 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0b9001c320 con 0x7f0b9c0f51a0 2024-01-21T11:44:45.211 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:44:45.211 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:44:45.212 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:45.212+0000 7f0b9bfff700 1 -- 172.21.15.134:0/3238010291 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f0b8402ff10 msgr2=0x7f0b840323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:44:45.213 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:45.212+0000 7f0b9bfff700 1 --2- 172.21.15.134:0/3238010291 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f0b8402ff10 0x7f0b840323d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f0b8c0096a0 tx=0x7f0b8c006b40).stop 2024-01-21T11:44:45.213 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:45.212+0000 7f0b9bfff700 1 -- 172.21.15.134:0/3238010291 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b9c0f51a0 msgr2=0x7f0b9c109200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:44:45.213 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:45.212+0000 7f0b9bfff700 1 --2- 172.21.15.134:0/3238010291 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b9c0f51a0 0x7f0b9c109200 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f0b90009830 tx=0x7f0b9000b040).stop 2024-01-21T11:44:45.213 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:45.212+0000 7f0b9bfff700 1 -- 172.21.15.134:0/3238010291 shutdown_connections 2024-01-21T11:44:45.213 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:45.212+0000 7f0b9bfff700 1 --2- 172.21.15.134:0/3238010291 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f0b8402ff10 0x7f0b840323d0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:44:45.213 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:45.212+0000 7f0b9bfff700 1 --2- 172.21.15.134:0/3238010291 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b9c0f51a0 0x7f0b9c109200 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:44:45.213 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:45.212+0000 7f0b9bfff700 1 -- 172.21.15.134:0/3238010291 >> 172.21.15.134:0/3238010291 conn(0x7f0b9c0f09a0 msgr2=0x7f0b9c0f13e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:44:45.213 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:45.212+0000 7f0b9bfff700 1 -- 172.21.15.134:0/3238010291 shutdown_connections 2024-01-21T11:44:45.213 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:45.212+0000 7f0b9bfff700 1 -- 172.21.15.134:0/3238010291 wait complete. 2024-01-21T11:44:45.213 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:44:45.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:44:45 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:44:43.827108+0000 mgr.smithi029.blhxxk (mgr.14164) 82 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:44:45.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:44:45 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:44:45.211521+0000 mon.smithi029 (mon.0) 169 : audit [DBG] from='client.? 172.21.15.134:0/3238010291' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:44:46.612 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:44:46.612 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:44:47.773 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:47.771+0000 7f8c7bcf0700 1 -- 172.21.15.134:0/1545190476 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c740f5410 msgr2=0x7f8c740f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:44:47.773 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:47.771+0000 7f8c7bcf0700 1 --2- 172.21.15.134:0/1545190476 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c740f5410 0x7f8c740f5830 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f8c64009fb0 tx=0x7f8c64009b30).stop 2024-01-21T11:44:47.773 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:47.771+0000 7f8c7bcf0700 1 -- 172.21.15.134:0/1545190476 shutdown_connections 2024-01-21T11:44:47.773 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:47.771+0000 7f8c7bcf0700 1 --2- 172.21.15.134:0/1545190476 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c740f5410 0x7f8c740f5830 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:44:47.773 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:47.771+0000 7f8c7bcf0700 1 -- 172.21.15.134:0/1545190476 >> 172.21.15.134:0/1545190476 conn(0x7f8c740f0b70 msgr2=0x7f8c740f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:44:47.774 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:47.771+0000 7f8c7bcf0700 1 -- 172.21.15.134:0/1545190476 shutdown_connections 2024-01-21T11:44:47.774 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:47.771+0000 7f8c7bcf0700 1 -- 172.21.15.134:0/1545190476 wait complete. 2024-01-21T11:44:47.774 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:47.772+0000 7f8c7bcf0700 1 Processor -- start 2024-01-21T11:44:47.774 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:47.772+0000 7f8c7bcf0700 1 -- start start 2024-01-21T11:44:47.774 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:47.772+0000 7f8c7bcf0700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c740f5410 0x7f8c74109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:44:47.774 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:47.772+0000 7f8c7bcf0700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c74060240 con 0x7f8c740f5410 2024-01-21T11:44:47.774 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:47.772+0000 7f8c79a8c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c740f5410 0x7f8c74109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:44:47.774 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:47.772+0000 7f8c79a8c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c740f5410 0x7f8c74109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:34486/0 (socket says 172.21.15.134:34486) 2024-01-21T11:44:47.774 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:47.772+0000 7f8c79a8c700 1 -- 172.21.15.134:0/2680361715 learned_addr learned my addr 172.21.15.134:0/2680361715 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:44:47.774 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:47.772+0000 7f8c79a8c700 1 -- 172.21.15.134:0/2680361715 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c64005040 con 0x7f8c740f5410 2024-01-21T11:44:47.774 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:47.773+0000 7f8c79a8c700 1 --2- 172.21.15.134:0/2680361715 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c740f5410 0x7f8c74109430 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f8c64013510 tx=0x7f8c6400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:44:47.774 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:47.773+0000 7f8c6a7fc700 1 -- 172.21.15.134:0/2680361715 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8c64009320 con 0x7f8c740f5410 2024-01-21T11:44:47.775 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:47.773+0000 7f8c6a7fc700 1 -- 172.21.15.134:0/2680361715 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8c64013f00 con 0x7f8c740f5410 2024-01-21T11:44:47.775 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:47.773+0000 7f8c7bcf0700 1 -- 172.21.15.134:0/2680361715 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c74060440 con 0x7f8c740f5410 2024-01-21T11:44:47.775 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:47.773+0000 7f8c6a7fc700 1 -- 172.21.15.134:0/2680361715 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8c64014d90 con 0x7f8c740f5410 2024-01-21T11:44:47.775 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:47.773+0000 7f8c7bcf0700 1 -- 172.21.15.134:0/2680361715 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c74109ae0 con 0x7f8c740f5410 2024-01-21T11:44:47.776 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:47.774+0000 7f8c6a7fc700 1 -- 172.21.15.134:0/2680361715 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8c640095f0 con 0x7f8c740f5410 2024-01-21T11:44:47.776 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:47.774+0000 7f8c7bcf0700 1 -- 172.21.15.134:0/2680361715 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c74045ad0 con 0x7f8c740f5410 2024-01-21T11:44:47.776 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:47.774+0000 7f8c6a7fc700 1 --2- 172.21.15.134:0/2680361715 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f8c6002ff60 0x7f8c60032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:44:47.776 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:47.774+0000 7f8c6a7fc700 1 -- 172.21.15.134:0/2680361715 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8c64041000 con 0x7f8c740f5410 2024-01-21T11:44:47.776 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:47.774+0000 7f8c7928b700 1 --2- 172.21.15.134:0/2680361715 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f8c6002ff60 0x7f8c60032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:44:47.776 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:47.775+0000 7f8c7928b700 1 --2- 172.21.15.134:0/2680361715 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f8c6002ff60 0x7f8c60032420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f8c700096a0 tx=0x7f8c70006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:44:47.779 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:47.779+0000 7f8c6a7fc700 1 -- 172.21.15.134:0/2680361715 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8c64014420 con 0x7f8c740f5410 2024-01-21T11:44:47.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:44:47 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:44:45.827382+0000 mgr.smithi029.blhxxk (mgr.14164) 83 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:44:47.985 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:47.984+0000 7f8c7bcf0700 1 -- 172.21.15.134:0/2680361715 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8c740fae60 con 0x7f8c740f5410 2024-01-21T11:44:47.985 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:47.985+0000 7f8c6a7fc700 1 -- 172.21.15.134:0/2680361715 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8c6401d7f0 con 0x7f8c740f5410 2024-01-21T11:44:47.986 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:44:47.986 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:44:47.987 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:47.987+0000 7f8c7bcf0700 1 -- 172.21.15.134:0/2680361715 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f8c6002ff60 msgr2=0x7f8c60032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:44:47.987 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:47.987+0000 7f8c7bcf0700 1 --2- 172.21.15.134:0/2680361715 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f8c6002ff60 0x7f8c60032420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f8c700096a0 tx=0x7f8c70006b40).stop 2024-01-21T11:44:47.987 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:47.987+0000 7f8c7bcf0700 1 -- 172.21.15.134:0/2680361715 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c740f5410 msgr2=0x7f8c74109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:44:47.987 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:47.987+0000 7f8c7bcf0700 1 --2- 172.21.15.134:0/2680361715 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c740f5410 0x7f8c74109430 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f8c64013510 tx=0x7f8c6400b040).stop 2024-01-21T11:44:47.987 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:47.987+0000 7f8c7bcf0700 1 -- 172.21.15.134:0/2680361715 shutdown_connections 2024-01-21T11:44:47.987 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:47.987+0000 7f8c7bcf0700 1 --2- 172.21.15.134:0/2680361715 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f8c6002ff60 0x7f8c60032420 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:44:47.987 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:47.987+0000 7f8c7bcf0700 1 --2- 172.21.15.134:0/2680361715 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c740f5410 0x7f8c74109430 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:44:47.987 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:47.987+0000 7f8c7bcf0700 1 -- 172.21.15.134:0/2680361715 >> 172.21.15.134:0/2680361715 conn(0x7f8c740f0b70 msgr2=0x7f8c740f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:44:47.987 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:47.987+0000 7f8c7bcf0700 1 -- 172.21.15.134:0/2680361715 shutdown_connections 2024-01-21T11:44:47.988 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:47.987+0000 7f8c7bcf0700 1 -- 172.21.15.134:0/2680361715 wait complete. 2024-01-21T11:44:47.988 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:44:48.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:44:48 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:44:47.986390+0000 mon.smithi029 (mon.0) 170 : audit [DBG] from='client.? 172.21.15.134:0/2680361715' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:44:49.394 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:44:49.394 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:44:49.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:44:49 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:44:47.827695+0000 mgr.smithi029.blhxxk (mgr.14164) 84 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:44:50.540 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:50.538+0000 7f9322e23700 1 -- 172.21.15.134:0/77317583 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f931c0f5410 msgr2=0x7f931c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:44:50.540 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:50.538+0000 7f9322e23700 1 --2- 172.21.15.134:0/77317583 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f931c0f5410 0x7f931c0f5830 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f9314009960 tx=0x7f9314004d10).stop 2024-01-21T11:44:50.541 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:50.538+0000 7f9322e23700 1 -- 172.21.15.134:0/77317583 shutdown_connections 2024-01-21T11:44:50.541 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:50.538+0000 7f9322e23700 1 --2- 172.21.15.134:0/77317583 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f931c0f5410 0x7f931c0f5830 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:44:50.541 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:50.538+0000 7f9322e23700 1 -- 172.21.15.134:0/77317583 >> 172.21.15.134:0/77317583 conn(0x7f931c0f0b70 msgr2=0x7f931c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:44:50.541 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:50.538+0000 7f9322e23700 1 -- 172.21.15.134:0/77317583 shutdown_connections 2024-01-21T11:44:50.541 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:50.538+0000 7f9322e23700 1 -- 172.21.15.134:0/77317583 wait complete. 2024-01-21T11:44:50.541 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:50.539+0000 7f9322e23700 1 Processor -- start 2024-01-21T11:44:50.541 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:50.539+0000 7f9322e23700 1 -- start start 2024-01-21T11:44:50.541 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:50.539+0000 7f9322e23700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f931c0f5410 0x7f931c0fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:44:50.541 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:50.539+0000 7f9322e23700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f931c0fb820 con 0x7f931c0f5410 2024-01-21T11:44:50.541 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:50.539+0000 7f9320bbf700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f931c0f5410 0x7f931c0fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:44:50.541 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:50.539+0000 7f9320bbf700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f931c0f5410 0x7f931c0fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:34512/0 (socket says 172.21.15.134:34512) 2024-01-21T11:44:50.541 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:50.539+0000 7f9320bbf700 1 -- 172.21.15.134:0/1373433428 learned_addr learned my addr 172.21.15.134:0/1373433428 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:44:50.541 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:50.539+0000 7f9320bbf700 1 -- 172.21.15.134:0/1373433428 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9314009640 con 0x7f931c0f5410 2024-01-21T11:44:50.541 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:50.540+0000 7f9320bbf700 1 --2- 172.21.15.134:0/1373433428 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f931c0f5410 0x7f931c0fd3a0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f9314004b70 tx=0x7f9314004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:44:50.541 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:50.540+0000 7f93197fa700 1 -- 172.21.15.134:0/1373433428 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9314018070 con 0x7f931c0f5410 2024-01-21T11:44:50.542 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:50.540+0000 7f93197fa700 1 -- 172.21.15.134:0/1373433428 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f931400b870 con 0x7f931c0f5410 2024-01-21T11:44:50.542 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:50.540+0000 7f9322e23700 1 -- 172.21.15.134:0/1373433428 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f931c0fb9c0 con 0x7f931c0f5410 2024-01-21T11:44:50.542 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:50.540+0000 7f93197fa700 1 -- 172.21.15.134:0/1373433428 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f931401c560 con 0x7f931c0f5410 2024-01-21T11:44:50.542 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:50.540+0000 7f9322e23700 1 -- 172.21.15.134:0/1373433428 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f931c0fbe50 con 0x7f931c0f5410 2024-01-21T11:44:50.543 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:50.541+0000 7f93197fa700 1 -- 172.21.15.134:0/1373433428 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9314013630 con 0x7f931c0f5410 2024-01-21T11:44:50.543 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:50.541+0000 7f9322e23700 1 -- 172.21.15.134:0/1373433428 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f931c059100 con 0x7f931c0f5410 2024-01-21T11:44:50.543 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:50.541+0000 7f93197fa700 1 --2- 172.21.15.134:0/1373433428 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f930402ff60 0x7f9304032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:44:50.543 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:50.541+0000 7f93197fa700 1 -- 172.21.15.134:0/1373433428 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9314041330 con 0x7f931c0f5410 2024-01-21T11:44:50.543 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:50.541+0000 7f931bfff700 1 --2- 172.21.15.134:0/1373433428 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f930402ff60 0x7f9304032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:44:50.543 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:50.542+0000 7f931bfff700 1 --2- 172.21.15.134:0/1373433428 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f930402ff60 0x7f9304032420 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f93100096a0 tx=0x7f9310006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:44:50.546 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:50.545+0000 7f93197fa700 1 -- 172.21.15.134:0/1373433428 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9314004810 con 0x7f931c0f5410 2024-01-21T11:44:50.752 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:50.752+0000 7f9322e23700 1 -- 172.21.15.134:0/1373433428 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f931c045ad0 con 0x7f931c0f5410 2024-01-21T11:44:50.753 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:50.753+0000 7f93197fa700 1 -- 172.21.15.134:0/1373433428 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9314012380 con 0x7f931c0f5410 2024-01-21T11:44:50.753 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:44:50.753 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:44:50.755 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:50.754+0000 7f9322e23700 1 -- 172.21.15.134:0/1373433428 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f930402ff60 msgr2=0x7f9304032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:44:50.755 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:50.754+0000 7f9322e23700 1 --2- 172.21.15.134:0/1373433428 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f930402ff60 0x7f9304032420 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f93100096a0 tx=0x7f9310006b40).stop 2024-01-21T11:44:50.755 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:50.754+0000 7f9322e23700 1 -- 172.21.15.134:0/1373433428 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f931c0f5410 msgr2=0x7f931c0fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:44:50.755 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:50.754+0000 7f9322e23700 1 --2- 172.21.15.134:0/1373433428 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f931c0f5410 0x7f931c0fd3a0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f9314004b70 tx=0x7f9314004d10).stop 2024-01-21T11:44:50.755 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:50.754+0000 7f9322e23700 1 -- 172.21.15.134:0/1373433428 shutdown_connections 2024-01-21T11:44:50.755 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:50.754+0000 7f9322e23700 1 --2- 172.21.15.134:0/1373433428 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f930402ff60 0x7f9304032420 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:44:50.755 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:50.754+0000 7f9322e23700 1 --2- 172.21.15.134:0/1373433428 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f931c0f5410 0x7f931c0fd3a0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:44:50.755 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:50.754+0000 7f9322e23700 1 -- 172.21.15.134:0/1373433428 >> 172.21.15.134:0/1373433428 conn(0x7f931c0f0b70 msgr2=0x7f931c05fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:44:50.755 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:50.755+0000 7f9322e23700 1 -- 172.21.15.134:0/1373433428 shutdown_connections 2024-01-21T11:44:50.755 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:50.755+0000 7f9322e23700 1 -- 172.21.15.134:0/1373433428 wait complete. 2024-01-21T11:44:50.756 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:44:51.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:44:51 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:44:49.827949+0000 mgr.smithi029.blhxxk (mgr.14164) 85 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:44:51.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:44:51 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:44:50.753723+0000 mon.smithi029 (mon.0) 171 : audit [DBG] from='client.? 172.21.15.134:0/1373433428' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:44:52.151 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:44:52.151 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:44:53.367 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:53.365+0000 7f5bc606e700 1 -- 172.21.15.134:0/4291371611 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5bc00f51a0 msgr2=0x7f5bc00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:44:53.368 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:53.365+0000 7f5bc606e700 1 --2- 172.21.15.134:0/4291371611 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5bc00f51a0 0x7f5bc00f55c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f5bb0009fb0 tx=0x7f5bb0009b30).stop 2024-01-21T11:44:53.368 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:53.365+0000 7f5bc606e700 1 -- 172.21.15.134:0/4291371611 shutdown_connections 2024-01-21T11:44:53.368 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:53.365+0000 7f5bc606e700 1 --2- 172.21.15.134:0/4291371611 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5bc00f51a0 0x7f5bc00f55c0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:44:53.368 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:53.365+0000 7f5bc606e700 1 -- 172.21.15.134:0/4291371611 >> 172.21.15.134:0/4291371611 conn(0x7f5bc00f09a0 msgr2=0x7f5bc00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:44:53.368 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:53.365+0000 7f5bc606e700 1 -- 172.21.15.134:0/4291371611 shutdown_connections 2024-01-21T11:44:53.368 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:53.365+0000 7f5bc606e700 1 -- 172.21.15.134:0/4291371611 wait complete. 2024-01-21T11:44:53.368 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:53.366+0000 7f5bc606e700 1 Processor -- start 2024-01-21T11:44:53.368 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:53.366+0000 7f5bc606e700 1 -- start start 2024-01-21T11:44:53.368 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:53.366+0000 7f5bc606e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5bc00f51a0 0x7f5bc01091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:44:53.369 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:53.366+0000 7f5bc606e700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bc0060220 con 0x7f5bc00f51a0 2024-01-21T11:44:53.369 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:53.367+0000 7f5bc506c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5bc00f51a0 0x7f5bc01091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:44:53.369 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:53.367+0000 7f5bc506c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5bc00f51a0 0x7f5bc01091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:34532/0 (socket says 172.21.15.134:34532) 2024-01-21T11:44:53.369 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:53.367+0000 7f5bc506c700 1 -- 172.21.15.134:0/3219156465 learned_addr learned my addr 172.21.15.134:0/3219156465 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:44:53.369 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:53.367+0000 7f5bc506c700 1 -- 172.21.15.134:0/3219156465 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bb0005040 con 0x7f5bc00f51a0 2024-01-21T11:44:53.369 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:53.367+0000 7f5bc506c700 1 --2- 172.21.15.134:0/3219156465 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5bc00f51a0 0x7f5bc01091d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f5bb0009830 tx=0x7f5bb000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:44:53.369 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:53.367+0000 7f5bb5ffb700 1 -- 172.21.15.134:0/3219156465 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5bb0009320 con 0x7f5bc00f51a0 2024-01-21T11:44:53.369 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:53.367+0000 7f5bc606e700 1 -- 172.21.15.134:0/3219156465 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bc0060420 con 0x7f5bc00f51a0 2024-01-21T11:44:53.369 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:53.367+0000 7f5bb5ffb700 1 -- 172.21.15.134:0/3219156465 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5bb00136c0 con 0x7f5bc00f51a0 2024-01-21T11:44:53.369 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:53.367+0000 7f5bb5ffb700 1 -- 172.21.15.134:0/3219156465 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5bb001ee90 con 0x7f5bc00f51a0 2024-01-21T11:44:53.369 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:53.367+0000 7f5bc606e700 1 -- 172.21.15.134:0/3219156465 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bc0109880 con 0x7f5bc00f51a0 2024-01-21T11:44:53.370 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:53.368+0000 7f5bb5ffb700 1 -- 172.21.15.134:0/3219156465 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5bb001e420 con 0x7f5bc00f51a0 2024-01-21T11:44:53.371 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:53.368+0000 7f5bc606e700 1 -- 172.21.15.134:0/3219156465 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bc0059100 con 0x7f5bc00f51a0 2024-01-21T11:44:53.371 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:53.368+0000 7f5bb5ffb700 1 --2- 172.21.15.134:0/3219156465 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f5bac02ff10 0x7f5bac0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:44:53.371 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:53.368+0000 7f5bb5ffb700 1 -- 172.21.15.134:0/3219156465 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5bb0041050 con 0x7f5bc00f51a0 2024-01-21T11:44:53.371 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:53.369+0000 7f5bc486b700 1 --2- 172.21.15.134:0/3219156465 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f5bac02ff10 0x7f5bac0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:44:53.371 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:53.369+0000 7f5bc486b700 1 --2- 172.21.15.134:0/3219156465 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f5bac02ff10 0x7f5bac0323d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f5bbc0096a0 tx=0x7f5bbc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:44:53.373 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:53.373+0000 7f5bb5ffb700 1 -- 172.21.15.134:0/3219156465 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5bb006d2b0 con 0x7f5bc00f51a0 2024-01-21T11:44:53.578 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:53.578+0000 7f5bc606e700 1 -- 172.21.15.134:0/3219156465 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5bc0045ad0 con 0x7f5bc00f51a0 2024-01-21T11:44:53.579 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:53.578+0000 7f5bb5ffb700 1 -- 172.21.15.134:0/3219156465 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5bb002c2d0 con 0x7f5bc00f51a0 2024-01-21T11:44:53.579 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:44:53.579 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:44:53.581 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:53.580+0000 7f5bc606e700 1 -- 172.21.15.134:0/3219156465 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f5bac02ff10 msgr2=0x7f5bac0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:44:53.581 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:53.580+0000 7f5bc606e700 1 --2- 172.21.15.134:0/3219156465 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f5bac02ff10 0x7f5bac0323d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f5bbc0096a0 tx=0x7f5bbc006b40).stop 2024-01-21T11:44:53.581 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:53.580+0000 7f5bc606e700 1 -- 172.21.15.134:0/3219156465 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5bc00f51a0 msgr2=0x7f5bc01091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:44:53.582 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:53.580+0000 7f5bc606e700 1 --2- 172.21.15.134:0/3219156465 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5bc00f51a0 0x7f5bc01091d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f5bb0009830 tx=0x7f5bb000b040).stop 2024-01-21T11:44:53.582 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:53.580+0000 7f5bc606e700 1 -- 172.21.15.134:0/3219156465 shutdown_connections 2024-01-21T11:44:53.582 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:53.580+0000 7f5bc606e700 1 --2- 172.21.15.134:0/3219156465 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f5bac02ff10 0x7f5bac0323d0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:44:53.582 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:53.580+0000 7f5bc606e700 1 --2- 172.21.15.134:0/3219156465 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5bc00f51a0 0x7f5bc01091d0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:44:53.582 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:53.580+0000 7f5bc606e700 1 -- 172.21.15.134:0/3219156465 >> 172.21.15.134:0/3219156465 conn(0x7f5bc00f09a0 msgr2=0x7f5bc00f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:44:53.582 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:53.580+0000 7f5bc606e700 1 -- 172.21.15.134:0/3219156465 shutdown_connections 2024-01-21T11:44:53.582 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:53.580+0000 7f5bc606e700 1 -- 172.21.15.134:0/3219156465 wait complete. 2024-01-21T11:44:53.583 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:44:53.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:44:53 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:44:51.828158+0000 mgr.smithi029.blhxxk (mgr.14164) 86 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:44:54.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:44:54 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:44:53.579462+0000 mon.smithi029 (mon.0) 172 : audit [DBG] from='client.? 172.21.15.134:0/3219156465' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:44:54.974 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:44:54.974 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:44:55.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:44:55 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:44:53.828425+0000 mgr.smithi029.blhxxk (mgr.14164) 87 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:44:56.107 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:56.105+0000 7f09b737b700 1 -- 172.21.15.134:0/1018165 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09b00f5410 msgr2=0x7f09b00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:44:56.107 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:56.105+0000 7f09b737b700 1 --2- 172.21.15.134:0/1018165 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09b00f5410 0x7f09b00f5830 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f09ac009fb0 tx=0x7f09ac009b30).stop 2024-01-21T11:44:56.107 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:56.105+0000 7f09b737b700 1 -- 172.21.15.134:0/1018165 shutdown_connections 2024-01-21T11:44:56.107 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:56.105+0000 7f09b737b700 1 --2- 172.21.15.134:0/1018165 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09b00f5410 0x7f09b00f5830 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:44:56.107 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:56.105+0000 7f09b737b700 1 -- 172.21.15.134:0/1018165 >> 172.21.15.134:0/1018165 conn(0x7f09b00f0b70 msgr2=0x7f09b00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:44:56.107 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:56.105+0000 7f09b737b700 1 -- 172.21.15.134:0/1018165 shutdown_connections 2024-01-21T11:44:56.107 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:56.105+0000 7f09b737b700 1 -- 172.21.15.134:0/1018165 wait complete. 2024-01-21T11:44:56.108 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:56.106+0000 7f09b737b700 1 Processor -- start 2024-01-21T11:44:56.108 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:56.106+0000 7f09b737b700 1 -- start start 2024-01-21T11:44:56.108 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:56.106+0000 7f09b737b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09b00f5410 0x7f09b0106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:44:56.108 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:56.106+0000 7f09b737b700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09b0106c10 con 0x7f09b00f5410 2024-01-21T11:44:56.108 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:56.106+0000 7f09b5117700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09b00f5410 0x7f09b0106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:44:56.108 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:56.106+0000 7f09b5117700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09b00f5410 0x7f09b0106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:40982/0 (socket says 172.21.15.134:40982) 2024-01-21T11:44:56.108 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:56.106+0000 7f09b5117700 1 -- 172.21.15.134:0/2918947335 learned_addr learned my addr 172.21.15.134:0/2918947335 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:44:56.108 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:56.106+0000 7f09b5117700 1 -- 172.21.15.134:0/2918947335 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09ac005040 con 0x7f09b00f5410 2024-01-21T11:44:56.108 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:56.106+0000 7f09b5117700 1 --2- 172.21.15.134:0/2918947335 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09b00f5410 0x7f09b0106720 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f09ac013510 tx=0x7f09ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:44:56.108 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:56.107+0000 7f09a5ffb700 1 -- 172.21.15.134:0/2918947335 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f09ac009320 con 0x7f09b00f5410 2024-01-21T11:44:56.108 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:56.107+0000 7f09b737b700 1 -- 172.21.15.134:0/2918947335 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09b0106e10 con 0x7f09b00f5410 2024-01-21T11:44:56.108 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:56.107+0000 7f09a5ffb700 1 -- 172.21.15.134:0/2918947335 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f09ac013f00 con 0x7f09b00f5410 2024-01-21T11:44:56.108 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:56.107+0000 7f09a5ffb700 1 -- 172.21.15.134:0/2918947335 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f09ac014d90 con 0x7f09b00f5410 2024-01-21T11:44:56.108 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:56.107+0000 7f09b737b700 1 -- 172.21.15.134:0/2918947335 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09b01072a0 con 0x7f09b00f5410 2024-01-21T11:44:56.109 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:56.108+0000 7f09b737b700 1 -- 172.21.15.134:0/2918947335 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09b0045ad0 con 0x7f09b00f5410 2024-01-21T11:44:56.109 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:56.108+0000 7f09a5ffb700 1 -- 172.21.15.134:0/2918947335 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f09ac0095f0 con 0x7f09b00f5410 2024-01-21T11:44:56.110 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:56.108+0000 7f09a5ffb700 1 --2- 172.21.15.134:0/2918947335 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f099c02ff60 0x7f099c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:44:56.110 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:56.108+0000 7f09a5ffb700 1 -- 172.21.15.134:0/2918947335 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f09ac040fe0 con 0x7f09b00f5410 2024-01-21T11:44:56.110 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:56.108+0000 7f09b4916700 1 --2- 172.21.15.134:0/2918947335 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f099c02ff60 0x7f099c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:44:56.110 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:56.108+0000 7f09b4916700 1 --2- 172.21.15.134:0/2918947335 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f099c02ff60 0x7f099c032420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f09a00096a0 tx=0x7f09a0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:44:56.113 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:56.113+0000 7f09a5ffb700 1 -- 172.21.15.134:0/2918947335 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f09ac014420 con 0x7f09b00f5410 2024-01-21T11:44:56.320 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:56.320+0000 7f09b737b700 1 -- 172.21.15.134:0/2918947335 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f09b0108150 con 0x7f09b00f5410 2024-01-21T11:44:56.320 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:56.320+0000 7f09a5ffb700 1 -- 172.21.15.134:0/2918947335 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f09ac010230 con 0x7f09b00f5410 2024-01-21T11:44:56.321 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:44:56.321 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:44:56.322 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:56.322+0000 7f09b737b700 1 -- 172.21.15.134:0/2918947335 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f099c02ff60 msgr2=0x7f099c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:44:56.322 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:56.322+0000 7f09b737b700 1 --2- 172.21.15.134:0/2918947335 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f099c02ff60 0x7f099c032420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f09a00096a0 tx=0x7f09a0006b40).stop 2024-01-21T11:44:56.323 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:56.322+0000 7f09b737b700 1 -- 172.21.15.134:0/2918947335 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09b00f5410 msgr2=0x7f09b0106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:44:56.323 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:56.322+0000 7f09b737b700 1 --2- 172.21.15.134:0/2918947335 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09b00f5410 0x7f09b0106720 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f09ac013510 tx=0x7f09ac00b040).stop 2024-01-21T11:44:56.323 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:56.322+0000 7f09b737b700 1 -- 172.21.15.134:0/2918947335 shutdown_connections 2024-01-21T11:44:56.323 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:56.322+0000 7f09b737b700 1 --2- 172.21.15.134:0/2918947335 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f099c02ff60 0x7f099c032420 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:44:56.323 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:56.322+0000 7f09b737b700 1 --2- 172.21.15.134:0/2918947335 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09b00f5410 0x7f09b0106720 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:44:56.323 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:56.322+0000 7f09b737b700 1 -- 172.21.15.134:0/2918947335 >> 172.21.15.134:0/2918947335 conn(0x7f09b00f0b70 msgr2=0x7f09b005fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:44:56.323 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:56.322+0000 7f09b737b700 1 -- 172.21.15.134:0/2918947335 shutdown_connections 2024-01-21T11:44:56.323 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:56.322+0000 7f09b737b700 1 -- 172.21.15.134:0/2918947335 wait complete. 2024-01-21T11:44:56.323 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:44:56.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:44:56 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:44:56.321467+0000 mon.smithi029 (mon.0) 173 : audit [DBG] from='client.? 172.21.15.134:0/2918947335' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:44:57.708 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:44:57.709 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:44:57.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:44:57 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:44:55.828725+0000 mgr.smithi029.blhxxk (mgr.14164) 88 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:44:58.757 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:58.754+0000 7f9326be8700 1 -- 172.21.15.134:0/3619392667 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93200f47c0 msgr2=0x7f93200f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:44:58.757 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:58.754+0000 7f9326be8700 1 --2- 172.21.15.134:0/3619392667 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93200f47c0 0x7f93200f4be0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f9314009fb0 tx=0x7f9314009b30).stop 2024-01-21T11:44:58.758 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:58.754+0000 7f9326be8700 1 -- 172.21.15.134:0/3619392667 shutdown_connections 2024-01-21T11:44:58.758 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:58.754+0000 7f9326be8700 1 --2- 172.21.15.134:0/3619392667 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93200f47c0 0x7f93200f4be0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:44:58.758 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:58.754+0000 7f9326be8700 1 -- 172.21.15.134:0/3619392667 >> 172.21.15.134:0/3619392667 conn(0x7f93200f02e0 msgr2=0x7f93200f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:44:58.758 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:58.754+0000 7f9326be8700 1 -- 172.21.15.134:0/3619392667 shutdown_connections 2024-01-21T11:44:58.758 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:58.754+0000 7f9326be8700 1 -- 172.21.15.134:0/3619392667 wait complete. 2024-01-21T11:44:58.758 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:58.754+0000 7f9326be8700 1 Processor -- start 2024-01-21T11:44:58.758 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:58.754+0000 7f9326be8700 1 -- start start 2024-01-21T11:44:58.758 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:58.755+0000 7f9326be8700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93200f47c0 0x7f9320106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:44:58.758 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:58.755+0000 7f9326be8700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9320106bf0 con 0x7f93200f47c0 2024-01-21T11:44:58.759 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:58.755+0000 7f9324984700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93200f47c0 0x7f9320106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:44:58.759 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:58.755+0000 7f9324984700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93200f47c0 0x7f9320106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:41002/0 (socket says 172.21.15.134:41002) 2024-01-21T11:44:58.759 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:58.755+0000 7f9324984700 1 -- 172.21.15.134:0/449233074 learned_addr learned my addr 172.21.15.134:0/449233074 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:44:58.759 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:58.755+0000 7f9324984700 1 -- 172.21.15.134:0/449233074 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9314005040 con 0x7f93200f47c0 2024-01-21T11:44:58.759 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:58.755+0000 7f9324984700 1 --2- 172.21.15.134:0/449233074 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93200f47c0 0x7f9320106700 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f9314013510 tx=0x7f931400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:44:58.759 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:58.756+0000 7f931d7fa700 1 -- 172.21.15.134:0/449233074 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9314009320 con 0x7f93200f47c0 2024-01-21T11:44:58.759 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:58.756+0000 7f9326be8700 1 -- 172.21.15.134:0/449233074 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9320106df0 con 0x7f93200f47c0 2024-01-21T11:44:58.759 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:58.756+0000 7f931d7fa700 1 -- 172.21.15.134:0/449233074 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9314013f00 con 0x7f93200f47c0 2024-01-21T11:44:58.759 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:58.756+0000 7f931d7fa700 1 -- 172.21.15.134:0/449233074 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9314014d90 con 0x7f93200f47c0 2024-01-21T11:44:58.759 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:58.756+0000 7f9326be8700 1 -- 172.21.15.134:0/449233074 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9320107280 con 0x7f93200f47c0 2024-01-21T11:44:58.759 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:58.756+0000 7f931d7fa700 1 -- 172.21.15.134:0/449233074 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9314014420 con 0x7f93200f47c0 2024-01-21T11:44:58.760 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:58.756+0000 7f9326be8700 1 -- 172.21.15.134:0/449233074 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9320045ad0 con 0x7f93200f47c0 2024-01-21T11:44:58.760 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:58.757+0000 7f931d7fa700 1 --2- 172.21.15.134:0/449233074 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f931002ffb0 0x7f9310032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:44:58.760 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:58.757+0000 7f931d7fa700 1 -- 172.21.15.134:0/449233074 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9314041090 con 0x7f93200f47c0 2024-01-21T11:44:58.760 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:58.757+0000 7f931ffff700 1 --2- 172.21.15.134:0/449233074 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f931002ffb0 0x7f9310032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:44:58.760 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:58.757+0000 7f931ffff700 1 --2- 172.21.15.134:0/449233074 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f931002ffb0 0x7f9310032470 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f930c0096a0 tx=0x7f930c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:44:58.761 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:58.761+0000 7f931d7fa700 1 -- 172.21.15.134:0/449233074 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f931406d2b0 con 0x7f93200f47c0 2024-01-21T11:44:58.968 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:58.968+0000 7f9326be8700 1 -- 172.21.15.134:0/449233074 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9320108130 con 0x7f93200f47c0 2024-01-21T11:44:58.968 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:58.968+0000 7f931d7fa700 1 -- 172.21.15.134:0/449233074 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f93140095f0 con 0x7f93200f47c0 2024-01-21T11:44:58.969 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:44:58.969 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:44:58.970 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:58.970+0000 7f9326be8700 1 -- 172.21.15.134:0/449233074 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f931002ffb0 msgr2=0x7f9310032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:44:58.970 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:58.970+0000 7f9326be8700 1 --2- 172.21.15.134:0/449233074 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f931002ffb0 0x7f9310032470 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f930c0096a0 tx=0x7f930c006b40).stop 2024-01-21T11:44:58.970 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:58.970+0000 7f9326be8700 1 -- 172.21.15.134:0/449233074 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93200f47c0 msgr2=0x7f9320106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:44:58.970 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:58.970+0000 7f9326be8700 1 --2- 172.21.15.134:0/449233074 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93200f47c0 0x7f9320106700 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f9314013510 tx=0x7f931400b040).stop 2024-01-21T11:44:58.971 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:58.970+0000 7f9326be8700 1 -- 172.21.15.134:0/449233074 shutdown_connections 2024-01-21T11:44:58.971 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:58.970+0000 7f9326be8700 1 --2- 172.21.15.134:0/449233074 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f931002ffb0 0x7f9310032470 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:44:58.971 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:58.970+0000 7f9326be8700 1 --2- 172.21.15.134:0/449233074 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93200f47c0 0x7f9320106700 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:44:58.971 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:58.970+0000 7f9326be8700 1 -- 172.21.15.134:0/449233074 >> 172.21.15.134:0/449233074 conn(0x7f93200f02e0 msgr2=0x7f93200f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:44:58.971 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:58.970+0000 7f9326be8700 1 -- 172.21.15.134:0/449233074 shutdown_connections 2024-01-21T11:44:58.971 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:44:58.970+0000 7f9326be8700 1 -- 172.21.15.134:0/449233074 wait complete. 2024-01-21T11:44:58.971 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:44:59.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:44:59 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:44:57.828981+0000 mgr.smithi029.blhxxk (mgr.14164) 89 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:44:59.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:44:59 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:44:58.969463+0000 mon.smithi029 (mon.0) 174 : audit [DBG] from='client.? 172.21.15.134:0/449233074' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:45:00.340 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:45:00.340 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:45:01.438 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:01.436+0000 7f28648e7700 1 -- 172.21.15.134:0/2832853753 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28600f51a0 msgr2=0x7f28600f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:45:01.438 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:01.436+0000 7f28648e7700 1 --2- 172.21.15.134:0/2832853753 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28600f51a0 0x7f28600f55c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f2850009fb0 tx=0x7f2850009b30).stop 2024-01-21T11:45:01.439 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:01.436+0000 7f28648e7700 1 -- 172.21.15.134:0/2832853753 shutdown_connections 2024-01-21T11:45:01.439 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:01.436+0000 7f28648e7700 1 --2- 172.21.15.134:0/2832853753 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28600f51a0 0x7f28600f55c0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:45:01.439 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:01.436+0000 7f28648e7700 1 -- 172.21.15.134:0/2832853753 >> 172.21.15.134:0/2832853753 conn(0x7f28600f09a0 msgr2=0x7f28600f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:45:01.439 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:01.436+0000 7f28648e7700 1 -- 172.21.15.134:0/2832853753 shutdown_connections 2024-01-21T11:45:01.439 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:01.436+0000 7f28648e7700 1 -- 172.21.15.134:0/2832853753 wait complete. 2024-01-21T11:45:01.439 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:01.437+0000 7f28648e7700 1 Processor -- start 2024-01-21T11:45:01.439 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:01.437+0000 7f28648e7700 1 -- start start 2024-01-21T11:45:01.439 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:01.437+0000 7f28648e7700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28600f51a0 0x7f28601053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:45:01.439 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:01.437+0000 7f28648e7700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28601058a0 con 0x7f28600f51a0 2024-01-21T11:45:01.439 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:01.437+0000 7f285ed9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28600f51a0 0x7f28601053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:45:01.439 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:01.437+0000 7f285ed9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28600f51a0 0x7f28601053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:41028/0 (socket says 172.21.15.134:41028) 2024-01-21T11:45:01.439 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:01.437+0000 7f285ed9d700 1 -- 172.21.15.134:0/3221565011 learned_addr learned my addr 172.21.15.134:0/3221565011 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:45:01.439 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:01.438+0000 7f285ed9d700 1 -- 172.21.15.134:0/3221565011 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2850005040 con 0x7f28600f51a0 2024-01-21T11:45:01.440 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:01.438+0000 7f285ed9d700 1 --2- 172.21.15.134:0/3221565011 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28600f51a0 0x7f28601053b0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f2850013510 tx=0x7f285000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:45:01.440 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:01.438+0000 7f28477fe700 1 -- 172.21.15.134:0/3221565011 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2850009320 con 0x7f28600f51a0 2024-01-21T11:45:01.440 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:01.438+0000 7f28648e7700 1 -- 172.21.15.134:0/3221565011 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2860105aa0 con 0x7f28600f51a0 2024-01-21T11:45:01.440 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:01.438+0000 7f28477fe700 1 -- 172.21.15.134:0/3221565011 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2850013f00 con 0x7f28600f51a0 2024-01-21T11:45:01.440 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:01.438+0000 7f28477fe700 1 -- 172.21.15.134:0/3221565011 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2850014dc0 con 0x7f28600f51a0 2024-01-21T11:45:01.440 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:01.438+0000 7f28648e7700 1 -- 172.21.15.134:0/3221565011 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2860105f30 con 0x7f28600f51a0 2024-01-21T11:45:01.441 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:01.439+0000 7f28477fe700 1 -- 172.21.15.134:0/3221565011 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f28500095f0 con 0x7f28600f51a0 2024-01-21T11:45:01.441 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:01.439+0000 7f28648e7700 1 -- 172.21.15.134:0/3221565011 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2860045ad0 con 0x7f28600f51a0 2024-01-21T11:45:01.441 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:01.439+0000 7f28477fe700 1 --2- 172.21.15.134:0/3221565011 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f284802ff60 0x7f2848032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:45:01.441 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:01.439+0000 7f28477fe700 1 -- 172.21.15.134:0/3221565011 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2850041000 con 0x7f28600f51a0 2024-01-21T11:45:01.441 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:01.439+0000 7f285e59c700 1 --2- 172.21.15.134:0/3221565011 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f284802ff60 0x7f2848032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:45:01.441 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:01.440+0000 7f285e59c700 1 --2- 172.21.15.134:0/3221565011 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f284802ff60 0x7f2848032420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f28540096a0 tx=0x7f2854006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:45:01.444 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:01.444+0000 7f28477fe700 1 -- 172.21.15.134:0/3221565011 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2850014420 con 0x7f28600f51a0 2024-01-21T11:45:01.652 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:01.652+0000 7f28648e7700 1 -- 172.21.15.134:0/3221565011 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2860106de0 con 0x7f28600f51a0 2024-01-21T11:45:01.653 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:01.653+0000 7f28477fe700 1 -- 172.21.15.134:0/3221565011 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2850010230 con 0x7f28600f51a0 2024-01-21T11:45:01.653 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:45:01.653 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:45:01.655 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:01.655+0000 7f28648e7700 1 -- 172.21.15.134:0/3221565011 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f284802ff60 msgr2=0x7f2848032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:45:01.655 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:01.655+0000 7f28648e7700 1 --2- 172.21.15.134:0/3221565011 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f284802ff60 0x7f2848032420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f28540096a0 tx=0x7f2854006b40).stop 2024-01-21T11:45:01.655 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:01.655+0000 7f28648e7700 1 -- 172.21.15.134:0/3221565011 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28600f51a0 msgr2=0x7f28601053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:45:01.655 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:01.655+0000 7f28648e7700 1 --2- 172.21.15.134:0/3221565011 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28600f51a0 0x7f28601053b0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f2850013510 tx=0x7f285000b040).stop 2024-01-21T11:45:01.655 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:01.655+0000 7f28648e7700 1 -- 172.21.15.134:0/3221565011 shutdown_connections 2024-01-21T11:45:01.655 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:01.655+0000 7f28648e7700 1 --2- 172.21.15.134:0/3221565011 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f284802ff60 0x7f2848032420 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:45:01.655 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:01.655+0000 7f28648e7700 1 --2- 172.21.15.134:0/3221565011 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28600f51a0 0x7f28601053b0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:45:01.655 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:01.655+0000 7f28648e7700 1 -- 172.21.15.134:0/3221565011 >> 172.21.15.134:0/3221565011 conn(0x7f28600f09a0 msgr2=0x7f28600f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:45:01.656 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:01.655+0000 7f28648e7700 1 -- 172.21.15.134:0/3221565011 shutdown_connections 2024-01-21T11:45:01.656 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:01.655+0000 7f28648e7700 1 -- 172.21.15.134:0/3221565011 wait complete. 2024-01-21T11:45:01.656 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:45:01.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:45:01 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:44:59.829324+0000 mgr.smithi029.blhxxk (mgr.14164) 90 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:45:02.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:45:02 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:45:01.654150+0000 mon.smithi029 (mon.0) 175 : audit [DBG] from='client.? 172.21.15.134:0/3221565011' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:45:02.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:45:02 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:45:01.939763+0000 mon.smithi029 (mon.0) 176 : audit [INF] from='mgr.14164 172.21.15.29:0/3252518221' entity='mgr.smithi029.blhxxk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi029.blhxxk/mirror_snapshot_schedule"}]: dispatch 2024-01-21T11:45:02.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:45:02 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:45:01.942096+0000 mon.smithi029 (mon.0) 177 : audit [INF] from='mgr.14164 172.21.15.29:0/3252518221' entity='mgr.smithi029.blhxxk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi029.blhxxk/trash_purge_schedule"}]: dispatch 2024-01-21T11:45:03.029 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:45:03.030 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:45:03.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:45:03 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:45:01.829708+0000 mgr.smithi029.blhxxk (mgr.14164) 91 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:45:04.179 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:04.177+0000 7f9ff5449700 1 -- 172.21.15.134:0/4129206774 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ff00f62a0 msgr2=0x7f9ff00f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:45:04.179 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:04.177+0000 7f9ff5449700 1 --2- 172.21.15.134:0/4129206774 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ff00f62a0 0x7f9ff00f66c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7f9fe0009fb0 tx=0x7f9fe0009b30).stop 2024-01-21T11:45:04.179 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:04.177+0000 7f9ff5449700 1 -- 172.21.15.134:0/4129206774 shutdown_connections 2024-01-21T11:45:04.180 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:04.177+0000 7f9ff5449700 1 --2- 172.21.15.134:0/4129206774 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ff00f62a0 0x7f9ff00f66c0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:45:04.180 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:04.177+0000 7f9ff5449700 1 -- 172.21.15.134:0/4129206774 >> 172.21.15.134:0/4129206774 conn(0x7f9ff00f1aa0 msgr2=0x7f9ff00f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:45:04.180 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:04.177+0000 7f9ff5449700 1 -- 172.21.15.134:0/4129206774 shutdown_connections 2024-01-21T11:45:04.180 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:04.177+0000 7f9ff5449700 1 -- 172.21.15.134:0/4129206774 wait complete. 2024-01-21T11:45:04.180 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:04.178+0000 7f9ff5449700 1 Processor -- start 2024-01-21T11:45:04.180 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:04.178+0000 7f9ff5449700 1 -- start start 2024-01-21T11:45:04.180 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:04.178+0000 7f9ff5449700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ff00f62a0 0x7f9ff01064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:45:04.180 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:04.178+0000 7f9ff5449700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ff01069a0 con 0x7f9ff00f62a0 2024-01-21T11:45:04.180 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:04.178+0000 7f9feffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ff00f62a0 0x7f9ff01064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:45:04.180 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:04.178+0000 7f9feffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ff00f62a0 0x7f9ff01064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:41048/0 (socket says 172.21.15.134:41048) 2024-01-21T11:45:04.180 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:04.178+0000 7f9feffff700 1 -- 172.21.15.134:0/3315937920 learned_addr learned my addr 172.21.15.134:0/3315937920 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:45:04.180 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:04.179+0000 7f9feffff700 1 -- 172.21.15.134:0/3315937920 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fe0005040 con 0x7f9ff00f62a0 2024-01-21T11:45:04.181 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:04.179+0000 7f9feffff700 1 --2- 172.21.15.134:0/3315937920 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ff00f62a0 0x7f9ff01064b0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f9fe0013510 tx=0x7f9fe000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:45:04.181 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:04.179+0000 7f9fecff9700 1 -- 172.21.15.134:0/3315937920 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9fe0009320 con 0x7f9ff00f62a0 2024-01-21T11:45:04.181 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:04.179+0000 7f9ff5449700 1 -- 172.21.15.134:0/3315937920 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ff0106ba0 con 0x7f9ff00f62a0 2024-01-21T11:45:04.181 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:04.179+0000 7f9fecff9700 1 -- 172.21.15.134:0/3315937920 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9fe0013f00 con 0x7f9ff00f62a0 2024-01-21T11:45:04.181 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:04.179+0000 7f9fecff9700 1 -- 172.21.15.134:0/3315937920 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9fe0014dc0 con 0x7f9ff00f62a0 2024-01-21T11:45:04.181 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:04.179+0000 7f9ff5449700 1 -- 172.21.15.134:0/3315937920 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ff0107030 con 0x7f9ff00f62a0 2024-01-21T11:45:04.182 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:04.180+0000 7f9fecff9700 1 -- 172.21.15.134:0/3315937920 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9fe0014420 con 0x7f9ff00f62a0 2024-01-21T11:45:04.182 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:04.180+0000 7f9fecff9700 1 --2- 172.21.15.134:0/3315937920 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f9fd802ff60 0x7f9fd8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:45:04.182 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:04.180+0000 7f9ff5449700 1 -- 172.21.15.134:0/3315937920 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ff0045ad0 con 0x7f9ff00f62a0 2024-01-21T11:45:04.182 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:04.180+0000 7f9fecff9700 1 -- 172.21.15.134:0/3315937920 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9fe0041110 con 0x7f9ff00f62a0 2024-01-21T11:45:04.182 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:04.180+0000 7f9fef7fe700 1 --2- 172.21.15.134:0/3315937920 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f9fd802ff60 0x7f9fd8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:45:04.182 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:04.181+0000 7f9fef7fe700 1 --2- 172.21.15.134:0/3315937920 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f9fd802ff60 0x7f9fd8032420 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f9fe40096a0 tx=0x7f9fe4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:45:04.185 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:04.185+0000 7f9fecff9700 1 -- 172.21.15.134:0/3315937920 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9fe00095f0 con 0x7f9ff00f62a0 2024-01-21T11:45:04.390 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:04.390+0000 7f9ff5449700 1 -- 172.21.15.134:0/3315937920 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9ff0107ee0 con 0x7f9ff00f62a0 2024-01-21T11:45:04.391 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:04.391+0000 7f9fecff9700 1 -- 172.21.15.134:0/3315937920 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9fe001cb10 con 0x7f9ff00f62a0 2024-01-21T11:45:04.392 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:45:04.392 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:45:04.393 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:04.393+0000 7f9ff5449700 1 -- 172.21.15.134:0/3315937920 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f9fd802ff60 msgr2=0x7f9fd8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:45:04.393 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:04.393+0000 7f9ff5449700 1 --2- 172.21.15.134:0/3315937920 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f9fd802ff60 0x7f9fd8032420 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f9fe40096a0 tx=0x7f9fe4006b40).stop 2024-01-21T11:45:04.393 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:04.393+0000 7f9ff5449700 1 -- 172.21.15.134:0/3315937920 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ff00f62a0 msgr2=0x7f9ff01064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:45:04.393 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:04.393+0000 7f9ff5449700 1 --2- 172.21.15.134:0/3315937920 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ff00f62a0 0x7f9ff01064b0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f9fe0013510 tx=0x7f9fe000b040).stop 2024-01-21T11:45:04.393 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:04.393+0000 7f9ff5449700 1 -- 172.21.15.134:0/3315937920 shutdown_connections 2024-01-21T11:45:04.393 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:04.393+0000 7f9ff5449700 1 --2- 172.21.15.134:0/3315937920 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f9fd802ff60 0x7f9fd8032420 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:45:04.393 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:04.393+0000 7f9ff5449700 1 --2- 172.21.15.134:0/3315937920 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ff00f62a0 0x7f9ff01064b0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:45:04.393 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:04.393+0000 7f9ff5449700 1 -- 172.21.15.134:0/3315937920 >> 172.21.15.134:0/3315937920 conn(0x7f9ff00f1aa0 msgr2=0x7f9ff00f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:45:04.393 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:04.393+0000 7f9ff5449700 1 -- 172.21.15.134:0/3315937920 shutdown_connections 2024-01-21T11:45:04.394 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:04.393+0000 7f9ff5449700 1 -- 172.21.15.134:0/3315937920 wait complete. 2024-01-21T11:45:04.394 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:45:04.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:45:04 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:45:04.392231+0000 mon.smithi029 (mon.0) 178 : audit [DBG] from='client.? 172.21.15.134:0/3315937920' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:45:05.773 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:45:05.773 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:45:05.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:45:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:45:03.830028+0000 mgr.smithi029.blhxxk (mgr.14164) 92 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:45:06.942 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:06.941+0000 7f366759e700 1 -- 172.21.15.134:0/3681742073 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36680f62a0 msgr2=0x7f36680f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:45:06.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:06.941+0000 7f366759e700 1 --2- 172.21.15.134:0/3681742073 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36680f62a0 0x7f36680f66c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f365c009fb0 tx=0x7f365c009b30).stop 2024-01-21T11:45:06.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:06.941+0000 7f366759e700 1 -- 172.21.15.134:0/3681742073 shutdown_connections 2024-01-21T11:45:06.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:06.941+0000 7f366759e700 1 --2- 172.21.15.134:0/3681742073 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36680f62a0 0x7f36680f66c0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:45:06.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:06.941+0000 7f366759e700 1 -- 172.21.15.134:0/3681742073 >> 172.21.15.134:0/3681742073 conn(0x7f36680f1aa0 msgr2=0x7f36680f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:45:06.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:06.941+0000 7f366759e700 1 -- 172.21.15.134:0/3681742073 shutdown_connections 2024-01-21T11:45:06.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:06.941+0000 7f366759e700 1 -- 172.21.15.134:0/3681742073 wait complete. 2024-01-21T11:45:06.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:06.941+0000 7f366759e700 1 Processor -- start 2024-01-21T11:45:06.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:06.941+0000 7f366759e700 1 -- start start 2024-01-21T11:45:06.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:06.941+0000 7f366759e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36680f62a0 0x7f36681064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:45:06.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:06.941+0000 7f366759e700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36681069a0 con 0x7f36680f62a0 2024-01-21T11:45:06.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:06.942+0000 7f366659c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36680f62a0 0x7f36681064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:45:06.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:06.942+0000 7f366659c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36680f62a0 0x7f36681064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:44680/0 (socket says 172.21.15.134:44680) 2024-01-21T11:45:06.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:06.942+0000 7f366659c700 1 -- 172.21.15.134:0/225173327 learned_addr learned my addr 172.21.15.134:0/225173327 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:45:06.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:06.942+0000 7f366659c700 1 -- 172.21.15.134:0/225173327 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f365c005040 con 0x7f36680f62a0 2024-01-21T11:45:06.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:06.942+0000 7f366659c700 1 --2- 172.21.15.134:0/225173327 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36680f62a0 0x7f36681064b0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f365c009830 tx=0x7f365c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:45:06.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:06.942+0000 7f3656ffd700 1 -- 172.21.15.134:0/225173327 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f365c009320 con 0x7f36680f62a0 2024-01-21T11:45:06.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:06.942+0000 7f3656ffd700 1 -- 172.21.15.134:0/225173327 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f365c0136c0 con 0x7f36680f62a0 2024-01-21T11:45:06.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:06.942+0000 7f366759e700 1 -- 172.21.15.134:0/225173327 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3668106ba0 con 0x7f36680f62a0 2024-01-21T11:45:06.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:06.942+0000 7f3656ffd700 1 -- 172.21.15.134:0/225173327 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f365c01ee90 con 0x7f36680f62a0 2024-01-21T11:45:06.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:06.942+0000 7f366759e700 1 -- 172.21.15.134:0/225173327 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3668107030 con 0x7f36680f62a0 2024-01-21T11:45:06.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:06.943+0000 7f3656ffd700 1 -- 172.21.15.134:0/225173327 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f365c01e420 con 0x7f36680f62a0 2024-01-21T11:45:06.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:06.943+0000 7f366759e700 1 -- 172.21.15.134:0/225173327 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3668045ad0 con 0x7f36680f62a0 2024-01-21T11:45:06.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:06.943+0000 7f3656ffd700 1 --2- 172.21.15.134:0/225173327 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f365002ff10 0x7f36500323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:45:06.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:06.943+0000 7f3656ffd700 1 -- 172.21.15.134:0/225173327 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f365c041010 con 0x7f36680f62a0 2024-01-21T11:45:06.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:06.943+0000 7f3665d9b700 1 --2- 172.21.15.134:0/225173327 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f365002ff10 0x7f36500323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:45:06.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:06.944+0000 7f3665d9b700 1 --2- 172.21.15.134:0/225173327 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f365002ff10 0x7f36500323d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f36580096a0 tx=0x7f3658006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:45:06.948 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:06.948+0000 7f3656ffd700 1 -- 172.21.15.134:0/225173327 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f365c0095f0 con 0x7f36680f62a0 2024-01-21T11:45:07.155 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:07.155+0000 7f366759e700 1 -- 172.21.15.134:0/225173327 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3668107ee0 con 0x7f36680f62a0 2024-01-21T11:45:07.155 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:07.155+0000 7f3656ffd700 1 -- 172.21.15.134:0/225173327 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f365c01cb10 con 0x7f36680f62a0 2024-01-21T11:45:07.156 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:45:07.156 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:45:07.157 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:07.157+0000 7f366759e700 1 -- 172.21.15.134:0/225173327 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f365002ff10 msgr2=0x7f36500323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:45:07.157 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:07.157+0000 7f366759e700 1 --2- 172.21.15.134:0/225173327 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f365002ff10 0x7f36500323d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f36580096a0 tx=0x7f3658006b40).stop 2024-01-21T11:45:07.157 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:07.157+0000 7f366759e700 1 -- 172.21.15.134:0/225173327 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36680f62a0 msgr2=0x7f36681064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:45:07.157 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:07.157+0000 7f366759e700 1 --2- 172.21.15.134:0/225173327 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36680f62a0 0x7f36681064b0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f365c009830 tx=0x7f365c00b040).stop 2024-01-21T11:45:07.157 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:07.157+0000 7f366759e700 1 -- 172.21.15.134:0/225173327 shutdown_connections 2024-01-21T11:45:07.157 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:07.157+0000 7f366759e700 1 --2- 172.21.15.134:0/225173327 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f365002ff10 0x7f36500323d0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:45:07.158 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:07.157+0000 7f366759e700 1 --2- 172.21.15.134:0/225173327 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36680f62a0 0x7f36681064b0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:45:07.158 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:07.157+0000 7f366759e700 1 -- 172.21.15.134:0/225173327 >> 172.21.15.134:0/225173327 conn(0x7f36680f1aa0 msgr2=0x7f36680f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:45:07.158 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:07.157+0000 7f366759e700 1 -- 172.21.15.134:0/225173327 shutdown_connections 2024-01-21T11:45:07.158 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:07.157+0000 7f366759e700 1 -- 172.21.15.134:0/225173327 wait complete. 2024-01-21T11:45:07.158 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:45:07.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:45:07 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:45:05.830382+0000 mgr.smithi029.blhxxk (mgr.14164) 93 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:45:07.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:45:07 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:45:07.156628+0000 mon.smithi029 (mon.0) 179 : audit [DBG] from='client.? 172.21.15.134:0/225173327' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:45:08.519 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:45:08.519 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:45:09.650 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:09.648+0000 7fb364f88700 1 -- 172.21.15.134:0/2904492120 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3600f4740 msgr2=0x7fb3600f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:45:09.650 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:09.648+0000 7fb364f88700 1 --2- 172.21.15.134:0/2904492120 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3600f4740 0x7fb3600f4b60 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7fb354009fb0 tx=0x7fb354009b30).stop 2024-01-21T11:45:09.650 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:09.648+0000 7fb364f88700 1 -- 172.21.15.134:0/2904492120 shutdown_connections 2024-01-21T11:45:09.650 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:09.648+0000 7fb364f88700 1 --2- 172.21.15.134:0/2904492120 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3600f4740 0x7fb3600f4b60 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:45:09.650 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:09.648+0000 7fb364f88700 1 -- 172.21.15.134:0/2904492120 >> 172.21.15.134:0/2904492120 conn(0x7fb3600f0260 msgr2=0x7fb3600f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:45:09.650 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:09.648+0000 7fb364f88700 1 -- 172.21.15.134:0/2904492120 shutdown_connections 2024-01-21T11:45:09.651 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:09.648+0000 7fb364f88700 1 -- 172.21.15.134:0/2904492120 wait complete. 2024-01-21T11:45:09.651 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:09.649+0000 7fb364f88700 1 Processor -- start 2024-01-21T11:45:09.651 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:09.649+0000 7fb364f88700 1 -- start start 2024-01-21T11:45:09.651 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:09.649+0000 7fb364f88700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3600f4740 0x7fb3601066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:45:09.651 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:09.649+0000 7fb364f88700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb360106bc0 con 0x7fb3600f4740 2024-01-21T11:45:09.651 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:09.650+0000 7fb35e59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3600f4740 0x7fb3601066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:45:09.651 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:09.650+0000 7fb35e59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3600f4740 0x7fb3601066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:44706/0 (socket says 172.21.15.134:44706) 2024-01-21T11:45:09.651 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:09.650+0000 7fb35e59c700 1 -- 172.21.15.134:0/1217753377 learned_addr learned my addr 172.21.15.134:0/1217753377 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:45:09.653 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:09.650+0000 7fb35e59c700 1 -- 172.21.15.134:0/1217753377 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb354005040 con 0x7fb3600f4740 2024-01-21T11:45:09.653 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:09.650+0000 7fb35e59c700 1 --2- 172.21.15.134:0/1217753377 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3600f4740 0x7fb3601066d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7fb354013510 tx=0x7fb35400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:45:09.654 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:09.650+0000 7fb34effd700 1 -- 172.21.15.134:0/1217753377 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb354009320 con 0x7fb3600f4740 2024-01-21T11:45:09.654 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:09.650+0000 7fb364f88700 1 -- 172.21.15.134:0/1217753377 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb360106dc0 con 0x7fb3600f4740 2024-01-21T11:45:09.654 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:09.650+0000 7fb34effd700 1 -- 172.21.15.134:0/1217753377 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb354013f00 con 0x7fb3600f4740 2024-01-21T11:45:09.654 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:09.650+0000 7fb34effd700 1 -- 172.21.15.134:0/1217753377 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb354014d90 con 0x7fb3600f4740 2024-01-21T11:45:09.654 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:09.650+0000 7fb364f88700 1 -- 172.21.15.134:0/1217753377 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb360107250 con 0x7fb3600f4740 2024-01-21T11:45:09.654 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:09.651+0000 7fb34effd700 1 -- 172.21.15.134:0/1217753377 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb3540095f0 con 0x7fb3600f4740 2024-01-21T11:45:09.654 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:09.651+0000 7fb364f88700 1 -- 172.21.15.134:0/1217753377 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb360045ad0 con 0x7fb3600f4740 2024-01-21T11:45:09.654 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:09.651+0000 7fb34effd700 1 --2- 172.21.15.134:0/1217753377 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fb34802ff60 0x7fb348032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:45:09.654 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:09.651+0000 7fb34effd700 1 -- 172.21.15.134:0/1217753377 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb3540410a0 con 0x7fb3600f4740 2024-01-21T11:45:09.654 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:09.652+0000 7fb35dd9b700 1 --2- 172.21.15.134:0/1217753377 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fb34802ff60 0x7fb348032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:45:09.654 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:09.652+0000 7fb35dd9b700 1 --2- 172.21.15.134:0/1217753377 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fb34802ff60 0x7fb348032420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fb3500096a0 tx=0x7fb350006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:45:09.656 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:09.656+0000 7fb34effd700 1 -- 172.21.15.134:0/1217753377 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb35401d7f0 con 0x7fb3600f4740 2024-01-21T11:45:09.862 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:09.862+0000 7fb364f88700 1 -- 172.21.15.134:0/1217753377 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb360108100 con 0x7fb3600f4740 2024-01-21T11:45:09.863 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:09.863+0000 7fb34effd700 1 -- 172.21.15.134:0/1217753377 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb354010230 con 0x7fb3600f4740 2024-01-21T11:45:09.863 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:45:09.863 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:45:09.865 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:09.864+0000 7fb364f88700 1 -- 172.21.15.134:0/1217753377 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fb34802ff60 msgr2=0x7fb348032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:45:09.865 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:09.864+0000 7fb364f88700 1 --2- 172.21.15.134:0/1217753377 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fb34802ff60 0x7fb348032420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fb3500096a0 tx=0x7fb350006b40).stop 2024-01-21T11:45:09.865 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:09.864+0000 7fb364f88700 1 -- 172.21.15.134:0/1217753377 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3600f4740 msgr2=0x7fb3601066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:45:09.865 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:09.864+0000 7fb364f88700 1 --2- 172.21.15.134:0/1217753377 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3600f4740 0x7fb3601066d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7fb354013510 tx=0x7fb35400b040).stop 2024-01-21T11:45:09.865 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:09.864+0000 7fb364f88700 1 -- 172.21.15.134:0/1217753377 shutdown_connections 2024-01-21T11:45:09.865 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:09.864+0000 7fb364f88700 1 --2- 172.21.15.134:0/1217753377 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fb34802ff60 0x7fb348032420 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:45:09.865 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:09.864+0000 7fb364f88700 1 --2- 172.21.15.134:0/1217753377 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3600f4740 0x7fb3601066d0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:45:09.865 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:09.864+0000 7fb364f88700 1 -- 172.21.15.134:0/1217753377 >> 172.21.15.134:0/1217753377 conn(0x7fb3600f0260 msgr2=0x7fb3600f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:45:09.865 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:09.864+0000 7fb364f88700 1 -- 172.21.15.134:0/1217753377 shutdown_connections 2024-01-21T11:45:09.865 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:09.864+0000 7fb364f88700 1 -- 172.21.15.134:0/1217753377 wait complete. 2024-01-21T11:45:09.866 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:45:09.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:45:09 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:45:07.830827+0000 mgr.smithi029.blhxxk (mgr.14164) 94 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:45:10.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:45:10 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:45:09.863664+0000 mon.smithi029 (mon.0) 180 : audit [DBG] from='client.? 172.21.15.134:0/1217753377' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:45:11.249 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:45:11.250 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:45:11.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:45:11 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:45:09.831268+0000 mgr.smithi029.blhxxk (mgr.14164) 95 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:45:12.399 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:12.398+0000 7fc93ba5f700 1 -- 172.21.15.134:0/4244610294 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9340f6480 msgr2=0x7fc9340f68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:45:12.399 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:12.398+0000 7fc93ba5f700 1 --2- 172.21.15.134:0/4244610294 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9340f6480 0x7fc9340f68a0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7fc924009fb0 tx=0x7fc924009b30).stop 2024-01-21T11:45:12.399 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:12.398+0000 7fc93ba5f700 1 -- 172.21.15.134:0/4244610294 shutdown_connections 2024-01-21T11:45:12.400 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:12.398+0000 7fc93ba5f700 1 --2- 172.21.15.134:0/4244610294 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9340f6480 0x7fc9340f68a0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:45:12.400 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:12.398+0000 7fc93ba5f700 1 -- 172.21.15.134:0/4244610294 >> 172.21.15.134:0/4244610294 conn(0x7fc9340f1c00 msgr2=0x7fc9340f4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:45:12.400 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:12.398+0000 7fc93ba5f700 1 -- 172.21.15.134:0/4244610294 shutdown_connections 2024-01-21T11:45:12.400 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:12.398+0000 7fc93ba5f700 1 -- 172.21.15.134:0/4244610294 wait complete. 2024-01-21T11:45:12.400 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:12.398+0000 7fc93ba5f700 1 Processor -- start 2024-01-21T11:45:12.400 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:12.398+0000 7fc93ba5f700 1 -- start start 2024-01-21T11:45:12.400 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:12.398+0000 7fc93ba5f700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9340f6480 0x7fc93406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:45:12.403 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:12.398+0000 7fc93ba5f700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc934069e20 con 0x7fc9340f6480 2024-01-21T11:45:12.403 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:12.399+0000 7fc9397fb700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9340f6480 0x7fc93406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:45:12.403 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:12.399+0000 7fc9397fb700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9340f6480 0x7fc93406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:44722/0 (socket says 172.21.15.134:44722) 2024-01-21T11:45:12.403 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:12.399+0000 7fc9397fb700 1 -- 172.21.15.134:0/3651880901 learned_addr learned my addr 172.21.15.134:0/3651880901 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:45:12.403 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:12.399+0000 7fc9397fb700 1 -- 172.21.15.134:0/3651880901 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc924005040 con 0x7fc9340f6480 2024-01-21T11:45:12.403 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:12.399+0000 7fc9397fb700 1 --2- 172.21.15.134:0/3651880901 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9340f6480 0x7fc93406b9a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7fc924013510 tx=0x7fc92400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:45:12.403 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:12.399+0000 7fc929ffb700 1 -- 172.21.15.134:0/3651880901 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc924009320 con 0x7fc9340f6480 2024-01-21T11:45:12.403 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:12.399+0000 7fc93ba5f700 1 -- 172.21.15.134:0/3651880901 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc934069fc0 con 0x7fc9340f6480 2024-01-21T11:45:12.403 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:12.399+0000 7fc929ffb700 1 -- 172.21.15.134:0/3651880901 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc924013f00 con 0x7fc9340f6480 2024-01-21T11:45:12.403 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:12.399+0000 7fc929ffb700 1 -- 172.21.15.134:0/3651880901 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc924014d90 con 0x7fc9340f6480 2024-01-21T11:45:12.403 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:12.399+0000 7fc93ba5f700 1 -- 172.21.15.134:0/3651880901 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc93406a450 con 0x7fc9340f6480 2024-01-21T11:45:12.404 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:12.400+0000 7fc929ffb700 1 -- 172.21.15.134:0/3651880901 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc9240095f0 con 0x7fc9340f6480 2024-01-21T11:45:12.404 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:12.400+0000 7fc93ba5f700 1 -- 172.21.15.134:0/3651880901 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc934059100 con 0x7fc9340f6480 2024-01-21T11:45:12.404 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:12.401+0000 7fc929ffb700 1 --2- 172.21.15.134:0/3651880901 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fc92002ff60 0x7fc920032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:45:12.404 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:12.401+0000 7fc929ffb700 1 -- 172.21.15.134:0/3651880901 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc924041000 con 0x7fc9340f6480 2024-01-21T11:45:12.404 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:12.401+0000 7fc938ffa700 1 --2- 172.21.15.134:0/3651880901 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fc92002ff60 0x7fc920032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:45:12.405 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:12.401+0000 7fc938ffa700 1 --2- 172.21.15.134:0/3651880901 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fc92002ff60 0x7fc920032420 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fc9300096a0 tx=0x7fc930006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:45:12.405 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:12.405+0000 7fc929ffb700 1 -- 172.21.15.134:0/3651880901 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc92401d7f0 con 0x7fc9340f6480 2024-01-21T11:45:12.611 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:12.611+0000 7fc93ba5f700 1 -- 172.21.15.134:0/3651880901 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc9340fbf40 con 0x7fc9340f6480 2024-01-21T11:45:12.612 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:12.612+0000 7fc929ffb700 1 -- 172.21.15.134:0/3651880901 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc924010230 con 0x7fc9340f6480 2024-01-21T11:45:12.612 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:45:12.612 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:45:12.614 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:12.614+0000 7fc93ba5f700 1 -- 172.21.15.134:0/3651880901 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fc92002ff60 msgr2=0x7fc920032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:45:12.614 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:12.614+0000 7fc93ba5f700 1 --2- 172.21.15.134:0/3651880901 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fc92002ff60 0x7fc920032420 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fc9300096a0 tx=0x7fc930006b40).stop 2024-01-21T11:45:12.614 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:12.614+0000 7fc93ba5f700 1 -- 172.21.15.134:0/3651880901 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9340f6480 msgr2=0x7fc93406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:45:12.614 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:12.614+0000 7fc93ba5f700 1 --2- 172.21.15.134:0/3651880901 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9340f6480 0x7fc93406b9a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7fc924013510 tx=0x7fc92400b040).stop 2024-01-21T11:45:12.614 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:12.614+0000 7fc93ba5f700 1 -- 172.21.15.134:0/3651880901 shutdown_connections 2024-01-21T11:45:12.614 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:12.614+0000 7fc93ba5f700 1 --2- 172.21.15.134:0/3651880901 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fc92002ff60 0x7fc920032420 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:45:12.614 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:12.614+0000 7fc93ba5f700 1 --2- 172.21.15.134:0/3651880901 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9340f6480 0x7fc93406b9a0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:45:12.614 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:12.614+0000 7fc93ba5f700 1 -- 172.21.15.134:0/3651880901 >> 172.21.15.134:0/3651880901 conn(0x7fc9340f1c00 msgr2=0x7fc9340f2720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:45:12.614 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:12.614+0000 7fc93ba5f700 1 -- 172.21.15.134:0/3651880901 shutdown_connections 2024-01-21T11:45:12.615 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:12.614+0000 7fc93ba5f700 1 -- 172.21.15.134:0/3651880901 wait complete. 2024-01-21T11:45:12.615 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:45:13.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:45:13 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:45:11.831563+0000 mgr.smithi029.blhxxk (mgr.14164) 96 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:45:13.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:45:13 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:45:12.613230+0000 mon.smithi029 (mon.0) 181 : audit [DBG] from='client.? 172.21.15.134:0/3651880901' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:45:14.041 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:45:14.042 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:45:15.177 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:15.176+0000 7f00142d3700 1 -- 172.21.15.134:0/2085723855 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f000c0f6510 msgr2=0x7f000c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:45:15.177 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:15.176+0000 7f00142d3700 1 --2- 172.21.15.134:0/2085723855 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f000c0f6510 0x7f000c0f6930 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f0008009fb0 tx=0x7f0008009b30).stop 2024-01-21T11:45:15.177 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:15.176+0000 7f00142d3700 1 -- 172.21.15.134:0/2085723855 shutdown_connections 2024-01-21T11:45:15.177 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:15.176+0000 7f00142d3700 1 --2- 172.21.15.134:0/2085723855 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f000c0f6510 0x7f000c0f6930 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:45:15.177 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:15.176+0000 7f00142d3700 1 -- 172.21.15.134:0/2085723855 >> 172.21.15.134:0/2085723855 conn(0x7f000c0f1c70 msgr2=0x7f000c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:45:15.177 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:15.176+0000 7f00142d3700 1 -- 172.21.15.134:0/2085723855 shutdown_connections 2024-01-21T11:45:15.177 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:15.176+0000 7f00142d3700 1 -- 172.21.15.134:0/2085723855 wait complete. 2024-01-21T11:45:15.177 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:15.176+0000 7f00142d3700 1 Processor -- start 2024-01-21T11:45:15.177 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:15.177+0000 7f00142d3700 1 -- start start 2024-01-21T11:45:15.178 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:15.177+0000 7f00142d3700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f000c0f6510 0x7f000c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:45:15.178 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:15.177+0000 7f00142d3700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f000c106c20 con 0x7f000c0f6510 2024-01-21T11:45:15.179 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:15.177+0000 7f001206f700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f000c0f6510 0x7f000c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:45:15.179 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:15.177+0000 7f001206f700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f000c0f6510 0x7f000c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:46908/0 (socket says 172.21.15.134:46908) 2024-01-21T11:45:15.180 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:15.177+0000 7f001206f700 1 -- 172.21.15.134:0/285748327 learned_addr learned my addr 172.21.15.134:0/285748327 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:45:15.180 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:15.177+0000 7f001206f700 1 -- 172.21.15.134:0/285748327 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0008005040 con 0x7f000c0f6510 2024-01-21T11:45:15.180 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:15.177+0000 7f001206f700 1 --2- 172.21.15.134:0/285748327 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f000c0f6510 0x7f000c106730 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f0008013510 tx=0x7f000800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:45:15.180 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:15.178+0000 7f0002ffd700 1 -- 172.21.15.134:0/285748327 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0008009320 con 0x7f000c0f6510 2024-01-21T11:45:15.180 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:15.178+0000 7f0002ffd700 1 -- 172.21.15.134:0/285748327 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0008013f00 con 0x7f000c0f6510 2024-01-21T11:45:15.180 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:15.178+0000 7f00142d3700 1 -- 172.21.15.134:0/285748327 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f000c106e20 con 0x7f000c0f6510 2024-01-21T11:45:15.180 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:15.178+0000 7f0002ffd700 1 -- 172.21.15.134:0/285748327 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0008014d90 con 0x7f000c0f6510 2024-01-21T11:45:15.180 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:15.178+0000 7f00142d3700 1 -- 172.21.15.134:0/285748327 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f000c1072b0 con 0x7f000c0f6510 2024-01-21T11:45:15.180 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:15.178+0000 7f0002ffd700 1 -- 172.21.15.134:0/285748327 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f00080263a0 con 0x7f000c0f6510 2024-01-21T11:45:15.180 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:15.178+0000 7f0002ffd700 1 --2- 172.21.15.134:0/285748327 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7efff802ff60 0x7efff8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:45:15.180 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:15.179+0000 7f0002ffd700 1 -- 172.21.15.134:0/285748327 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0008041060 con 0x7f000c0f6510 2024-01-21T11:45:15.180 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:15.179+0000 7f00142d3700 1 -- 172.21.15.134:0/285748327 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f000c045ad0 con 0x7f000c0f6510 2024-01-21T11:45:15.180 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:15.179+0000 7f001186e700 1 --2- 172.21.15.134:0/285748327 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7efff802ff60 0x7efff8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:45:15.181 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:15.179+0000 7f001186e700 1 --2- 172.21.15.134:0/285748327 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7efff802ff60 0x7efff8032420 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7efffc0096a0 tx=0x7efffc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:45:15.183 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:15.183+0000 7f0002ffd700 1 -- 172.21.15.134:0/285748327 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0008012920 con 0x7f000c0f6510 2024-01-21T11:45:15.391 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:15.391+0000 7f00142d3700 1 -- 172.21.15.134:0/285748327 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f000c0fc050 con 0x7f000c0f6510 2024-01-21T11:45:15.392 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:15.392+0000 7f0002ffd700 1 -- 172.21.15.134:0/285748327 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f000802b020 con 0x7f000c0f6510 2024-01-21T11:45:15.392 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:45:15.392 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:45:15.394 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:15.393+0000 7f00142d3700 1 -- 172.21.15.134:0/285748327 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7efff802ff60 msgr2=0x7efff8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:45:15.394 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:15.393+0000 7f00142d3700 1 --2- 172.21.15.134:0/285748327 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7efff802ff60 0x7efff8032420 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7efffc0096a0 tx=0x7efffc006b40).stop 2024-01-21T11:45:15.394 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:15.393+0000 7f00142d3700 1 -- 172.21.15.134:0/285748327 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f000c0f6510 msgr2=0x7f000c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:45:15.394 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:15.393+0000 7f00142d3700 1 --2- 172.21.15.134:0/285748327 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f000c0f6510 0x7f000c106730 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f0008013510 tx=0x7f000800b040).stop 2024-01-21T11:45:15.394 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:15.394+0000 7f00142d3700 1 -- 172.21.15.134:0/285748327 shutdown_connections 2024-01-21T11:45:15.394 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:15.394+0000 7f00142d3700 1 --2- 172.21.15.134:0/285748327 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7efff802ff60 0x7efff8032420 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:45:15.394 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:15.394+0000 7f00142d3700 1 --2- 172.21.15.134:0/285748327 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f000c0f6510 0x7f000c106730 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:45:15.394 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:15.394+0000 7f00142d3700 1 -- 172.21.15.134:0/285748327 >> 172.21.15.134:0/285748327 conn(0x7f000c0f1c70 msgr2=0x7f000c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:45:15.394 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:15.394+0000 7f00142d3700 1 -- 172.21.15.134:0/285748327 shutdown_connections 2024-01-21T11:45:15.394 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:15.394+0000 7f00142d3700 1 -- 172.21.15.134:0/285748327 wait complete. 2024-01-21T11:45:15.395 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:45:15.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:45:15 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:45:13.831944+0000 mgr.smithi029.blhxxk (mgr.14164) 97 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:45:15.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:45:15 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:45:15.392943+0000 mon.smithi029 (mon.0) 182 : audit [DBG] from='client.? 172.21.15.134:0/285748327' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:45:16.819 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:45:16.820 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:45:17.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:45:17 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:45:15.832323+0000 mgr.smithi029.blhxxk (mgr.14164) 98 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:45:17.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:17.943+0000 7fee664f9700 1 -- 172.21.15.134:0/1663022510 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee600f51a0 msgr2=0x7fee600f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:45:17.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:17.943+0000 7fee664f9700 1 --2- 172.21.15.134:0/1663022510 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee600f51a0 0x7fee600f55c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7fee5c009fb0 tx=0x7fee5c009b30).stop 2024-01-21T11:45:17.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:17.943+0000 7fee664f9700 1 -- 172.21.15.134:0/1663022510 shutdown_connections 2024-01-21T11:45:17.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:17.943+0000 7fee664f9700 1 --2- 172.21.15.134:0/1663022510 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee600f51a0 0x7fee600f55c0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:45:17.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:17.943+0000 7fee664f9700 1 -- 172.21.15.134:0/1663022510 >> 172.21.15.134:0/1663022510 conn(0x7fee600f09a0 msgr2=0x7fee600f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:45:17.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:17.943+0000 7fee664f9700 1 -- 172.21.15.134:0/1663022510 shutdown_connections 2024-01-21T11:45:17.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:17.943+0000 7fee664f9700 1 -- 172.21.15.134:0/1663022510 wait complete. 2024-01-21T11:45:17.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:17.943+0000 7fee664f9700 1 Processor -- start 2024-01-21T11:45:17.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:17.943+0000 7fee664f9700 1 -- start start 2024-01-21T11:45:17.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:17.943+0000 7fee664f9700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee600f51a0 0x7fee601053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:45:17.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:17.943+0000 7fee664f9700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee601058a0 con 0x7fee600f51a0 2024-01-21T11:45:17.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:17.944+0000 7fee654f7700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee600f51a0 0x7fee601053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:45:17.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:17.944+0000 7fee654f7700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee600f51a0 0x7fee601053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:46926/0 (socket says 172.21.15.134:46926) 2024-01-21T11:45:17.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:17.944+0000 7fee654f7700 1 -- 172.21.15.134:0/2618484589 learned_addr learned my addr 172.21.15.134:0/2618484589 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:45:17.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:17.944+0000 7fee654f7700 1 -- 172.21.15.134:0/2618484589 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee5c005040 con 0x7fee600f51a0 2024-01-21T11:45:17.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:17.944+0000 7fee654f7700 1 --2- 172.21.15.134:0/2618484589 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee600f51a0 0x7fee601053b0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7fee5c013510 tx=0x7fee5c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:45:17.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:17.944+0000 7fee55ffb700 1 -- 172.21.15.134:0/2618484589 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee5c009320 con 0x7fee600f51a0 2024-01-21T11:45:17.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:17.944+0000 7fee664f9700 1 -- 172.21.15.134:0/2618484589 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee60105aa0 con 0x7fee600f51a0 2024-01-21T11:45:17.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:17.944+0000 7fee55ffb700 1 -- 172.21.15.134:0/2618484589 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fee5c013f00 con 0x7fee600f51a0 2024-01-21T11:45:17.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:17.944+0000 7fee55ffb700 1 -- 172.21.15.134:0/2618484589 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee5c014dc0 con 0x7fee600f51a0 2024-01-21T11:45:17.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:17.945+0000 7fee664f9700 1 -- 172.21.15.134:0/2618484589 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee60105f30 con 0x7fee600f51a0 2024-01-21T11:45:17.947 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:17.945+0000 7fee55ffb700 1 -- 172.21.15.134:0/2618484589 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fee5c0095f0 con 0x7fee600f51a0 2024-01-21T11:45:17.948 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:17.945+0000 7fee664f9700 1 -- 172.21.15.134:0/2618484589 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee60045ad0 con 0x7fee600f51a0 2024-01-21T11:45:17.948 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:17.946+0000 7fee55ffb700 1 --2- 172.21.15.134:0/2618484589 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fee4c02ff60 0x7fee4c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:45:17.948 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:17.946+0000 7fee55ffb700 1 -- 172.21.15.134:0/2618484589 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fee5c041040 con 0x7fee600f51a0 2024-01-21T11:45:17.948 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:17.946+0000 7fee64cf6700 1 --2- 172.21.15.134:0/2618484589 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fee4c02ff60 0x7fee4c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:45:17.948 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:17.946+0000 7fee64cf6700 1 --2- 172.21.15.134:0/2618484589 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fee4c02ff60 0x7fee4c032420 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7fee500096a0 tx=0x7fee50006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:45:17.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:17.950+0000 7fee55ffb700 1 -- 172.21.15.134:0/2618484589 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fee5c014420 con 0x7fee600f51a0 2024-01-21T11:45:18.159 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:18.159+0000 7fee664f9700 1 -- 172.21.15.134:0/2618484589 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fee60106de0 con 0x7fee600f51a0 2024-01-21T11:45:18.159 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:18.159+0000 7fee55ffb700 1 -- 172.21.15.134:0/2618484589 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fee5c010230 con 0x7fee600f51a0 2024-01-21T11:45:18.160 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:45:18.160 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:45:18.161 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:18.161+0000 7fee664f9700 1 -- 172.21.15.134:0/2618484589 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fee4c02ff60 msgr2=0x7fee4c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:45:18.161 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:18.161+0000 7fee664f9700 1 --2- 172.21.15.134:0/2618484589 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fee4c02ff60 0x7fee4c032420 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7fee500096a0 tx=0x7fee50006b40).stop 2024-01-21T11:45:18.161 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:18.161+0000 7fee664f9700 1 -- 172.21.15.134:0/2618484589 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee600f51a0 msgr2=0x7fee601053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:45:18.161 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:18.161+0000 7fee664f9700 1 --2- 172.21.15.134:0/2618484589 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee600f51a0 0x7fee601053b0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7fee5c013510 tx=0x7fee5c00b040).stop 2024-01-21T11:45:18.161 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:18.161+0000 7fee664f9700 1 -- 172.21.15.134:0/2618484589 shutdown_connections 2024-01-21T11:45:18.162 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:18.161+0000 7fee664f9700 1 --2- 172.21.15.134:0/2618484589 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fee4c02ff60 0x7fee4c032420 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:45:18.162 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:18.161+0000 7fee664f9700 1 --2- 172.21.15.134:0/2618484589 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee600f51a0 0x7fee601053b0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:45:18.162 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:18.161+0000 7fee664f9700 1 -- 172.21.15.134:0/2618484589 >> 172.21.15.134:0/2618484589 conn(0x7fee600f09a0 msgr2=0x7fee600f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:45:18.162 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:18.161+0000 7fee664f9700 1 -- 172.21.15.134:0/2618484589 shutdown_connections 2024-01-21T11:45:18.162 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:18.161+0000 7fee664f9700 1 -- 172.21.15.134:0/2618484589 wait complete. 2024-01-21T11:45:18.162 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:45:18.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:45:18 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:45:18.160466+0000 mon.smithi029 (mon.0) 183 : audit [DBG] from='client.? 172.21.15.134:0/2618484589' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:45:19.548 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:45:19.549 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:45:19.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:45:19 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:45:17.832683+0000 mgr.smithi029.blhxxk (mgr.14164) 99 : cluster [DBG] pgmap v92: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:45:20.703 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:20.702+0000 7f9cc688f700 1 -- 172.21.15.134:0/2869990065 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cc00f6510 msgr2=0x7f9cc00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:45:20.703 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:20.702+0000 7f9cc688f700 1 --2- 172.21.15.134:0/2869990065 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cc00f6510 0x7f9cc00f6930 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7f9cb0009fb0 tx=0x7f9cb0009b30).stop 2024-01-21T11:45:20.703 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:20.702+0000 7f9cc688f700 1 -- 172.21.15.134:0/2869990065 shutdown_connections 2024-01-21T11:45:20.703 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:20.702+0000 7f9cc688f700 1 --2- 172.21.15.134:0/2869990065 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cc00f6510 0x7f9cc00f6930 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:45:20.703 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:20.702+0000 7f9cc688f700 1 -- 172.21.15.134:0/2869990065 >> 172.21.15.134:0/2869990065 conn(0x7f9cc00f1c70 msgr2=0x7f9cc00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:45:20.703 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:20.702+0000 7f9cc688f700 1 -- 172.21.15.134:0/2869990065 shutdown_connections 2024-01-21T11:45:20.703 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:20.702+0000 7f9cc688f700 1 -- 172.21.15.134:0/2869990065 wait complete. 2024-01-21T11:45:20.703 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:20.703+0000 7f9cc688f700 1 Processor -- start 2024-01-21T11:45:20.703 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:20.703+0000 7f9cc688f700 1 -- start start 2024-01-21T11:45:20.705 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:20.703+0000 7f9cc688f700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cc00f6510 0x7f9cc0106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:45:20.705 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:20.703+0000 7f9cc688f700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cc0106c20 con 0x7f9cc00f6510 2024-01-21T11:45:20.705 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:20.703+0000 7f9cbffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cc00f6510 0x7f9cc0106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:45:20.705 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:20.703+0000 7f9cbffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cc00f6510 0x7f9cc0106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:46938/0 (socket says 172.21.15.134:46938) 2024-01-21T11:45:20.705 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:20.703+0000 7f9cbffff700 1 -- 172.21.15.134:0/2635472303 learned_addr learned my addr 172.21.15.134:0/2635472303 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:45:20.706 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:20.704+0000 7f9cbffff700 1 -- 172.21.15.134:0/2635472303 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cb0005040 con 0x7f9cc00f6510 2024-01-21T11:45:20.706 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:20.704+0000 7f9cbffff700 1 --2- 172.21.15.134:0/2635472303 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cc00f6510 0x7f9cc0106730 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f9cb0013510 tx=0x7f9cb000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:45:20.706 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:20.704+0000 7f9cbcff9700 1 -- 172.21.15.134:0/2635472303 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9cb0009320 con 0x7f9cc00f6510 2024-01-21T11:45:20.706 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:20.704+0000 7f9cbcff9700 1 -- 172.21.15.134:0/2635472303 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9cb0013f00 con 0x7f9cc00f6510 2024-01-21T11:45:20.706 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:20.704+0000 7f9cc688f700 1 -- 172.21.15.134:0/2635472303 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cc0106e20 con 0x7f9cc00f6510 2024-01-21T11:45:20.706 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:20.704+0000 7f9cbcff9700 1 -- 172.21.15.134:0/2635472303 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9cb0014d90 con 0x7f9cc00f6510 2024-01-21T11:45:20.706 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:20.704+0000 7f9cc688f700 1 -- 172.21.15.134:0/2635472303 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cc01072b0 con 0x7f9cc00f6510 2024-01-21T11:45:20.706 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:20.705+0000 7f9cbcff9700 1 -- 172.21.15.134:0/2635472303 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9cb00263a0 con 0x7f9cc00f6510 2024-01-21T11:45:20.706 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:20.705+0000 7f9cbcff9700 1 --2- 172.21.15.134:0/2635472303 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f9cac02ff60 0x7f9cac032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:45:20.706 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:20.705+0000 7f9cbcff9700 1 -- 172.21.15.134:0/2635472303 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9cb00410f0 con 0x7f9cc00f6510 2024-01-21T11:45:20.706 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:20.705+0000 7f9cc688f700 1 -- 172.21.15.134:0/2635472303 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cc0045ad0 con 0x7f9cc00f6510 2024-01-21T11:45:20.706 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:20.705+0000 7f9cbf7fe700 1 --2- 172.21.15.134:0/2635472303 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f9cac02ff60 0x7f9cac032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:45:20.706 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:20.705+0000 7f9cbf7fe700 1 --2- 172.21.15.134:0/2635472303 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f9cac02ff60 0x7f9cac032420 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f9ca80096a0 tx=0x7f9ca8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:45:20.710 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:20.710+0000 7f9cbcff9700 1 -- 172.21.15.134:0/2635472303 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9cb0012920 con 0x7f9cc00f6510 2024-01-21T11:45:20.918 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:20.918+0000 7f9cc688f700 1 -- 172.21.15.134:0/2635472303 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9cc00fc050 con 0x7f9cc00f6510 2024-01-21T11:45:20.919 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:20.919+0000 7f9cbcff9700 1 -- 172.21.15.134:0/2635472303 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9cb002b020 con 0x7f9cc00f6510 2024-01-21T11:45:20.919 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:45:20.919 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:45:20.920 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:20.920+0000 7f9cc688f700 1 -- 172.21.15.134:0/2635472303 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f9cac02ff60 msgr2=0x7f9cac032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:45:20.920 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:20.920+0000 7f9cc688f700 1 --2- 172.21.15.134:0/2635472303 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f9cac02ff60 0x7f9cac032420 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f9ca80096a0 tx=0x7f9ca8006b40).stop 2024-01-21T11:45:20.921 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:20.920+0000 7f9cc688f700 1 -- 172.21.15.134:0/2635472303 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cc00f6510 msgr2=0x7f9cc0106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:45:20.921 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:20.920+0000 7f9cc688f700 1 --2- 172.21.15.134:0/2635472303 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cc00f6510 0x7f9cc0106730 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f9cb0013510 tx=0x7f9cb000b040).stop 2024-01-21T11:45:20.921 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:20.920+0000 7f9cc688f700 1 -- 172.21.15.134:0/2635472303 shutdown_connections 2024-01-21T11:45:20.921 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:20.920+0000 7f9cc688f700 1 --2- 172.21.15.134:0/2635472303 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f9cac02ff60 0x7f9cac032420 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:45:20.921 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:20.920+0000 7f9cc688f700 1 --2- 172.21.15.134:0/2635472303 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cc00f6510 0x7f9cc0106730 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:45:20.921 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:20.920+0000 7f9cc688f700 1 -- 172.21.15.134:0/2635472303 >> 172.21.15.134:0/2635472303 conn(0x7f9cc00f1c70 msgr2=0x7f9cc00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:45:20.921 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:20.920+0000 7f9cc688f700 1 -- 172.21.15.134:0/2635472303 shutdown_connections 2024-01-21T11:45:20.921 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:20.920+0000 7f9cc688f700 1 -- 172.21.15.134:0/2635472303 wait complete. 2024-01-21T11:45:20.921 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:45:21.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:45:21 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:45:19.833108+0000 mgr.smithi029.blhxxk (mgr.14164) 100 : cluster [DBG] pgmap v93: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:45:21.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:45:21 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:45:20.919860+0000 mon.smithi029 (mon.0) 184 : audit [DBG] from='client.? 172.21.15.134:0/2635472303' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:45:22.309 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:45:22.309 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:45:23.501 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:23.498+0000 7f8107ae9700 1 -- 172.21.15.134:0/4039888814 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81000f5410 msgr2=0x7f81000f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:45:23.501 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:23.498+0000 7f8107ae9700 1 --2- 172.21.15.134:0/4039888814 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81000f5410 0x7f81000f5830 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7f80fc009fb0 tx=0x7f80fc009b30).stop 2024-01-21T11:45:23.501 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:23.498+0000 7f8107ae9700 1 -- 172.21.15.134:0/4039888814 shutdown_connections 2024-01-21T11:45:23.501 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:23.498+0000 7f8107ae9700 1 --2- 172.21.15.134:0/4039888814 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81000f5410 0x7f81000f5830 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:45:23.501 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:23.498+0000 7f8107ae9700 1 -- 172.21.15.134:0/4039888814 >> 172.21.15.134:0/4039888814 conn(0x7f81000f0b70 msgr2=0x7f81000f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:45:23.501 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:23.498+0000 7f8107ae9700 1 -- 172.21.15.134:0/4039888814 shutdown_connections 2024-01-21T11:45:23.501 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:23.498+0000 7f8107ae9700 1 -- 172.21.15.134:0/4039888814 wait complete. 2024-01-21T11:45:23.502 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:23.499+0000 7f8107ae9700 1 Processor -- start 2024-01-21T11:45:23.502 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:23.499+0000 7f8107ae9700 1 -- start start 2024-01-21T11:45:23.502 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:23.499+0000 7f8107ae9700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81000f5410 0x7f81001093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:45:23.502 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:23.499+0000 7f8107ae9700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8100060280 con 0x7f81000f5410 2024-01-21T11:45:23.502 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:23.499+0000 7f8105885700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81000f5410 0x7f81001093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:45:23.502 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:23.499+0000 7f8105885700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81000f5410 0x7f81001093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:46954/0 (socket says 172.21.15.134:46954) 2024-01-21T11:45:23.502 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:23.499+0000 7f8105885700 1 -- 172.21.15.134:0/1289565710 learned_addr learned my addr 172.21.15.134:0/1289565710 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:45:23.502 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:23.499+0000 7f8105885700 1 -- 172.21.15.134:0/1289565710 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80fc005040 con 0x7f81000f5410 2024-01-21T11:45:23.502 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:23.500+0000 7f8105885700 1 --2- 172.21.15.134:0/1289565710 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81000f5410 0x7f81001093f0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f80fc013510 tx=0x7f80fc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:45:23.502 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:23.500+0000 7f80f67fc700 1 -- 172.21.15.134:0/1289565710 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f80fc009320 con 0x7f81000f5410 2024-01-21T11:45:23.502 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:23.500+0000 7f8107ae9700 1 -- 172.21.15.134:0/1289565710 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8100060480 con 0x7f81000f5410 2024-01-21T11:45:23.502 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:23.500+0000 7f80f67fc700 1 -- 172.21.15.134:0/1289565710 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f80fc013f00 con 0x7f81000f5410 2024-01-21T11:45:23.502 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:23.500+0000 7f80f67fc700 1 -- 172.21.15.134:0/1289565710 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f80fc014d90 con 0x7f81000f5410 2024-01-21T11:45:23.502 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:23.500+0000 7f8107ae9700 1 -- 172.21.15.134:0/1289565710 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8100109b20 con 0x7f81000f5410 2024-01-21T11:45:23.502 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:23.501+0000 7f80f67fc700 1 -- 172.21.15.134:0/1289565710 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f80fc0263a0 con 0x7f81000f5410 2024-01-21T11:45:23.503 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:23.501+0000 7f80f67fc700 1 --2- 172.21.15.134:0/1289565710 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f80ec02ff60 0x7f80ec032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:45:23.503 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:23.501+0000 7f80f67fc700 1 -- 172.21.15.134:0/1289565710 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f80fc040ff0 con 0x7f81000f5410 2024-01-21T11:45:23.503 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:23.501+0000 7f8107ae9700 1 -- 172.21.15.134:0/1289565710 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8100045ad0 con 0x7f81000f5410 2024-01-21T11:45:23.503 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:23.501+0000 7f8105084700 1 --2- 172.21.15.134:0/1289565710 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f80ec02ff60 0x7f80ec032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:45:23.503 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:23.501+0000 7f8105084700 1 --2- 172.21.15.134:0/1289565710 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f80ec02ff60 0x7f80ec032420 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f80f00096a0 tx=0x7f80f0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:45:23.506 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:23.506+0000 7f80f67fc700 1 -- 172.21.15.134:0/1289565710 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f80fc012920 con 0x7f81000f5410 2024-01-21T11:45:23.712 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:23.712+0000 7f8107ae9700 1 -- 172.21.15.134:0/1289565710 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f810010a270 con 0x7f81000f5410 2024-01-21T11:45:23.712 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:23.712+0000 7f80f67fc700 1 -- 172.21.15.134:0/1289565710 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f80fc02b020 con 0x7f81000f5410 2024-01-21T11:45:23.713 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:45:23.713 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:45:23.714 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:23.714+0000 7f8107ae9700 1 -- 172.21.15.134:0/1289565710 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f80ec02ff60 msgr2=0x7f80ec032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:45:23.714 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:23.714+0000 7f8107ae9700 1 --2- 172.21.15.134:0/1289565710 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f80ec02ff60 0x7f80ec032420 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f80f00096a0 tx=0x7f80f0006b40).stop 2024-01-21T11:45:23.714 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:23.714+0000 7f8107ae9700 1 -- 172.21.15.134:0/1289565710 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81000f5410 msgr2=0x7f81001093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:45:23.714 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:23.714+0000 7f8107ae9700 1 --2- 172.21.15.134:0/1289565710 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81000f5410 0x7f81001093f0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f80fc013510 tx=0x7f80fc00b040).stop 2024-01-21T11:45:23.714 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:23.714+0000 7f8107ae9700 1 -- 172.21.15.134:0/1289565710 shutdown_connections 2024-01-21T11:45:23.714 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:23.714+0000 7f8107ae9700 1 --2- 172.21.15.134:0/1289565710 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f80ec02ff60 0x7f80ec032420 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:45:23.715 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:23.714+0000 7f8107ae9700 1 --2- 172.21.15.134:0/1289565710 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81000f5410 0x7f81001093f0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:45:23.715 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:23.714+0000 7f8107ae9700 1 -- 172.21.15.134:0/1289565710 >> 172.21.15.134:0/1289565710 conn(0x7f81000f0b70 msgr2=0x7f81000f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:45:23.715 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:23.714+0000 7f8107ae9700 1 -- 172.21.15.134:0/1289565710 shutdown_connections 2024-01-21T11:45:23.715 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:23.714+0000 7f8107ae9700 1 -- 172.21.15.134:0/1289565710 wait complete. 2024-01-21T11:45:23.715 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:45:23.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:45:23 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:45:21.833402+0000 mgr.smithi029.blhxxk (mgr.14164) 101 : cluster [DBG] pgmap v94: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:45:24.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:45:24 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:45:23.713496+0000 mon.smithi029 (mon.0) 185 : audit [DBG] from='client.? 172.21.15.134:0/1289565710' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:45:25.103 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:45:25.104 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:45:25.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:45:25 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:45:23.833690+0000 mgr.smithi029.blhxxk (mgr.14164) 102 : cluster [DBG] pgmap v95: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:45:26.277 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:26.275+0000 7f34a6154700 1 -- 172.21.15.134:0/3555759485 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34a00f51a0 msgr2=0x7f34a00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:45:26.277 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:26.275+0000 7f34a6154700 1 --2- 172.21.15.134:0/3555759485 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34a00f51a0 0x7f34a00f55c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7f349c009fb0 tx=0x7f349c009b30).stop 2024-01-21T11:45:26.277 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:26.275+0000 7f34a6154700 1 -- 172.21.15.134:0/3555759485 shutdown_connections 2024-01-21T11:45:26.278 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:26.275+0000 7f34a6154700 1 --2- 172.21.15.134:0/3555759485 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34a00f51a0 0x7f34a00f55c0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:45:26.278 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:26.275+0000 7f34a6154700 1 -- 172.21.15.134:0/3555759485 >> 172.21.15.134:0/3555759485 conn(0x7f34a00f09a0 msgr2=0x7f34a00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:45:26.278 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:26.275+0000 7f34a6154700 1 -- 172.21.15.134:0/3555759485 shutdown_connections 2024-01-21T11:45:26.278 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:26.275+0000 7f34a6154700 1 -- 172.21.15.134:0/3555759485 wait complete. 2024-01-21T11:45:26.278 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:26.275+0000 7f34a6154700 1 Processor -- start 2024-01-21T11:45:26.278 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:26.275+0000 7f34a6154700 1 -- start start 2024-01-21T11:45:26.278 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:26.276+0000 7f34a6154700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34a00f51a0 0x7f34a01064a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:45:26.278 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:26.276+0000 7f34a6154700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34a0106990 con 0x7f34a00f51a0 2024-01-21T11:45:26.278 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:26.276+0000 7f34a5152700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34a00f51a0 0x7f34a01064a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:45:26.278 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:26.276+0000 7f34a5152700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34a00f51a0 0x7f34a01064a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:54054/0 (socket says 172.21.15.134:54054) 2024-01-21T11:45:26.278 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:26.276+0000 7f34a5152700 1 -- 172.21.15.134:0/2195229006 learned_addr learned my addr 172.21.15.134:0/2195229006 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:45:26.279 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:26.276+0000 7f34a5152700 1 -- 172.21.15.134:0/2195229006 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f349c005040 con 0x7f34a00f51a0 2024-01-21T11:45:26.279 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:26.276+0000 7f34a5152700 1 --2- 172.21.15.134:0/2195229006 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34a00f51a0 0x7f34a01064a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f349c013510 tx=0x7f349c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:45:26.279 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:26.277+0000 7f3495ffb700 1 -- 172.21.15.134:0/2195229006 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f349c009320 con 0x7f34a00f51a0 2024-01-21T11:45:26.279 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:26.277+0000 7f3495ffb700 1 -- 172.21.15.134:0/2195229006 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f349c013f00 con 0x7f34a00f51a0 2024-01-21T11:45:26.279 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:26.277+0000 7f34a6154700 1 -- 172.21.15.134:0/2195229006 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34a0106b90 con 0x7f34a00f51a0 2024-01-21T11:45:26.279 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:26.277+0000 7f3495ffb700 1 -- 172.21.15.134:0/2195229006 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f349c014dc0 con 0x7f34a00f51a0 2024-01-21T11:45:26.279 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:26.277+0000 7f34a6154700 1 -- 172.21.15.134:0/2195229006 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34a0107020 con 0x7f34a00f51a0 2024-01-21T11:45:26.280 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:26.278+0000 7f3495ffb700 1 -- 172.21.15.134:0/2195229006 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f349c0095f0 con 0x7f34a00f51a0 2024-01-21T11:45:26.280 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:26.278+0000 7f34a6154700 1 -- 172.21.15.134:0/2195229006 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34a0045ad0 con 0x7f34a00f51a0 2024-01-21T11:45:26.280 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:26.278+0000 7f3495ffb700 1 --2- 172.21.15.134:0/2195229006 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f348c02ff60 0x7f348c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:45:26.280 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:26.278+0000 7f3495ffb700 1 -- 172.21.15.134:0/2195229006 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f349c041040 con 0x7f34a00f51a0 2024-01-21T11:45:26.280 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:26.278+0000 7f34a4951700 1 --2- 172.21.15.134:0/2195229006 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f348c02ff60 0x7f348c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:45:26.280 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:26.278+0000 7f34a4951700 1 --2- 172.21.15.134:0/2195229006 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f348c02ff60 0x7f348c032420 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f34900096a0 tx=0x7f3490006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:45:26.283 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:26.283+0000 7f3495ffb700 1 -- 172.21.15.134:0/2195229006 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f349c014420 con 0x7f34a00f51a0 2024-01-21T11:45:26.490 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:26.490+0000 7f34a6154700 1 -- 172.21.15.134:0/2195229006 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f34a0107ed0 con 0x7f34a00f51a0 2024-01-21T11:45:26.491 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:26.491+0000 7f3495ffb700 1 -- 172.21.15.134:0/2195229006 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f349c010230 con 0x7f34a00f51a0 2024-01-21T11:45:26.491 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:45:26.491 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:45:26.492 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:26.492+0000 7f34a6154700 1 -- 172.21.15.134:0/2195229006 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f348c02ff60 msgr2=0x7f348c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:45:26.493 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:26.492+0000 7f34a6154700 1 --2- 172.21.15.134:0/2195229006 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f348c02ff60 0x7f348c032420 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f34900096a0 tx=0x7f3490006b40).stop 2024-01-21T11:45:26.493 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:26.492+0000 7f34a6154700 1 -- 172.21.15.134:0/2195229006 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34a00f51a0 msgr2=0x7f34a01064a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:45:26.493 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:26.492+0000 7f34a6154700 1 --2- 172.21.15.134:0/2195229006 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34a00f51a0 0x7f34a01064a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f349c013510 tx=0x7f349c00b040).stop 2024-01-21T11:45:26.493 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:26.492+0000 7f34a6154700 1 -- 172.21.15.134:0/2195229006 shutdown_connections 2024-01-21T11:45:26.493 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:26.492+0000 7f34a6154700 1 --2- 172.21.15.134:0/2195229006 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f348c02ff60 0x7f348c032420 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:45:26.493 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:26.492+0000 7f34a6154700 1 --2- 172.21.15.134:0/2195229006 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34a00f51a0 0x7f34a01064a0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:45:26.493 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:26.492+0000 7f34a6154700 1 -- 172.21.15.134:0/2195229006 >> 172.21.15.134:0/2195229006 conn(0x7f34a00f09a0 msgr2=0x7f34a005fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:45:26.493 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:26.492+0000 7f34a6154700 1 -- 172.21.15.134:0/2195229006 shutdown_connections 2024-01-21T11:45:26.493 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:26.492+0000 7f34a6154700 1 -- 172.21.15.134:0/2195229006 wait complete. 2024-01-21T11:45:26.494 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:45:26.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:45:26 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:45:26.491790+0000 mon.smithi029 (mon.0) 186 : audit [DBG] from='client.? 172.21.15.134:0/2195229006' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:45:27.869 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:45:27.869 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:45:27.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:45:27 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:45:25.833983+0000 mgr.smithi029.blhxxk (mgr.14164) 103 : cluster [DBG] pgmap v96: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:45:29.060 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:29.058+0000 7fda31dc2700 1 -- 172.21.15.134:0/3238779026 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda2c060280 msgr2=0x7fda2c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:45:29.060 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:29.058+0000 7fda31dc2700 1 --2- 172.21.15.134:0/3238779026 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda2c060280 0x7fda2c0f90f0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7fda20009fb0 tx=0x7fda20009b30).stop 2024-01-21T11:45:29.060 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:29.058+0000 7fda31dc2700 1 -- 172.21.15.134:0/3238779026 shutdown_connections 2024-01-21T11:45:29.060 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:29.058+0000 7fda31dc2700 1 --2- 172.21.15.134:0/3238779026 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda2c060280 0x7fda2c0f90f0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:45:29.060 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:29.058+0000 7fda31dc2700 1 -- 172.21.15.134:0/3238779026 >> 172.21.15.134:0/3238779026 conn(0x7fda2c0f0b70 msgr2=0x7fda2c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:45:29.060 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:29.058+0000 7fda31dc2700 1 -- 172.21.15.134:0/3238779026 shutdown_connections 2024-01-21T11:45:29.060 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:29.058+0000 7fda31dc2700 1 -- 172.21.15.134:0/3238779026 wait complete. 2024-01-21T11:45:29.060 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:29.059+0000 7fda31dc2700 1 Processor -- start 2024-01-21T11:45:29.060 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:29.059+0000 7fda31dc2700 1 -- start start 2024-01-21T11:45:29.060 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:29.059+0000 7fda31dc2700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda2c060280 0x7fda2c1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:45:29.061 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:29.059+0000 7fda31dc2700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda2c106bd0 con 0x7fda2c060280 2024-01-21T11:45:29.061 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:29.059+0000 7fda2b7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda2c060280 0x7fda2c1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:45:29.061 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:29.059+0000 7fda2b7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda2c060280 0x7fda2c1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:54062/0 (socket says 172.21.15.134:54062) 2024-01-21T11:45:29.061 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:29.059+0000 7fda2b7fe700 1 -- 172.21.15.134:0/1939591507 learned_addr learned my addr 172.21.15.134:0/1939591507 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:45:29.061 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:29.060+0000 7fda2b7fe700 1 -- 172.21.15.134:0/1939591507 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda20005040 con 0x7fda2c060280 2024-01-21T11:45:29.061 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:29.060+0000 7fda2b7fe700 1 --2- 172.21.15.134:0/1939591507 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda2c060280 0x7fda2c1066e0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7fda20013510 tx=0x7fda2000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:45:29.062 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:29.060+0000 7fda13fff700 1 -- 172.21.15.134:0/1939591507 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda20009320 con 0x7fda2c060280 2024-01-21T11:45:29.062 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:29.060+0000 7fda31dc2700 1 -- 172.21.15.134:0/1939591507 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda2c106dd0 con 0x7fda2c060280 2024-01-21T11:45:29.062 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:29.060+0000 7fda13fff700 1 -- 172.21.15.134:0/1939591507 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fda20013f00 con 0x7fda2c060280 2024-01-21T11:45:29.062 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:29.060+0000 7fda13fff700 1 -- 172.21.15.134:0/1939591507 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda20014dc0 con 0x7fda2c060280 2024-01-21T11:45:29.062 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:29.060+0000 7fda31dc2700 1 -- 172.21.15.134:0/1939591507 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda2c107260 con 0x7fda2c060280 2024-01-21T11:45:29.062 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:29.061+0000 7fda13fff700 1 -- 172.21.15.134:0/1939591507 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fda200095f0 con 0x7fda2c060280 2024-01-21T11:45:29.062 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:29.061+0000 7fda31dc2700 1 -- 172.21.15.134:0/1939591507 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda2c045ad0 con 0x7fda2c060280 2024-01-21T11:45:29.062 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:29.061+0000 7fda13fff700 1 --2- 172.21.15.134:0/1939591507 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fda1402ff60 0x7fda14032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:45:29.062 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:29.061+0000 7fda13fff700 1 -- 172.21.15.134:0/1939591507 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fda20041080 con 0x7fda2c060280 2024-01-21T11:45:29.063 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:29.061+0000 7fda2affd700 1 --2- 172.21.15.134:0/1939591507 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fda1402ff60 0x7fda14032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:45:29.063 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:29.062+0000 7fda2affd700 1 --2- 172.21.15.134:0/1939591507 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fda1402ff60 0x7fda14032420 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fda1c0096a0 tx=0x7fda1c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:45:29.066 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:29.066+0000 7fda13fff700 1 -- 172.21.15.134:0/1939591507 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fda20014420 con 0x7fda2c060280 2024-01-21T11:45:29.269 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:29.269+0000 7fda31dc2700 1 -- 172.21.15.134:0/1939591507 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fda2c108110 con 0x7fda2c060280 2024-01-21T11:45:29.270 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:29.270+0000 7fda13fff700 1 -- 172.21.15.134:0/1939591507 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fda20010230 con 0x7fda2c060280 2024-01-21T11:45:29.270 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:45:29.270 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:45:29.272 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:29.272+0000 7fda31dc2700 1 -- 172.21.15.134:0/1939591507 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fda1402ff60 msgr2=0x7fda14032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:45:29.272 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:29.272+0000 7fda31dc2700 1 --2- 172.21.15.134:0/1939591507 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fda1402ff60 0x7fda14032420 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fda1c0096a0 tx=0x7fda1c006b40).stop 2024-01-21T11:45:29.272 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:29.272+0000 7fda31dc2700 1 -- 172.21.15.134:0/1939591507 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda2c060280 msgr2=0x7fda2c1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:45:29.272 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:29.272+0000 7fda31dc2700 1 --2- 172.21.15.134:0/1939591507 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda2c060280 0x7fda2c1066e0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7fda20013510 tx=0x7fda2000b040).stop 2024-01-21T11:45:29.272 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:29.272+0000 7fda31dc2700 1 -- 172.21.15.134:0/1939591507 shutdown_connections 2024-01-21T11:45:29.272 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:29.272+0000 7fda31dc2700 1 --2- 172.21.15.134:0/1939591507 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fda1402ff60 0x7fda14032420 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:45:29.272 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:29.272+0000 7fda31dc2700 1 --2- 172.21.15.134:0/1939591507 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda2c060280 0x7fda2c1066e0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:45:29.272 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:29.272+0000 7fda31dc2700 1 -- 172.21.15.134:0/1939591507 >> 172.21.15.134:0/1939591507 conn(0x7fda2c0f0b70 msgr2=0x7fda2c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:45:29.273 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:29.272+0000 7fda31dc2700 1 -- 172.21.15.134:0/1939591507 shutdown_connections 2024-01-21T11:45:29.273 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:29.272+0000 7fda31dc2700 1 -- 172.21.15.134:0/1939591507 wait complete. 2024-01-21T11:45:29.273 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:45:29.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:45:29 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:45:27.834328+0000 mgr.smithi029.blhxxk (mgr.14164) 104 : cluster [DBG] pgmap v97: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:45:29.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:45:29 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:45:29.271092+0000 mon.smithi029 (mon.0) 187 : audit [DBG] from='client.? 172.21.15.134:0/1939591507' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:45:30.653 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:45:30.654 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:45:31.838 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:31.836+0000 7f1f85378700 1 -- 172.21.15.134:0/4222034719 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f800f5410 msgr2=0x7f1f800f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:45:31.838 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:31.836+0000 7f1f85378700 1 --2- 172.21.15.134:0/4222034719 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f800f5410 0x7f1f800f5830 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f1f68009fb0 tx=0x7f1f68009b30).stop 2024-01-21T11:45:31.838 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:31.836+0000 7f1f85378700 1 -- 172.21.15.134:0/4222034719 shutdown_connections 2024-01-21T11:45:31.838 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:31.836+0000 7f1f85378700 1 --2- 172.21.15.134:0/4222034719 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f800f5410 0x7f1f800f5830 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:45:31.838 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:31.836+0000 7f1f85378700 1 -- 172.21.15.134:0/4222034719 >> 172.21.15.134:0/4222034719 conn(0x7f1f800f0b70 msgr2=0x7f1f800f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:45:31.838 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:31.836+0000 7f1f85378700 1 -- 172.21.15.134:0/4222034719 shutdown_connections 2024-01-21T11:45:31.838 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:31.836+0000 7f1f85378700 1 -- 172.21.15.134:0/4222034719 wait complete. 2024-01-21T11:45:31.838 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:31.837+0000 7f1f85378700 1 Processor -- start 2024-01-21T11:45:31.839 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:31.837+0000 7f1f85378700 1 -- start start 2024-01-21T11:45:31.839 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:31.837+0000 7f1f85378700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f80106830 0x7f1f80106c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:45:31.839 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:31.837+0000 7f1f85378700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f80107140 con 0x7f1f80106830 2024-01-21T11:45:31.839 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:31.837+0000 7f1f7effd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f80106830 0x7f1f80106c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:45:31.839 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:31.837+0000 7f1f7effd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f80106830 0x7f1f80106c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:54086/0 (socket says 172.21.15.134:54086) 2024-01-21T11:45:31.839 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:31.837+0000 7f1f7effd700 1 -- 172.21.15.134:0/2461424906 learned_addr learned my addr 172.21.15.134:0/2461424906 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:45:31.839 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:31.837+0000 7f1f7effd700 1 -- 172.21.15.134:0/2461424906 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f68005040 con 0x7f1f80106830 2024-01-21T11:45:31.839 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:31.838+0000 7f1f7effd700 1 --2- 172.21.15.134:0/2461424906 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f80106830 0x7f1f80106c50 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f1f68009f80 tx=0x7f1f68009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:45:31.839 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:31.838+0000 7f1f777fe700 1 -- 172.21.15.134:0/2461424906 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1f680136c0 con 0x7f1f80106830 2024-01-21T11:45:31.839 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:31.838+0000 7f1f85378700 1 -- 172.21.15.134:0/2461424906 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f80107340 con 0x7f1f80106830 2024-01-21T11:45:31.839 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:31.838+0000 7f1f777fe700 1 -- 172.21.15.134:0/2461424906 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1f68013e70 con 0x7f1f80106830 2024-01-21T11:45:31.839 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:31.838+0000 7f1f777fe700 1 -- 172.21.15.134:0/2461424906 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1f68014420 con 0x7f1f80106830 2024-01-21T11:45:31.839 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:31.838+0000 7f1f85378700 1 -- 172.21.15.134:0/2461424906 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f80107e90 con 0x7f1f80106830 2024-01-21T11:45:31.840 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:31.839+0000 7f1f777fe700 1 -- 172.21.15.134:0/2461424906 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1f68025060 con 0x7f1f80106830 2024-01-21T11:45:31.840 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:31.839+0000 7f1f85378700 1 -- 172.21.15.134:0/2461424906 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f80059100 con 0x7f1f80106830 2024-01-21T11:45:31.840 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:31.839+0000 7f1f777fe700 1 --2- 172.21.15.134:0/2461424906 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f1f6c02ff60 0x7f1f6c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:45:31.841 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:31.839+0000 7f1f777fe700 1 -- 172.21.15.134:0/2461424906 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1f68040e50 con 0x7f1f80106830 2024-01-21T11:45:31.841 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:31.839+0000 7f1f7e7fc700 1 --2- 172.21.15.134:0/2461424906 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f1f6c02ff60 0x7f1f6c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:45:31.841 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:31.840+0000 7f1f7e7fc700 1 --2- 172.21.15.134:0/2461424906 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f1f6c02ff60 0x7f1f6c032420 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f1f700096a0 tx=0x7f1f70006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:45:31.844 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:31.844+0000 7f1f777fe700 1 -- 172.21.15.134:0/2461424906 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1f6801cb30 con 0x7f1f80106830 2024-01-21T11:45:31.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:45:31 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:45:29.834735+0000 mgr.smithi029.blhxxk (mgr.14164) 105 : cluster [DBG] pgmap v98: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:45:32.049 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:32.048+0000 7f1f85378700 1 -- 172.21.15.134:0/2461424906 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1f80045ad0 con 0x7f1f80106830 2024-01-21T11:45:32.050 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:32.049+0000 7f1f777fe700 1 -- 172.21.15.134:0/2461424906 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1f6802d2d0 con 0x7f1f80106830 2024-01-21T11:45:32.050 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:45:32.050 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:45:32.051 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:32.051+0000 7f1f85378700 1 -- 172.21.15.134:0/2461424906 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f1f6c02ff60 msgr2=0x7f1f6c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:45:32.052 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:32.051+0000 7f1f85378700 1 --2- 172.21.15.134:0/2461424906 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f1f6c02ff60 0x7f1f6c032420 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f1f700096a0 tx=0x7f1f70006b40).stop 2024-01-21T11:45:32.052 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:32.051+0000 7f1f85378700 1 -- 172.21.15.134:0/2461424906 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f80106830 msgr2=0x7f1f80106c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:45:32.052 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:32.051+0000 7f1f85378700 1 --2- 172.21.15.134:0/2461424906 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f80106830 0x7f1f80106c50 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f1f68009f80 tx=0x7f1f68009b30).stop 2024-01-21T11:45:32.052 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:32.051+0000 7f1f85378700 1 -- 172.21.15.134:0/2461424906 shutdown_connections 2024-01-21T11:45:32.052 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:32.051+0000 7f1f85378700 1 --2- 172.21.15.134:0/2461424906 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f1f6c02ff60 0x7f1f6c032420 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:45:32.052 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:32.051+0000 7f1f85378700 1 --2- 172.21.15.134:0/2461424906 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f80106830 0x7f1f80106c50 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:45:32.052 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:32.051+0000 7f1f85378700 1 -- 172.21.15.134:0/2461424906 >> 172.21.15.134:0/2461424906 conn(0x7f1f800f0b70 msgr2=0x7f1f8005fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:45:32.052 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:32.051+0000 7f1f85378700 1 -- 172.21.15.134:0/2461424906 shutdown_connections 2024-01-21T11:45:32.052 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:32.051+0000 7f1f85378700 1 -- 172.21.15.134:0/2461424906 wait complete. 2024-01-21T11:45:32.052 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:45:32.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:45:32 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:45:32.050448+0000 mon.smithi029 (mon.0) 188 : audit [DBG] from='client.? 172.21.15.134:0/2461424906' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:45:33.425 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:45:33.425 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:45:33.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:45:33 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:45:31.835025+0000 mgr.smithi029.blhxxk (mgr.14164) 106 : cluster [DBG] pgmap v99: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:45:34.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:34.553+0000 7ff36e659700 1 -- 172.21.15.134:0/907080100 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3680f51a0 msgr2=0x7ff3680f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:45:34.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:34.553+0000 7ff36e659700 1 --2- 172.21.15.134:0/907080100 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3680f51a0 0x7ff3680f55c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7ff358009fb0 tx=0x7ff358009b30).stop 2024-01-21T11:45:34.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:34.554+0000 7ff36e659700 1 -- 172.21.15.134:0/907080100 shutdown_connections 2024-01-21T11:45:34.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:34.554+0000 7ff36e659700 1 --2- 172.21.15.134:0/907080100 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3680f51a0 0x7ff3680f55c0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:45:34.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:34.554+0000 7ff36e659700 1 -- 172.21.15.134:0/907080100 >> 172.21.15.134:0/907080100 conn(0x7ff3680f09a0 msgr2=0x7ff3680f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:45:34.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:34.554+0000 7ff36e659700 1 -- 172.21.15.134:0/907080100 shutdown_connections 2024-01-21T11:45:34.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:34.554+0000 7ff36e659700 1 -- 172.21.15.134:0/907080100 wait complete. 2024-01-21T11:45:34.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:34.554+0000 7ff36e659700 1 Processor -- start 2024-01-21T11:45:34.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:34.554+0000 7ff36e659700 1 -- start start 2024-01-21T11:45:34.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:34.554+0000 7ff36e659700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3680f51a0 0x7ff3681091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:45:34.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:34.554+0000 7ff36e659700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff368060240 con 0x7ff3680f51a0 2024-01-21T11:45:34.557 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:34.555+0000 7ff36d657700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3680f51a0 0x7ff3681091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:45:34.557 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:34.555+0000 7ff36d657700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3680f51a0 0x7ff3681091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:37778/0 (socket says 172.21.15.134:37778) 2024-01-21T11:45:34.557 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:34.555+0000 7ff36d657700 1 -- 172.21.15.134:0/275123988 learned_addr learned my addr 172.21.15.134:0/275123988 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:45:34.557 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:34.555+0000 7ff36d657700 1 -- 172.21.15.134:0/275123988 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff358005040 con 0x7ff3680f51a0 2024-01-21T11:45:34.557 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:34.555+0000 7ff36d657700 1 --2- 172.21.15.134:0/275123988 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3680f51a0 0x7ff3681091b0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7ff358013510 tx=0x7ff35800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:45:34.557 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:34.555+0000 7ff35dffb700 1 -- 172.21.15.134:0/275123988 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff358009320 con 0x7ff3680f51a0 2024-01-21T11:45:34.557 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:34.555+0000 7ff36e659700 1 -- 172.21.15.134:0/275123988 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff368060440 con 0x7ff3680f51a0 2024-01-21T11:45:34.557 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:34.555+0000 7ff35dffb700 1 -- 172.21.15.134:0/275123988 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff358013f00 con 0x7ff3680f51a0 2024-01-21T11:45:34.557 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:34.555+0000 7ff35dffb700 1 -- 172.21.15.134:0/275123988 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff358014dc0 con 0x7ff3680f51a0 2024-01-21T11:45:34.557 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:34.555+0000 7ff36e659700 1 -- 172.21.15.134:0/275123988 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff368109860 con 0x7ff3680f51a0 2024-01-21T11:45:34.558 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:34.556+0000 7ff35dffb700 1 -- 172.21.15.134:0/275123988 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff358014420 con 0x7ff3680f51a0 2024-01-21T11:45:34.558 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:34.556+0000 7ff36e659700 1 -- 172.21.15.134:0/275123988 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff368045ad0 con 0x7ff3680f51a0 2024-01-21T11:45:34.558 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:34.556+0000 7ff35dffb700 1 --2- 172.21.15.134:0/275123988 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7ff35402ff60 0x7ff354032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:45:34.558 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:34.556+0000 7ff35dffb700 1 -- 172.21.15.134:0/275123988 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff3580410f0 con 0x7ff3680f51a0 2024-01-21T11:45:34.559 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:34.557+0000 7ff36ce56700 1 --2- 172.21.15.134:0/275123988 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7ff35402ff60 0x7ff354032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:45:34.559 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:34.557+0000 7ff36ce56700 1 --2- 172.21.15.134:0/275123988 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7ff35402ff60 0x7ff354032420 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7ff3640096a0 tx=0x7ff364006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:45:34.561 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:34.561+0000 7ff35dffb700 1 -- 172.21.15.134:0/275123988 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff3580095f0 con 0x7ff3680f51a0 2024-01-21T11:45:34.768 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:34.767+0000 7ff36e659700 1 -- 172.21.15.134:0/275123988 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff36810a080 con 0x7ff3680f51a0 2024-01-21T11:45:34.768 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:34.768+0000 7ff35dffb700 1 -- 172.21.15.134:0/275123988 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff35801cb10 con 0x7ff3680f51a0 2024-01-21T11:45:34.768 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:45:34.769 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:45:34.770 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:34.770+0000 7ff36e659700 1 -- 172.21.15.134:0/275123988 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7ff35402ff60 msgr2=0x7ff354032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:45:34.770 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:34.770+0000 7ff36e659700 1 --2- 172.21.15.134:0/275123988 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7ff35402ff60 0x7ff354032420 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7ff3640096a0 tx=0x7ff364006b40).stop 2024-01-21T11:45:34.770 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:34.770+0000 7ff36e659700 1 -- 172.21.15.134:0/275123988 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3680f51a0 msgr2=0x7ff3681091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:45:34.770 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:34.770+0000 7ff36e659700 1 --2- 172.21.15.134:0/275123988 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3680f51a0 0x7ff3681091b0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7ff358013510 tx=0x7ff35800b040).stop 2024-01-21T11:45:34.770 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:34.770+0000 7ff36e659700 1 -- 172.21.15.134:0/275123988 shutdown_connections 2024-01-21T11:45:34.770 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:34.770+0000 7ff36e659700 1 --2- 172.21.15.134:0/275123988 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7ff35402ff60 0x7ff354032420 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:45:34.770 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:34.770+0000 7ff36e659700 1 --2- 172.21.15.134:0/275123988 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3680f51a0 0x7ff3681091b0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:45:34.770 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:34.770+0000 7ff36e659700 1 -- 172.21.15.134:0/275123988 >> 172.21.15.134:0/275123988 conn(0x7ff3680f09a0 msgr2=0x7ff3680f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:45:34.770 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:34.770+0000 7ff36e659700 1 -- 172.21.15.134:0/275123988 shutdown_connections 2024-01-21T11:45:34.771 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:34.770+0000 7ff36e659700 1 -- 172.21.15.134:0/275123988 wait complete. 2024-01-21T11:45:34.771 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:45:35.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:45:35 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:45:33.835347+0000 mgr.smithi029.blhxxk (mgr.14164) 107 : cluster [DBG] pgmap v100: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:45:35.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:45:35 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:45:34.769244+0000 mon.smithi029 (mon.0) 189 : audit [DBG] from='client.? 172.21.15.134:0/275123988' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:45:36.153 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:45:36.154 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:45:36.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:45:36 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:45:35.835639+0000 mgr.smithi029.blhxxk (mgr.14164) 108 : cluster [DBG] pgmap v101: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:45:37.366 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:37.366+0000 7f1d72ec8700 1 -- 172.21.15.134:0/641507089 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d6c060220 msgr2=0x7f1d6c0f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:45:37.367 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:37.366+0000 7f1d72ec8700 1 --2- 172.21.15.134:0/641507089 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d6c060220 0x7f1d6c0f9150 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f1d60009fb0 tx=0x7f1d60009b30).stop 2024-01-21T11:45:37.367 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:37.366+0000 7f1d72ec8700 1 -- 172.21.15.134:0/641507089 shutdown_connections 2024-01-21T11:45:37.367 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:37.366+0000 7f1d72ec8700 1 --2- 172.21.15.134:0/641507089 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d6c060220 0x7f1d6c0f9150 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:45:37.367 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:37.366+0000 7f1d72ec8700 1 -- 172.21.15.134:0/641507089 >> 172.21.15.134:0/641507089 conn(0x7f1d6c0f0b70 msgr2=0x7f1d6c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:45:37.367 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:37.366+0000 7f1d72ec8700 1 -- 172.21.15.134:0/641507089 shutdown_connections 2024-01-21T11:45:37.367 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:37.366+0000 7f1d72ec8700 1 -- 172.21.15.134:0/641507089 wait complete. 2024-01-21T11:45:37.367 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:37.366+0000 7f1d72ec8700 1 Processor -- start 2024-01-21T11:45:37.367 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:37.367+0000 7f1d72ec8700 1 -- start start 2024-01-21T11:45:37.369 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:37.367+0000 7f1d72ec8700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d6c060220 0x7f1d6c1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:45:37.369 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:37.367+0000 7f1d72ec8700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d6c106bd0 con 0x7f1d6c060220 2024-01-21T11:45:37.369 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:37.367+0000 7f1d70c64700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d6c060220 0x7f1d6c1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:45:37.369 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:37.367+0000 7f1d70c64700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d6c060220 0x7f1d6c1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:37802/0 (socket says 172.21.15.134:37802) 2024-01-21T11:45:37.369 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:37.367+0000 7f1d70c64700 1 -- 172.21.15.134:0/1924609744 learned_addr learned my addr 172.21.15.134:0/1924609744 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:45:37.369 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:37.367+0000 7f1d70c64700 1 -- 172.21.15.134:0/1924609744 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d60005040 con 0x7f1d6c060220 2024-01-21T11:45:37.369 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:37.367+0000 7f1d70c64700 1 --2- 172.21.15.134:0/1924609744 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d6c060220 0x7f1d6c1066e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f1d60013510 tx=0x7f1d6000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:45:37.369 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:37.368+0000 7f1d697fa700 1 -- 172.21.15.134:0/1924609744 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1d60009320 con 0x7f1d6c060220 2024-01-21T11:45:37.369 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:37.368+0000 7f1d72ec8700 1 -- 172.21.15.134:0/1924609744 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d6c106dd0 con 0x7f1d6c060220 2024-01-21T11:45:37.370 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:37.368+0000 7f1d697fa700 1 -- 172.21.15.134:0/1924609744 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1d60013f00 con 0x7f1d6c060220 2024-01-21T11:45:37.370 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:37.368+0000 7f1d697fa700 1 -- 172.21.15.134:0/1924609744 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1d60014dc0 con 0x7f1d6c060220 2024-01-21T11:45:37.370 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:37.368+0000 7f1d72ec8700 1 -- 172.21.15.134:0/1924609744 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d6c107260 con 0x7f1d6c060220 2024-01-21T11:45:37.370 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:37.368+0000 7f1d697fa700 1 -- 172.21.15.134:0/1924609744 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1d60014420 con 0x7f1d6c060220 2024-01-21T11:45:37.370 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:37.368+0000 7f1d72ec8700 1 -- 172.21.15.134:0/1924609744 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d6c045ad0 con 0x7f1d6c060220 2024-01-21T11:45:37.370 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:37.369+0000 7f1d697fa700 1 --2- 172.21.15.134:0/1924609744 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f1d5c02ff60 0x7f1d5c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:45:37.370 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:37.369+0000 7f1d697fa700 1 -- 172.21.15.134:0/1924609744 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1d600410f0 con 0x7f1d6c060220 2024-01-21T11:45:37.370 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:37.369+0000 7f1d6bfff700 1 --2- 172.21.15.134:0/1924609744 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f1d5c02ff60 0x7f1d5c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:45:37.370 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:37.369+0000 7f1d6bfff700 1 --2- 172.21.15.134:0/1924609744 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f1d5c02ff60 0x7f1d5c032420 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f1d580096a0 tx=0x7f1d58006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:45:37.373 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:37.374+0000 7f1d697fa700 1 -- 172.21.15.134:0/1924609744 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1d600095f0 con 0x7f1d6c060220 2024-01-21T11:45:37.577 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:37.577+0000 7f1d72ec8700 1 -- 172.21.15.134:0/1924609744 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1d6c1080b0 con 0x7f1d6c060220 2024-01-21T11:45:37.577 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:37.577+0000 7f1d697fa700 1 -- 172.21.15.134:0/1924609744 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1d6001cb10 con 0x7f1d6c060220 2024-01-21T11:45:37.578 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:45:37.578 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:45:37.579 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:37.579+0000 7f1d72ec8700 1 -- 172.21.15.134:0/1924609744 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f1d5c02ff60 msgr2=0x7f1d5c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:45:37.579 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:37.579+0000 7f1d72ec8700 1 --2- 172.21.15.134:0/1924609744 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f1d5c02ff60 0x7f1d5c032420 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f1d580096a0 tx=0x7f1d58006b40).stop 2024-01-21T11:45:37.579 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:37.579+0000 7f1d72ec8700 1 -- 172.21.15.134:0/1924609744 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d6c060220 msgr2=0x7f1d6c1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:45:37.579 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:37.579+0000 7f1d72ec8700 1 --2- 172.21.15.134:0/1924609744 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d6c060220 0x7f1d6c1066e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f1d60013510 tx=0x7f1d6000b040).stop 2024-01-21T11:45:37.579 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:37.579+0000 7f1d72ec8700 1 -- 172.21.15.134:0/1924609744 shutdown_connections 2024-01-21T11:45:37.580 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:37.579+0000 7f1d72ec8700 1 --2- 172.21.15.134:0/1924609744 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f1d5c02ff60 0x7f1d5c032420 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:45:37.580 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:37.579+0000 7f1d72ec8700 1 --2- 172.21.15.134:0/1924609744 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d6c060220 0x7f1d6c1066e0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:45:37.580 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:37.579+0000 7f1d72ec8700 1 -- 172.21.15.134:0/1924609744 >> 172.21.15.134:0/1924609744 conn(0x7f1d6c0f0b70 msgr2=0x7f1d6c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:45:37.580 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:37.579+0000 7f1d72ec8700 1 -- 172.21.15.134:0/1924609744 shutdown_connections 2024-01-21T11:45:37.580 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:37.579+0000 7f1d72ec8700 1 -- 172.21.15.134:0/1924609744 wait complete. 2024-01-21T11:45:37.580 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:45:37.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:45:37 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:45:37.578504+0000 mon.smithi029 (mon.0) 190 : audit [DBG] from='client.? 172.21.15.134:0/1924609744' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:45:38.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:45:38 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:45:37.835988+0000 mgr.smithi029.blhxxk (mgr.14164) 109 : cluster [DBG] pgmap v102: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:45:38.972 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:45:38.972 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:45:40.136 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:40.135+0000 7feb13a2b700 1 -- 172.21.15.134:0/181975978 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb0c0f5410 msgr2=0x7feb0c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:45:40.137 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:40.135+0000 7feb13a2b700 1 --2- 172.21.15.134:0/181975978 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb0c0f5410 0x7feb0c0f5830 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7feafc009fb0 tx=0x7feafc009b30).stop 2024-01-21T11:45:40.137 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:40.135+0000 7feb13a2b700 1 -- 172.21.15.134:0/181975978 shutdown_connections 2024-01-21T11:45:40.137 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:40.135+0000 7feb13a2b700 1 --2- 172.21.15.134:0/181975978 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb0c0f5410 0x7feb0c0f5830 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:45:40.137 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:40.135+0000 7feb13a2b700 1 -- 172.21.15.134:0/181975978 >> 172.21.15.134:0/181975978 conn(0x7feb0c0f0b70 msgr2=0x7feb0c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:45:40.137 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:40.135+0000 7feb13a2b700 1 -- 172.21.15.134:0/181975978 shutdown_connections 2024-01-21T11:45:40.137 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:40.135+0000 7feb13a2b700 1 -- 172.21.15.134:0/181975978 wait complete. 2024-01-21T11:45:40.137 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:40.136+0000 7feb13a2b700 1 Processor -- start 2024-01-21T11:45:40.137 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:40.136+0000 7feb13a2b700 1 -- start start 2024-01-21T11:45:40.137 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:40.136+0000 7feb13a2b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb0c0f5410 0x7feb0c109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:45:40.137 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:40.136+0000 7feb13a2b700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb0c060220 con 0x7feb0c0f5410 2024-01-21T11:45:40.137 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:40.136+0000 7feb117c7700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb0c0f5410 0x7feb0c109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:45:40.137 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:40.136+0000 7feb117c7700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb0c0f5410 0x7feb0c109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:37822/0 (socket says 172.21.15.134:37822) 2024-01-21T11:45:40.137 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:40.136+0000 7feb117c7700 1 -- 172.21.15.134:0/4197931842 learned_addr learned my addr 172.21.15.134:0/4197931842 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:45:40.139 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:40.137+0000 7feb117c7700 1 -- 172.21.15.134:0/4197931842 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feafc005040 con 0x7feb0c0f5410 2024-01-21T11:45:40.139 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:40.137+0000 7feb117c7700 1 --2- 172.21.15.134:0/4197931842 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb0c0f5410 0x7feb0c109450 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7feafc014000 tx=0x7feafc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:45:40.139 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:40.137+0000 7feb01ffb700 1 -- 172.21.15.134:0/4197931842 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feafc0095f0 con 0x7feb0c0f5410 2024-01-21T11:45:40.139 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:40.137+0000 7feb01ffb700 1 -- 172.21.15.134:0/4197931842 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7feafc007e70 con 0x7feb0c0f5410 2024-01-21T11:45:40.139 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:40.137+0000 7feb01ffb700 1 -- 172.21.15.134:0/4197931842 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feafc014ea0 con 0x7feb0c0f5410 2024-01-21T11:45:40.139 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:40.137+0000 7feb13a2b700 1 -- 172.21.15.134:0/4197931842 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb0c060420 con 0x7feb0c0f5410 2024-01-21T11:45:40.139 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:40.137+0000 7feb13a2b700 1 -- 172.21.15.134:0/4197931842 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb0c109b00 con 0x7feb0c0f5410 2024-01-21T11:45:40.139 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:40.138+0000 7feb01ffb700 1 -- 172.21.15.134:0/4197931842 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7feafc025070 con 0x7feb0c0f5410 2024-01-21T11:45:40.139 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:40.138+0000 7feb13a2b700 1 -- 172.21.15.134:0/4197931842 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb0c059100 con 0x7feb0c0f5410 2024-01-21T11:45:40.140 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:40.138+0000 7feb01ffb700 1 --2- 172.21.15.134:0/4197931842 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7feaf802ffb0 0x7feaf8032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:45:40.140 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:40.138+0000 7feb01ffb700 1 -- 172.21.15.134:0/4197931842 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7feafc040a10 con 0x7feb0c0f5410 2024-01-21T11:45:40.140 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:40.138+0000 7feb10fc6700 1 --2- 172.21.15.134:0/4197931842 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7feaf802ffb0 0x7feaf8032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:45:40.140 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:40.139+0000 7feb10fc6700 1 --2- 172.21.15.134:0/4197931842 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7feaf802ffb0 0x7feaf8032470 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7feb080096a0 tx=0x7feb08006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:45:40.143 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:40.143+0000 7feb01ffb700 1 -- 172.21.15.134:0/4197931842 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7feafc00f300 con 0x7feb0c0f5410 2024-01-21T11:45:40.352 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:40.351+0000 7feb13a2b700 1 -- 172.21.15.134:0/4197931842 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7feb0c045ad0 con 0x7feb0c0f5410 2024-01-21T11:45:40.352 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:40.352+0000 7feb01ffb700 1 -- 172.21.15.134:0/4197931842 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7feafc00f300 con 0x7feb0c0f5410 2024-01-21T11:45:40.353 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:45:40.353 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:45:40.354 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:40.354+0000 7feb13a2b700 1 -- 172.21.15.134:0/4197931842 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7feaf802ffb0 msgr2=0x7feaf8032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:45:40.354 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:40.354+0000 7feb13a2b700 1 --2- 172.21.15.134:0/4197931842 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7feaf802ffb0 0x7feaf8032470 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7feb080096a0 tx=0x7feb08006b40).stop 2024-01-21T11:45:40.354 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:40.354+0000 7feb13a2b700 1 -- 172.21.15.134:0/4197931842 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb0c0f5410 msgr2=0x7feb0c109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:45:40.354 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:40.354+0000 7feb13a2b700 1 --2- 172.21.15.134:0/4197931842 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb0c0f5410 0x7feb0c109450 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7feafc014000 tx=0x7feafc00b040).stop 2024-01-21T11:45:40.355 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:40.354+0000 7feb13a2b700 1 -- 172.21.15.134:0/4197931842 shutdown_connections 2024-01-21T11:45:40.355 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:40.354+0000 7feb13a2b700 1 --2- 172.21.15.134:0/4197931842 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7feaf802ffb0 0x7feaf8032470 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:45:40.355 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:40.354+0000 7feb13a2b700 1 --2- 172.21.15.134:0/4197931842 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb0c0f5410 0x7feb0c109450 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:45:40.355 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:40.354+0000 7feb13a2b700 1 -- 172.21.15.134:0/4197931842 >> 172.21.15.134:0/4197931842 conn(0x7feb0c0f0b70 msgr2=0x7feb0c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:45:40.355 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:40.354+0000 7feb13a2b700 1 -- 172.21.15.134:0/4197931842 shutdown_connections 2024-01-21T11:45:40.355 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:40.354+0000 7feb13a2b700 1 -- 172.21.15.134:0/4197931842 wait complete. 2024-01-21T11:45:40.355 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:45:41.168 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:45:40 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:45:39.836291+0000 mgr.smithi029.blhxxk (mgr.14164) 110 : cluster [DBG] pgmap v103: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:45:41.169 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:45:40 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:45:40.353392+0000 mon.smithi029 (mon.0) 191 : audit [DBG] from='client.? 172.21.15.134:0/4197931842' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:45:41.731 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:45:41.731 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:45:42.873 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:42.872+0000 7f3f2a5ad700 1 -- 172.21.15.134:0/2340062286 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f240f6510 msgr2=0x7f3f240f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:45:42.874 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:42.872+0000 7f3f2a5ad700 1 --2- 172.21.15.134:0/2340062286 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f240f6510 0x7f3f240f6930 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7f3f0c009fb0 tx=0x7f3f0c009b30).stop 2024-01-21T11:45:42.874 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:42.872+0000 7f3f2a5ad700 1 -- 172.21.15.134:0/2340062286 shutdown_connections 2024-01-21T11:45:42.874 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:42.872+0000 7f3f2a5ad700 1 --2- 172.21.15.134:0/2340062286 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f240f6510 0x7f3f240f6930 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:45:42.874 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:42.872+0000 7f3f2a5ad700 1 -- 172.21.15.134:0/2340062286 >> 172.21.15.134:0/2340062286 conn(0x7f3f240f1c70 msgr2=0x7f3f240f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:45:42.874 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:42.872+0000 7f3f2a5ad700 1 -- 172.21.15.134:0/2340062286 shutdown_connections 2024-01-21T11:45:42.874 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:42.872+0000 7f3f2a5ad700 1 -- 172.21.15.134:0/2340062286 wait complete. 2024-01-21T11:45:42.874 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:42.873+0000 7f3f2a5ad700 1 Processor -- start 2024-01-21T11:45:42.874 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:42.873+0000 7f3f2a5ad700 1 -- start start 2024-01-21T11:45:42.874 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:42.873+0000 7f3f2a5ad700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f240f6510 0x7f3f24106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:45:42.874 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:42.873+0000 7f3f2a5ad700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f24106c20 con 0x7f3f240f6510 2024-01-21T11:45:42.874 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:42.874+0000 7f3f23fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f240f6510 0x7f3f24106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:45:42.875 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:42.874+0000 7f3f23fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f240f6510 0x7f3f24106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:37846/0 (socket says 172.21.15.134:37846) 2024-01-21T11:45:42.875 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:42.874+0000 7f3f23fff700 1 -- 172.21.15.134:0/601808110 learned_addr learned my addr 172.21.15.134:0/601808110 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:45:42.876 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:42.874+0000 7f3f23fff700 1 -- 172.21.15.134:0/601808110 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f0c005040 con 0x7f3f240f6510 2024-01-21T11:45:42.876 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:42.874+0000 7f3f23fff700 1 --2- 172.21.15.134:0/601808110 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f240f6510 0x7f3f24106730 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f3f0c013510 tx=0x7f3f0c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:45:42.876 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:42.874+0000 7f3f20ff9700 1 -- 172.21.15.134:0/601808110 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3f0c009320 con 0x7f3f240f6510 2024-01-21T11:45:42.876 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:42.874+0000 7f3f2a5ad700 1 -- 172.21.15.134:0/601808110 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f24106e20 con 0x7f3f240f6510 2024-01-21T11:45:42.877 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:42.874+0000 7f3f20ff9700 1 -- 172.21.15.134:0/601808110 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3f0c013f00 con 0x7f3f240f6510 2024-01-21T11:45:42.877 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:42.874+0000 7f3f20ff9700 1 -- 172.21.15.134:0/601808110 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3f0c014d90 con 0x7f3f240f6510 2024-01-21T11:45:42.877 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:42.874+0000 7f3f2a5ad700 1 -- 172.21.15.134:0/601808110 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f241072b0 con 0x7f3f240f6510 2024-01-21T11:45:42.877 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:42.875+0000 7f3f20ff9700 1 -- 172.21.15.134:0/601808110 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3f0c014420 con 0x7f3f240f6510 2024-01-21T11:45:42.877 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:42.875+0000 7f3f2a5ad700 1 -- 172.21.15.134:0/601808110 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f24045ad0 con 0x7f3f240f6510 2024-01-21T11:45:42.877 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:42.875+0000 7f3f20ff9700 1 --2- 172.21.15.134:0/601808110 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f3f1002ff60 0x7f3f10032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:45:42.877 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:42.875+0000 7f3f20ff9700 1 -- 172.21.15.134:0/601808110 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3f0c041000 con 0x7f3f240f6510 2024-01-21T11:45:42.877 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:42.875+0000 7f3f237fe700 1 --2- 172.21.15.134:0/601808110 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f3f1002ff60 0x7f3f10032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:45:42.877 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:42.876+0000 7f3f237fe700 1 --2- 172.21.15.134:0/601808110 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f3f1002ff60 0x7f3f10032420 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f3f140096a0 tx=0x7f3f14006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:45:42.880 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:42.880+0000 7f3f20ff9700 1 -- 172.21.15.134:0/601808110 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3f0c0095f0 con 0x7f3f240f6510 2024-01-21T11:45:43.089 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:43.089+0000 7f3f2a5ad700 1 -- 172.21.15.134:0/601808110 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3f240fbf60 con 0x7f3f240f6510 2024-01-21T11:45:43.089 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:43.090+0000 7f3f20ff9700 1 -- 172.21.15.134:0/601808110 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3f0c01db70 con 0x7f3f240f6510 2024-01-21T11:45:43.090 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:45:43.090 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:45:43.091 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:43.091+0000 7f3f2a5ad700 1 -- 172.21.15.134:0/601808110 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f3f1002ff60 msgr2=0x7f3f10032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:45:43.091 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:43.091+0000 7f3f2a5ad700 1 --2- 172.21.15.134:0/601808110 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f3f1002ff60 0x7f3f10032420 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f3f140096a0 tx=0x7f3f14006b40).stop 2024-01-21T11:45:43.091 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:43.091+0000 7f3f2a5ad700 1 -- 172.21.15.134:0/601808110 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f240f6510 msgr2=0x7f3f24106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:45:43.092 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:43.091+0000 7f3f2a5ad700 1 --2- 172.21.15.134:0/601808110 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f240f6510 0x7f3f24106730 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f3f0c013510 tx=0x7f3f0c00b040).stop 2024-01-21T11:45:43.092 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:43.091+0000 7f3f2a5ad700 1 -- 172.21.15.134:0/601808110 shutdown_connections 2024-01-21T11:45:43.092 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:43.091+0000 7f3f2a5ad700 1 --2- 172.21.15.134:0/601808110 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f3f1002ff60 0x7f3f10032420 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:45:43.092 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:43.091+0000 7f3f2a5ad700 1 --2- 172.21.15.134:0/601808110 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f240f6510 0x7f3f24106730 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:45:43.092 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:43.091+0000 7f3f2a5ad700 1 -- 172.21.15.134:0/601808110 >> 172.21.15.134:0/601808110 conn(0x7f3f240f1c70 msgr2=0x7f3f240f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:45:43.092 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:43.091+0000 7f3f2a5ad700 1 -- 172.21.15.134:0/601808110 shutdown_connections 2024-01-21T11:45:43.092 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:43.091+0000 7f3f2a5ad700 1 -- 172.21.15.134:0/601808110 wait complete. 2024-01-21T11:45:43.092 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:45:43.168 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:45:42 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:45:41.836614+0000 mgr.smithi029.blhxxk (mgr.14164) 111 : cluster [DBG] pgmap v104: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:45:44.168 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:45:43 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:45:43.090706+0000 mon.smithi029 (mon.0) 192 : audit [DBG] from='client.? 172.21.15.134:0/601808110' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:45:44.468 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:45:44.468 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:45:45.168 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:45:44 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:45:43.836996+0000 mgr.smithi029.blhxxk (mgr.14164) 112 : cluster [DBG] pgmap v105: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:45:45.656 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:45.654+0000 7efc720c7700 1 -- 172.21.15.134:0/3633295269 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc6c0f4880 msgr2=0x7efc6c0f4ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:45:45.656 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:45.654+0000 7efc720c7700 1 --2- 172.21.15.134:0/3633295269 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc6c0f4880 0x7efc6c0f4ca0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7efc5c009fb0 tx=0x7efc5c009b30).stop 2024-01-21T11:45:45.656 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:45.654+0000 7efc720c7700 1 -- 172.21.15.134:0/3633295269 shutdown_connections 2024-01-21T11:45:45.656 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:45.654+0000 7efc720c7700 1 --2- 172.21.15.134:0/3633295269 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc6c0f4880 0x7efc6c0f4ca0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:45:45.656 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:45.654+0000 7efc720c7700 1 -- 172.21.15.134:0/3633295269 >> 172.21.15.134:0/3633295269 conn(0x7efc6c0f02e0 msgr2=0x7efc6c0f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:45:45.656 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:45.654+0000 7efc720c7700 1 -- 172.21.15.134:0/3633295269 shutdown_connections 2024-01-21T11:45:45.656 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:45.654+0000 7efc720c7700 1 -- 172.21.15.134:0/3633295269 wait complete. 2024-01-21T11:45:45.656 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:45.655+0000 7efc720c7700 1 Processor -- start 2024-01-21T11:45:45.656 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:45.655+0000 7efc720c7700 1 -- start start 2024-01-21T11:45:45.656 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:45.655+0000 7efc720c7700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc6c0f4880 0x7efc6c1066b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:45:45.656 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:45.655+0000 7efc720c7700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc6c106ba0 con 0x7efc6c0f4880 2024-01-21T11:45:45.657 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:45.655+0000 7efc6b7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc6c0f4880 0x7efc6c1066b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:45:45.657 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:45.655+0000 7efc6b7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc6c0f4880 0x7efc6c1066b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:42526/0 (socket says 172.21.15.134:42526) 2024-01-21T11:45:45.657 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:45.655+0000 7efc6b7fe700 1 -- 172.21.15.134:0/2969710654 learned_addr learned my addr 172.21.15.134:0/2969710654 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:45:45.657 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:45.656+0000 7efc6b7fe700 1 -- 172.21.15.134:0/2969710654 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc5c005040 con 0x7efc6c0f4880 2024-01-21T11:45:45.657 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:45.656+0000 7efc6b7fe700 1 --2- 172.21.15.134:0/2969710654 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc6c0f4880 0x7efc6c1066b0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7efc5c013510 tx=0x7efc5c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:45:45.658 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:45.656+0000 7efc63fff700 1 -- 172.21.15.134:0/2969710654 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efc5c009320 con 0x7efc6c0f4880 2024-01-21T11:45:45.658 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:45.656+0000 7efc720c7700 1 -- 172.21.15.134:0/2969710654 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc6c106da0 con 0x7efc6c0f4880 2024-01-21T11:45:45.658 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:45.656+0000 7efc63fff700 1 -- 172.21.15.134:0/2969710654 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efc5c013f00 con 0x7efc6c0f4880 2024-01-21T11:45:45.658 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:45.656+0000 7efc63fff700 1 -- 172.21.15.134:0/2969710654 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efc5c014dc0 con 0x7efc6c0f4880 2024-01-21T11:45:45.658 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:45.656+0000 7efc720c7700 1 -- 172.21.15.134:0/2969710654 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc6c107230 con 0x7efc6c0f4880 2024-01-21T11:45:45.659 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:45.657+0000 7efc63fff700 1 -- 172.21.15.134:0/2969710654 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efc5c014420 con 0x7efc6c0f4880 2024-01-21T11:45:45.659 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:45.657+0000 7efc720c7700 1 -- 172.21.15.134:0/2969710654 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc6c045ad0 con 0x7efc6c0f4880 2024-01-21T11:45:45.659 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:45.657+0000 7efc63fff700 1 --2- 172.21.15.134:0/2969710654 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7efc5802ff60 0x7efc58032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:45:45.659 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:45.657+0000 7efc63fff700 1 -- 172.21.15.134:0/2969710654 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efc5c0410f0 con 0x7efc6c0f4880 2024-01-21T11:45:45.659 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:45.657+0000 7efc6affd700 1 --2- 172.21.15.134:0/2969710654 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7efc5802ff60 0x7efc58032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:45:45.659 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:45.658+0000 7efc6affd700 1 --2- 172.21.15.134:0/2969710654 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7efc5802ff60 0x7efc58032420 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7efc540096a0 tx=0x7efc54006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:45:45.662 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:45.662+0000 7efc63fff700 1 -- 172.21.15.134:0/2969710654 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efc5c0095f0 con 0x7efc6c0f4880 2024-01-21T11:45:45.868 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:45.868+0000 7efc720c7700 1 -- 172.21.15.134:0/2969710654 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efc6c0fbf50 con 0x7efc6c0f4880 2024-01-21T11:45:45.868 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:45.868+0000 7efc63fff700 1 -- 172.21.15.134:0/2969710654 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7efc5c01cb10 con 0x7efc6c0f4880 2024-01-21T11:45:45.869 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:45:45.869 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:45:45.870 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:45.870+0000 7efc720c7700 1 -- 172.21.15.134:0/2969710654 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7efc5802ff60 msgr2=0x7efc58032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:45:45.870 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:45.870+0000 7efc720c7700 1 --2- 172.21.15.134:0/2969710654 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7efc5802ff60 0x7efc58032420 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7efc540096a0 tx=0x7efc54006b40).stop 2024-01-21T11:45:45.871 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:45.870+0000 7efc720c7700 1 -- 172.21.15.134:0/2969710654 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc6c0f4880 msgr2=0x7efc6c1066b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:45:45.871 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:45.870+0000 7efc720c7700 1 --2- 172.21.15.134:0/2969710654 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc6c0f4880 0x7efc6c1066b0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7efc5c013510 tx=0x7efc5c00b040).stop 2024-01-21T11:45:45.871 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:45.870+0000 7efc720c7700 1 -- 172.21.15.134:0/2969710654 shutdown_connections 2024-01-21T11:45:45.871 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:45.870+0000 7efc720c7700 1 --2- 172.21.15.134:0/2969710654 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7efc5802ff60 0x7efc58032420 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:45:45.871 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:45.870+0000 7efc720c7700 1 --2- 172.21.15.134:0/2969710654 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc6c0f4880 0x7efc6c1066b0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:45:45.871 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:45.870+0000 7efc720c7700 1 -- 172.21.15.134:0/2969710654 >> 172.21.15.134:0/2969710654 conn(0x7efc6c0f02e0 msgr2=0x7efc6c0f0de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:45:45.871 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:45.870+0000 7efc720c7700 1 -- 172.21.15.134:0/2969710654 shutdown_connections 2024-01-21T11:45:45.871 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:45.870+0000 7efc720c7700 1 -- 172.21.15.134:0/2969710654 wait complete. 2024-01-21T11:45:45.871 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:45:46.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:45:46 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:45:45.869606+0000 mon.smithi029 (mon.0) 193 : audit [DBG] from='client.? 172.21.15.134:0/2969710654' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:45:47.254 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:45:47.255 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:45:47.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:45:47 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:45:45.837236+0000 mgr.smithi029.blhxxk (mgr.14164) 113 : cluster [DBG] pgmap v106: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:45:48.380 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:48.378+0000 7f8d62b25700 1 -- 172.21.15.134:0/2082589939 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d5c0f6510 msgr2=0x7f8d5c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:45:48.381 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:48.378+0000 7f8d62b25700 1 --2- 172.21.15.134:0/2082589939 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d5c0f6510 0x7f8d5c0f6930 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7f8d50009fb0 tx=0x7f8d50009b30).stop 2024-01-21T11:45:48.381 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:48.379+0000 7f8d62b25700 1 -- 172.21.15.134:0/2082589939 shutdown_connections 2024-01-21T11:45:48.381 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:48.379+0000 7f8d62b25700 1 --2- 172.21.15.134:0/2082589939 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d5c0f6510 0x7f8d5c0f6930 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:45:48.381 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:48.379+0000 7f8d62b25700 1 -- 172.21.15.134:0/2082589939 >> 172.21.15.134:0/2082589939 conn(0x7f8d5c0f1c70 msgr2=0x7f8d5c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:45:48.381 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:48.379+0000 7f8d62b25700 1 -- 172.21.15.134:0/2082589939 shutdown_connections 2024-01-21T11:45:48.381 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:48.379+0000 7f8d62b25700 1 -- 172.21.15.134:0/2082589939 wait complete. 2024-01-21T11:45:48.381 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:48.379+0000 7f8d62b25700 1 Processor -- start 2024-01-21T11:45:48.381 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:48.379+0000 7f8d62b25700 1 -- start start 2024-01-21T11:45:48.381 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:48.379+0000 7f8d62b25700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d5c0f6510 0x7f8d5c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:45:48.381 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:48.379+0000 7f8d62b25700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d5c106c20 con 0x7f8d5c0f6510 2024-01-21T11:45:48.381 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:48.380+0000 7f8d608c1700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d5c0f6510 0x7f8d5c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:45:48.381 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:48.380+0000 7f8d608c1700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d5c0f6510 0x7f8d5c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:42540/0 (socket says 172.21.15.134:42540) 2024-01-21T11:45:48.381 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:48.380+0000 7f8d608c1700 1 -- 172.21.15.134:0/3273086704 learned_addr learned my addr 172.21.15.134:0/3273086704 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:45:48.381 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:48.380+0000 7f8d608c1700 1 -- 172.21.15.134:0/3273086704 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d50005040 con 0x7f8d5c0f6510 2024-01-21T11:45:48.382 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:48.380+0000 7f8d608c1700 1 --2- 172.21.15.134:0/3273086704 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d5c0f6510 0x7f8d5c106730 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f8d50013510 tx=0x7f8d5000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:45:48.382 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:48.380+0000 7f8d597fa700 1 -- 172.21.15.134:0/3273086704 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8d50009320 con 0x7f8d5c0f6510 2024-01-21T11:45:48.382 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:48.380+0000 7f8d62b25700 1 -- 172.21.15.134:0/3273086704 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d5c106e20 con 0x7f8d5c0f6510 2024-01-21T11:45:48.382 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:48.380+0000 7f8d597fa700 1 -- 172.21.15.134:0/3273086704 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8d50013f00 con 0x7f8d5c0f6510 2024-01-21T11:45:48.382 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:48.380+0000 7f8d597fa700 1 -- 172.21.15.134:0/3273086704 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8d50014d90 con 0x7f8d5c0f6510 2024-01-21T11:45:48.382 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:48.380+0000 7f8d62b25700 1 -- 172.21.15.134:0/3273086704 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d5c1072b0 con 0x7f8d5c0f6510 2024-01-21T11:45:48.383 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:48.381+0000 7f8d597fa700 1 -- 172.21.15.134:0/3273086704 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8d500095f0 con 0x7f8d5c0f6510 2024-01-21T11:45:48.383 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:48.381+0000 7f8d597fa700 1 --2- 172.21.15.134:0/3273086704 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f8d4c02ff60 0x7f8d4c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:45:48.383 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:48.381+0000 7f8d62b25700 1 -- 172.21.15.134:0/3273086704 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d5c045ad0 con 0x7f8d5c0f6510 2024-01-21T11:45:48.383 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:48.381+0000 7f8d597fa700 1 -- 172.21.15.134:0/3273086704 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8d500410a0 con 0x7f8d5c0f6510 2024-01-21T11:45:48.383 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:48.381+0000 7f8d5bfff700 1 --2- 172.21.15.134:0/3273086704 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f8d4c02ff60 0x7f8d4c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:45:48.383 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:48.382+0000 7f8d5bfff700 1 --2- 172.21.15.134:0/3273086704 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f8d4c02ff60 0x7f8d4c032420 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f8d480096a0 tx=0x7f8d48006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:45:48.386 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:48.386+0000 7f8d597fa700 1 -- 172.21.15.134:0/3273086704 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8d5006d2b0 con 0x7f8d5c0f6510 2024-01-21T11:45:48.591 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:48.591+0000 7f8d62b25700 1 -- 172.21.15.134:0/3273086704 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8d5c0fc050 con 0x7f8d5c0f6510 2024-01-21T11:45:48.592 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:48.592+0000 7f8d597fa700 1 -- 172.21.15.134:0/3273086704 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8d5001c320 con 0x7f8d5c0f6510 2024-01-21T11:45:48.592 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:45:48.592 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:45:48.594 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:48.593+0000 7f8d62b25700 1 -- 172.21.15.134:0/3273086704 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f8d4c02ff60 msgr2=0x7f8d4c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:45:48.594 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:48.593+0000 7f8d62b25700 1 --2- 172.21.15.134:0/3273086704 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f8d4c02ff60 0x7f8d4c032420 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f8d480096a0 tx=0x7f8d48006b40).stop 2024-01-21T11:45:48.594 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:48.593+0000 7f8d62b25700 1 -- 172.21.15.134:0/3273086704 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d5c0f6510 msgr2=0x7f8d5c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:45:48.594 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:48.593+0000 7f8d62b25700 1 --2- 172.21.15.134:0/3273086704 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d5c0f6510 0x7f8d5c106730 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f8d50013510 tx=0x7f8d5000b040).stop 2024-01-21T11:45:48.594 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:48.593+0000 7f8d62b25700 1 -- 172.21.15.134:0/3273086704 shutdown_connections 2024-01-21T11:45:48.594 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:48.593+0000 7f8d62b25700 1 --2- 172.21.15.134:0/3273086704 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f8d4c02ff60 0x7f8d4c032420 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:45:48.594 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:48.593+0000 7f8d62b25700 1 --2- 172.21.15.134:0/3273086704 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d5c0f6510 0x7f8d5c106730 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:45:48.594 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:48.593+0000 7f8d62b25700 1 -- 172.21.15.134:0/3273086704 >> 172.21.15.134:0/3273086704 conn(0x7f8d5c0f1c70 msgr2=0x7f8d5c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:45:48.594 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:48.593+0000 7f8d62b25700 1 -- 172.21.15.134:0/3273086704 shutdown_connections 2024-01-21T11:45:48.594 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:48.593+0000 7f8d62b25700 1 -- 172.21.15.134:0/3273086704 wait complete. 2024-01-21T11:45:48.595 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:45:49.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:45:49 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:45:47.837578+0000 mgr.smithi029.blhxxk (mgr.14164) 114 : cluster [DBG] pgmap v107: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:45:49.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:45:49 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:45:48.592767+0000 mon.smithi029 (mon.0) 194 : audit [DBG] from='client.? 172.21.15.134:0/3273086704' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:45:49.965 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:45:49.965 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:45:51.084 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:51.083+0000 7ff36da48700 1 -- 172.21.15.134:0/3548885836 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3680f51a0 msgr2=0x7ff3680f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:45:51.084 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:51.083+0000 7ff36da48700 1 --2- 172.21.15.134:0/3548885836 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3680f51a0 0x7ff3680f55c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7ff35c009fb0 tx=0x7ff35c009b30).stop 2024-01-21T11:45:51.085 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:51.083+0000 7ff36da48700 1 -- 172.21.15.134:0/3548885836 shutdown_connections 2024-01-21T11:45:51.085 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:51.083+0000 7ff36da48700 1 --2- 172.21.15.134:0/3548885836 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3680f51a0 0x7ff3680f55c0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:45:51.085 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:51.083+0000 7ff36da48700 1 -- 172.21.15.134:0/3548885836 >> 172.21.15.134:0/3548885836 conn(0x7ff3680f09a0 msgr2=0x7ff3680f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:45:51.085 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:51.083+0000 7ff36da48700 1 -- 172.21.15.134:0/3548885836 shutdown_connections 2024-01-21T11:45:51.085 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:51.083+0000 7ff36da48700 1 -- 172.21.15.134:0/3548885836 wait complete. 2024-01-21T11:45:51.085 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:51.083+0000 7ff36da48700 1 Processor -- start 2024-01-21T11:45:51.085 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:51.083+0000 7ff36da48700 1 -- start start 2024-01-21T11:45:51.085 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:51.083+0000 7ff36da48700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3680f51a0 0x7ff3681091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:45:51.085 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:51.083+0000 7ff36da48700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff368060240 con 0x7ff3680f51a0 2024-01-21T11:45:51.085 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:51.084+0000 7ff36ca46700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3680f51a0 0x7ff3681091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:45:51.085 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:51.084+0000 7ff36ca46700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3680f51a0 0x7ff3681091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:42560/0 (socket says 172.21.15.134:42560) 2024-01-21T11:45:51.085 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:51.084+0000 7ff36ca46700 1 -- 172.21.15.134:0/451164877 learned_addr learned my addr 172.21.15.134:0/451164877 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:45:51.086 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:51.084+0000 7ff36ca46700 1 -- 172.21.15.134:0/451164877 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff35c005040 con 0x7ff3680f51a0 2024-01-21T11:45:51.086 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:51.084+0000 7ff36ca46700 1 --2- 172.21.15.134:0/451164877 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3680f51a0 0x7ff3681091b0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7ff35c013510 tx=0x7ff35c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:45:51.086 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:51.084+0000 7ff3657fa700 1 -- 172.21.15.134:0/451164877 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff35c009320 con 0x7ff3680f51a0 2024-01-21T11:45:51.086 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:51.084+0000 7ff36da48700 1 -- 172.21.15.134:0/451164877 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff368060440 con 0x7ff3680f51a0 2024-01-21T11:45:51.086 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:51.084+0000 7ff3657fa700 1 -- 172.21.15.134:0/451164877 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff35c013f00 con 0x7ff3680f51a0 2024-01-21T11:45:51.086 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:51.084+0000 7ff3657fa700 1 -- 172.21.15.134:0/451164877 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff35c014dc0 con 0x7ff3680f51a0 2024-01-21T11:45:51.086 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:51.084+0000 7ff36da48700 1 -- 172.21.15.134:0/451164877 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff368109860 con 0x7ff3680f51a0 2024-01-21T11:45:51.087 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:51.085+0000 7ff3657fa700 1 -- 172.21.15.134:0/451164877 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff35c014420 con 0x7ff3680f51a0 2024-01-21T11:45:51.087 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:51.085+0000 7ff36da48700 1 -- 172.21.15.134:0/451164877 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff368045ad0 con 0x7ff3680f51a0 2024-01-21T11:45:51.087 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:51.085+0000 7ff3657fa700 1 --2- 172.21.15.134:0/451164877 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7ff35802ff60 0x7ff358032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:45:51.087 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:51.085+0000 7ff3657fa700 1 -- 172.21.15.134:0/451164877 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff35c041030 con 0x7ff3680f51a0 2024-01-21T11:45:51.087 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:51.085+0000 7ff367fff700 1 --2- 172.21.15.134:0/451164877 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7ff35802ff60 0x7ff358032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:45:51.087 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:51.086+0000 7ff367fff700 1 --2- 172.21.15.134:0/451164877 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7ff35802ff60 0x7ff358032420 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7ff3540096a0 tx=0x7ff354006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:45:51.090 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:51.090+0000 7ff3657fa700 1 -- 172.21.15.134:0/451164877 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff35c0095f0 con 0x7ff3680f51a0 2024-01-21T11:45:51.294 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:51.294+0000 7ff36da48700 1 -- 172.21.15.134:0/451164877 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff36810a080 con 0x7ff3680f51a0 2024-01-21T11:45:51.295 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:51.295+0000 7ff3657fa700 1 -- 172.21.15.134:0/451164877 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff35c01cb10 con 0x7ff3680f51a0 2024-01-21T11:45:51.295 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:45:51.295 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:45:51.296 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:51.296+0000 7ff36da48700 1 -- 172.21.15.134:0/451164877 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7ff35802ff60 msgr2=0x7ff358032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:45:51.297 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:51.296+0000 7ff36da48700 1 --2- 172.21.15.134:0/451164877 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7ff35802ff60 0x7ff358032420 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7ff3540096a0 tx=0x7ff354006b40).stop 2024-01-21T11:45:51.297 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:51.296+0000 7ff36da48700 1 -- 172.21.15.134:0/451164877 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3680f51a0 msgr2=0x7ff3681091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:45:51.297 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:51.296+0000 7ff36da48700 1 --2- 172.21.15.134:0/451164877 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3680f51a0 0x7ff3681091b0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7ff35c013510 tx=0x7ff35c00b040).stop 2024-01-21T11:45:51.297 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:51.296+0000 7ff36da48700 1 -- 172.21.15.134:0/451164877 shutdown_connections 2024-01-21T11:45:51.297 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:51.296+0000 7ff36da48700 1 --2- 172.21.15.134:0/451164877 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7ff35802ff60 0x7ff358032420 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:45:51.297 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:51.296+0000 7ff36da48700 1 --2- 172.21.15.134:0/451164877 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3680f51a0 0x7ff3681091b0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:45:51.297 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:51.296+0000 7ff36da48700 1 -- 172.21.15.134:0/451164877 >> 172.21.15.134:0/451164877 conn(0x7ff3680f09a0 msgr2=0x7ff3680f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:45:51.297 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:51.296+0000 7ff36da48700 1 -- 172.21.15.134:0/451164877 shutdown_connections 2024-01-21T11:45:51.297 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:51.296+0000 7ff36da48700 1 -- 172.21.15.134:0/451164877 wait complete. 2024-01-21T11:45:51.297 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:45:51.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:45:51 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:45:49.837994+0000 mgr.smithi029.blhxxk (mgr.14164) 115 : cluster [DBG] pgmap v108: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:45:51.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:45:51 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:45:51.295945+0000 mon.smithi029 (mon.0) 195 : audit [DBG] from='client.? 172.21.15.134:0/451164877' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:45:52.730 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:45:52.731 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:45:53.865 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:53.863+0000 7fcb36493700 1 -- 172.21.15.134:0/1884130918 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcb300601e0 msgr2=0x7fcb300f9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:45:53.865 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:53.863+0000 7fcb36493700 1 --2- 172.21.15.134:0/1884130918 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcb300601e0 0x7fcb300f9190 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7fcb20009fb0 tx=0x7fcb20009b30).stop 2024-01-21T11:45:53.865 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:53.863+0000 7fcb36493700 1 -- 172.21.15.134:0/1884130918 shutdown_connections 2024-01-21T11:45:53.865 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:53.863+0000 7fcb36493700 1 --2- 172.21.15.134:0/1884130918 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcb300601e0 0x7fcb300f9190 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:45:53.865 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:53.863+0000 7fcb36493700 1 -- 172.21.15.134:0/1884130918 >> 172.21.15.134:0/1884130918 conn(0x7fcb300f0b70 msgr2=0x7fcb300f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:45:53.865 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:53.863+0000 7fcb36493700 1 -- 172.21.15.134:0/1884130918 shutdown_connections 2024-01-21T11:45:53.865 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:53.863+0000 7fcb36493700 1 -- 172.21.15.134:0/1884130918 wait complete. 2024-01-21T11:45:53.865 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:53.864+0000 7fcb36493700 1 Processor -- start 2024-01-21T11:45:53.865 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:53.864+0000 7fcb36493700 1 -- start start 2024-01-21T11:45:53.866 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:53.864+0000 7fcb36493700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcb300601e0 0x7fcb301066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:45:53.866 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:53.864+0000 7fcb36493700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb30106bd0 con 0x7fcb300601e0 2024-01-21T11:45:53.866 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:53.864+0000 7fcb2ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcb300601e0 0x7fcb301066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:45:53.866 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:53.864+0000 7fcb2ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcb300601e0 0x7fcb301066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:42578/0 (socket says 172.21.15.134:42578) 2024-01-21T11:45:53.866 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:53.864+0000 7fcb2ffff700 1 -- 172.21.15.134:0/88809585 learned_addr learned my addr 172.21.15.134:0/88809585 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:45:53.866 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:53.865+0000 7fcb2ffff700 1 -- 172.21.15.134:0/88809585 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb20005040 con 0x7fcb300601e0 2024-01-21T11:45:53.866 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:53.865+0000 7fcb2ffff700 1 --2- 172.21.15.134:0/88809585 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcb300601e0 0x7fcb301066e0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7fcb20004dd0 tx=0x7fcb20009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:45:53.866 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:53.865+0000 7fcb2cff9700 1 -- 172.21.15.134:0/88809585 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb20018070 con 0x7fcb300601e0 2024-01-21T11:45:53.866 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:53.865+0000 7fcb36493700 1 -- 172.21.15.134:0/88809585 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb30106dd0 con 0x7fcb300601e0 2024-01-21T11:45:53.866 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:53.865+0000 7fcb2cff9700 1 -- 172.21.15.134:0/88809585 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcb20013410 con 0x7fcb300601e0 2024-01-21T11:45:53.866 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:53.865+0000 7fcb2cff9700 1 -- 172.21.15.134:0/88809585 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb2001c3c0 con 0x7fcb300601e0 2024-01-21T11:45:53.866 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:53.865+0000 7fcb36493700 1 -- 172.21.15.134:0/88809585 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb30107260 con 0x7fcb300601e0 2024-01-21T11:45:53.867 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:53.866+0000 7fcb2cff9700 1 -- 172.21.15.134:0/88809585 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcb2001c630 con 0x7fcb300601e0 2024-01-21T11:45:53.867 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:53.866+0000 7fcb2cff9700 1 --2- 172.21.15.134:0/88809585 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fcb1802ff60 0x7fcb18032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:45:53.867 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:53.866+0000 7fcb2cff9700 1 -- 172.21.15.134:0/88809585 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcb200413a0 con 0x7fcb300601e0 2024-01-21T11:45:53.867 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:53.866+0000 7fcb36493700 1 -- 172.21.15.134:0/88809585 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb30045ad0 con 0x7fcb300601e0 2024-01-21T11:45:53.867 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:53.866+0000 7fcb2f7fe700 1 --2- 172.21.15.134:0/88809585 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fcb1802ff60 0x7fcb18032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:45:53.867 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:53.866+0000 7fcb2f7fe700 1 --2- 172.21.15.134:0/88809585 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fcb1802ff60 0x7fcb18032420 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7fcb240096a0 tx=0x7fcb24006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:45:53.871 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:53.871+0000 7fcb2cff9700 1 -- 172.21.15.134:0/88809585 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcb2001ca30 con 0x7fcb300601e0 2024-01-21T11:45:53.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:45:53 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:45:51.838306+0000 mgr.smithi029.blhxxk (mgr.14164) 116 : cluster [DBG] pgmap v109: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:45:54.079 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:54.078+0000 7fcb36493700 1 -- 172.21.15.134:0/88809585 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcb30107f40 con 0x7fcb300601e0 2024-01-21T11:45:54.079 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:54.079+0000 7fcb2cff9700 1 -- 172.21.15.134:0/88809585 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcb2001b9c0 con 0x7fcb300601e0 2024-01-21T11:45:54.080 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:45:54.080 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:45:54.081 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:54.081+0000 7fcb36493700 1 -- 172.21.15.134:0/88809585 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fcb1802ff60 msgr2=0x7fcb18032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:45:54.081 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:54.081+0000 7fcb36493700 1 --2- 172.21.15.134:0/88809585 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fcb1802ff60 0x7fcb18032420 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7fcb240096a0 tx=0x7fcb24006b40).stop 2024-01-21T11:45:54.081 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:54.081+0000 7fcb36493700 1 -- 172.21.15.134:0/88809585 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcb300601e0 msgr2=0x7fcb301066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:45:54.082 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:54.081+0000 7fcb36493700 1 --2- 172.21.15.134:0/88809585 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcb300601e0 0x7fcb301066e0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7fcb20004dd0 tx=0x7fcb20009b30).stop 2024-01-21T11:45:54.082 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:54.081+0000 7fcb36493700 1 -- 172.21.15.134:0/88809585 shutdown_connections 2024-01-21T11:45:54.082 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:54.081+0000 7fcb36493700 1 --2- 172.21.15.134:0/88809585 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fcb1802ff60 0x7fcb18032420 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:45:54.082 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:54.081+0000 7fcb36493700 1 --2- 172.21.15.134:0/88809585 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcb300601e0 0x7fcb301066e0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:45:54.082 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:54.081+0000 7fcb36493700 1 -- 172.21.15.134:0/88809585 >> 172.21.15.134:0/88809585 conn(0x7fcb300f0b70 msgr2=0x7fcb300f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:45:54.082 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:54.081+0000 7fcb36493700 1 -- 172.21.15.134:0/88809585 shutdown_connections 2024-01-21T11:45:54.082 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:54.081+0000 7fcb36493700 1 -- 172.21.15.134:0/88809585 wait complete. 2024-01-21T11:45:54.082 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:45:54.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:45:54 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:45:54.080442+0000 mon.smithi029 (mon.0) 196 : audit [DBG] from='client.? 172.21.15.134:0/88809585' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:45:55.448 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:45:55.449 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:45:55.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:45:55 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:45:53.838635+0000 mgr.smithi029.blhxxk (mgr.14164) 117 : cluster [DBG] pgmap v110: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:45:56.607 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:56.607+0000 7f8936c8a700 1 -- 172.21.15.134:0/992250675 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89300f62a0 msgr2=0x7f89300f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:45:56.608 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:56.607+0000 7f8936c8a700 1 --2- 172.21.15.134:0/992250675 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89300f62a0 0x7f89300f66c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7f8920009fb0 tx=0x7f8920009b30).stop 2024-01-21T11:45:56.608 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:56.607+0000 7f8936c8a700 1 -- 172.21.15.134:0/992250675 shutdown_connections 2024-01-21T11:45:56.608 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:56.607+0000 7f8936c8a700 1 --2- 172.21.15.134:0/992250675 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89300f62a0 0x7f89300f66c0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:45:56.608 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:56.607+0000 7f8936c8a700 1 -- 172.21.15.134:0/992250675 >> 172.21.15.134:0/992250675 conn(0x7f89300f1aa0 msgr2=0x7f89300f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:45:56.608 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:56.607+0000 7f8936c8a700 1 -- 172.21.15.134:0/992250675 shutdown_connections 2024-01-21T11:45:56.608 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:56.607+0000 7f8936c8a700 1 -- 172.21.15.134:0/992250675 wait complete. 2024-01-21T11:45:56.608 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:56.607+0000 7f8936c8a700 1 Processor -- start 2024-01-21T11:45:56.608 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:56.607+0000 7f8936c8a700 1 -- start start 2024-01-21T11:45:56.608 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:56.608+0000 7f8936c8a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89300f62a0 0x7f89301064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:45:56.608 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:56.608+0000 7f8936c8a700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89301069a0 con 0x7f89300f62a0 2024-01-21T11:45:56.610 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:56.608+0000 7f8935c88700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89300f62a0 0x7f89301064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:45:56.610 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:56.608+0000 7f8935c88700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89300f62a0 0x7f89301064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:51698/0 (socket says 172.21.15.134:51698) 2024-01-21T11:45:56.610 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:56.608+0000 7f8935c88700 1 -- 172.21.15.134:0/865735895 learned_addr learned my addr 172.21.15.134:0/865735895 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:45:56.610 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:56.608+0000 7f8935c88700 1 -- 172.21.15.134:0/865735895 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8920005040 con 0x7f89300f62a0 2024-01-21T11:45:56.610 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:56.608+0000 7f8935c88700 1 --2- 172.21.15.134:0/865735895 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89300f62a0 0x7f89301064b0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f8920013510 tx=0x7f892000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:45:56.611 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:56.609+0000 7f89267fc700 1 -- 172.21.15.134:0/865735895 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8920009320 con 0x7f89300f62a0 2024-01-21T11:45:56.611 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:56.609+0000 7f89267fc700 1 -- 172.21.15.134:0/865735895 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8920013f00 con 0x7f89300f62a0 2024-01-21T11:45:56.611 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:56.609+0000 7f8936c8a700 1 -- 172.21.15.134:0/865735895 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8930106ba0 con 0x7f89300f62a0 2024-01-21T11:45:56.611 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:56.609+0000 7f89267fc700 1 -- 172.21.15.134:0/865735895 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8920014dc0 con 0x7f89300f62a0 2024-01-21T11:45:56.611 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:56.609+0000 7f8936c8a700 1 -- 172.21.15.134:0/865735895 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8930107030 con 0x7f89300f62a0 2024-01-21T11:45:56.611 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:56.609+0000 7f89267fc700 1 -- 172.21.15.134:0/865735895 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f89200095f0 con 0x7f89300f62a0 2024-01-21T11:45:56.611 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:56.609+0000 7f8936c8a700 1 -- 172.21.15.134:0/865735895 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8930045ad0 con 0x7f89300f62a0 2024-01-21T11:45:56.611 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:56.610+0000 7f89267fc700 1 --2- 172.21.15.134:0/865735895 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f891c02ff60 0x7f891c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:45:56.611 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:56.610+0000 7f89267fc700 1 -- 172.21.15.134:0/865735895 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8920041100 con 0x7f89300f62a0 2024-01-21T11:45:56.611 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:56.610+0000 7f8935487700 1 --2- 172.21.15.134:0/865735895 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f891c02ff60 0x7f891c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:45:56.611 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:56.610+0000 7f8935487700 1 --2- 172.21.15.134:0/865735895 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f891c02ff60 0x7f891c032420 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f892c0096a0 tx=0x7f892c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:45:56.614 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:56.614+0000 7f89267fc700 1 -- 172.21.15.134:0/865735895 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8920014420 con 0x7f89300f62a0 2024-01-21T11:45:56.821 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:56.821+0000 7f8936c8a700 1 -- 172.21.15.134:0/865735895 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8930107ee0 con 0x7f89300f62a0 2024-01-21T11:45:56.822 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:56.822+0000 7f89267fc700 1 -- 172.21.15.134:0/865735895 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8920010230 con 0x7f89300f62a0 2024-01-21T11:45:56.822 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:45:56.822 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:45:56.823 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:56.823+0000 7f8936c8a700 1 -- 172.21.15.134:0/865735895 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f891c02ff60 msgr2=0x7f891c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:45:56.824 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:56.823+0000 7f8936c8a700 1 --2- 172.21.15.134:0/865735895 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f891c02ff60 0x7f891c032420 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f892c0096a0 tx=0x7f892c006b40).stop 2024-01-21T11:45:56.824 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:56.823+0000 7f8936c8a700 1 -- 172.21.15.134:0/865735895 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89300f62a0 msgr2=0x7f89301064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:45:56.824 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:56.823+0000 7f8936c8a700 1 --2- 172.21.15.134:0/865735895 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89300f62a0 0x7f89301064b0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f8920013510 tx=0x7f892000b040).stop 2024-01-21T11:45:56.824 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:56.823+0000 7f8936c8a700 1 -- 172.21.15.134:0/865735895 shutdown_connections 2024-01-21T11:45:56.824 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:56.823+0000 7f8936c8a700 1 --2- 172.21.15.134:0/865735895 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f891c02ff60 0x7f891c032420 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:45:56.824 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:56.823+0000 7f8936c8a700 1 --2- 172.21.15.134:0/865735895 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89300f62a0 0x7f89301064b0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:45:56.824 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:56.823+0000 7f8936c8a700 1 -- 172.21.15.134:0/865735895 >> 172.21.15.134:0/865735895 conn(0x7f89300f1aa0 msgr2=0x7f89300f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:45:56.824 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:56.823+0000 7f8936c8a700 1 -- 172.21.15.134:0/865735895 shutdown_connections 2024-01-21T11:45:56.824 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:56.823+0000 7f8936c8a700 1 -- 172.21.15.134:0/865735895 wait complete. 2024-01-21T11:45:56.824 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:45:57.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:45:57 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:45:55.839014+0000 mgr.smithi029.blhxxk (mgr.14164) 118 : cluster [DBG] pgmap v111: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:45:57.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:45:57 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:45:56.822746+0000 mon.smithi029 (mon.0) 197 : audit [DBG] from='client.? 172.21.15.134:0/865735895' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:45:58.204 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:45:58.204 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:45:59.376 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:59.375+0000 7fd7982b6700 1 -- 172.21.15.134:0/1531870081 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7900f5410 msgr2=0x7fd7900f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:45:59.376 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:59.375+0000 7fd7982b6700 1 --2- 172.21.15.134:0/1531870081 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7900f5410 0x7fd7900f5830 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7fd780009fb0 tx=0x7fd780009b30).stop 2024-01-21T11:45:59.376 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:59.375+0000 7fd7982b6700 1 -- 172.21.15.134:0/1531870081 shutdown_connections 2024-01-21T11:45:59.376 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:59.375+0000 7fd7982b6700 1 --2- 172.21.15.134:0/1531870081 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7900f5410 0x7fd7900f5830 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:45:59.376 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:59.375+0000 7fd7982b6700 1 -- 172.21.15.134:0/1531870081 >> 172.21.15.134:0/1531870081 conn(0x7fd7900f0b70 msgr2=0x7fd7900f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:45:59.376 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:59.375+0000 7fd7982b6700 1 -- 172.21.15.134:0/1531870081 shutdown_connections 2024-01-21T11:45:59.376 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:59.375+0000 7fd7982b6700 1 -- 172.21.15.134:0/1531870081 wait complete. 2024-01-21T11:45:59.378 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:59.376+0000 7fd7982b6700 1 Processor -- start 2024-01-21T11:45:59.378 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:59.376+0000 7fd7982b6700 1 -- start start 2024-01-21T11:45:59.378 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:59.376+0000 7fd7982b6700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7900f5410 0x7fd790105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:45:59.378 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:59.376+0000 7fd7982b6700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd790105b20 con 0x7fd7900f5410 2024-01-21T11:45:59.378 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:59.376+0000 7fd796052700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7900f5410 0x7fd790105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:45:59.378 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:59.376+0000 7fd796052700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7900f5410 0x7fd790105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:51706/0 (socket says 172.21.15.134:51706) 2024-01-21T11:45:59.378 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:59.376+0000 7fd796052700 1 -- 172.21.15.134:0/3299527316 learned_addr learned my addr 172.21.15.134:0/3299527316 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:45:59.378 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:59.377+0000 7fd796052700 1 -- 172.21.15.134:0/3299527316 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd780005040 con 0x7fd7900f5410 2024-01-21T11:45:59.378 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:59.377+0000 7fd796052700 1 --2- 172.21.15.134:0/3299527316 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7900f5410 0x7fd790105630 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7fd780013510 tx=0x7fd78000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:45:59.378 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:59.377+0000 7fd786ffd700 1 -- 172.21.15.134:0/3299527316 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd780009320 con 0x7fd7900f5410 2024-01-21T11:45:59.378 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:59.377+0000 7fd7982b6700 1 -- 172.21.15.134:0/3299527316 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd790105d20 con 0x7fd7900f5410 2024-01-21T11:45:59.378 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:59.377+0000 7fd786ffd700 1 -- 172.21.15.134:0/3299527316 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd780013f00 con 0x7fd7900f5410 2024-01-21T11:45:59.378 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:59.377+0000 7fd786ffd700 1 -- 172.21.15.134:0/3299527316 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd780014d90 con 0x7fd7900f5410 2024-01-21T11:45:59.379 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:59.377+0000 7fd7982b6700 1 -- 172.21.15.134:0/3299527316 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7901061b0 con 0x7fd7900f5410 2024-01-21T11:45:59.379 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:59.378+0000 7fd786ffd700 1 -- 172.21.15.134:0/3299527316 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd7800095f0 con 0x7fd7900f5410 2024-01-21T11:45:59.379 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:59.378+0000 7fd7982b6700 1 -- 172.21.15.134:0/3299527316 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd790045ad0 con 0x7fd7900f5410 2024-01-21T11:45:59.380 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:59.378+0000 7fd786ffd700 1 --2- 172.21.15.134:0/3299527316 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fd77c02ff60 0x7fd77c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:45:59.380 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:59.378+0000 7fd786ffd700 1 -- 172.21.15.134:0/3299527316 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd780040fc0 con 0x7fd7900f5410 2024-01-21T11:45:59.380 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:59.378+0000 7fd795851700 1 --2- 172.21.15.134:0/3299527316 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fd77c02ff60 0x7fd77c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:45:59.380 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:59.379+0000 7fd795851700 1 --2- 172.21.15.134:0/3299527316 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fd77c02ff60 0x7fd77c032420 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7fd78c0096a0 tx=0x7fd78c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:45:59.382 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:59.383+0000 7fd786ffd700 1 -- 172.21.15.134:0/3299527316 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd780014420 con 0x7fd7900f5410 2024-01-21T11:45:59.592 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:59.592+0000 7fd7982b6700 1 -- 172.21.15.134:0/3299527316 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd7900fae60 con 0x7fd7900f5410 2024-01-21T11:45:59.593 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:59.593+0000 7fd786ffd700 1 -- 172.21.15.134:0/3299527316 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd780010230 con 0x7fd7900f5410 2024-01-21T11:45:59.593 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:45:59.593 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:45:59.595 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:59.595+0000 7fd7982b6700 1 -- 172.21.15.134:0/3299527316 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fd77c02ff60 msgr2=0x7fd77c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:45:59.595 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:59.595+0000 7fd7982b6700 1 --2- 172.21.15.134:0/3299527316 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fd77c02ff60 0x7fd77c032420 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7fd78c0096a0 tx=0x7fd78c006b40).stop 2024-01-21T11:45:59.595 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:59.595+0000 7fd7982b6700 1 -- 172.21.15.134:0/3299527316 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7900f5410 msgr2=0x7fd790105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:45:59.595 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:59.595+0000 7fd7982b6700 1 --2- 172.21.15.134:0/3299527316 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7900f5410 0x7fd790105630 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7fd780013510 tx=0x7fd78000b040).stop 2024-01-21T11:45:59.596 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:59.595+0000 7fd7982b6700 1 -- 172.21.15.134:0/3299527316 shutdown_connections 2024-01-21T11:45:59.596 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:59.595+0000 7fd7982b6700 1 --2- 172.21.15.134:0/3299527316 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fd77c02ff60 0x7fd77c032420 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:45:59.596 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:59.595+0000 7fd7982b6700 1 --2- 172.21.15.134:0/3299527316 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7900f5410 0x7fd790105630 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:45:59.596 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:59.595+0000 7fd7982b6700 1 -- 172.21.15.134:0/3299527316 >> 172.21.15.134:0/3299527316 conn(0x7fd7900f0b70 msgr2=0x7fd7900f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:45:59.596 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:59.595+0000 7fd7982b6700 1 -- 172.21.15.134:0/3299527316 shutdown_connections 2024-01-21T11:45:59.596 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:45:59.595+0000 7fd7982b6700 1 -- 172.21.15.134:0/3299527316 wait complete. 2024-01-21T11:45:59.596 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:45:59.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:45:59 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:45:57.839339+0000 mgr.smithi029.blhxxk (mgr.14164) 119 : cluster [DBG] pgmap v112: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:46:00.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:46:00 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:45:59.594354+0000 mon.smithi029 (mon.0) 198 : audit [DBG] from='client.? 172.21.15.134:0/3299527316' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:46:00.991 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:46:00.991 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:46:01.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:46:01 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:45:59.839591+0000 mgr.smithi029.blhxxk (mgr.14164) 120 : cluster [DBG] pgmap v113: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:46:02.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:02.198+0000 7faa7359e700 1 -- 172.21.15.134:0/3872196010 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa740f56a0 msgr2=0x7faa740f5ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:46:02.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:02.198+0000 7faa7359e700 1 --2- 172.21.15.134:0/3872196010 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa740f56a0 0x7faa740f5ac0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7faa68009fb0 tx=0x7faa68009b30).stop 2024-01-21T11:46:02.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:02.198+0000 7faa7359e700 1 -- 172.21.15.134:0/3872196010 shutdown_connections 2024-01-21T11:46:02.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:02.198+0000 7faa7359e700 1 --2- 172.21.15.134:0/3872196010 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa740f56a0 0x7faa740f5ac0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:46:02.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:02.198+0000 7faa7359e700 1 -- 172.21.15.134:0/3872196010 >> 172.21.15.134:0/3872196010 conn(0x7faa740effc0 msgr2=0x7faa740f23e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:46:02.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:02.198+0000 7faa7359e700 1 -- 172.21.15.134:0/3872196010 shutdown_connections 2024-01-21T11:46:02.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:02.198+0000 7faa7359e700 1 -- 172.21.15.134:0/3872196010 wait complete. 2024-01-21T11:46:02.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:02.199+0000 7faa7359e700 1 Processor -- start 2024-01-21T11:46:02.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:02.199+0000 7faa7359e700 1 -- start start 2024-01-21T11:46:02.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:02.199+0000 7faa7359e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa740f56a0 0x7faa740fd150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:46:02.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:02.199+0000 7faa7359e700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa740fb5d0 con 0x7faa740f56a0 2024-01-21T11:46:02.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:02.199+0000 7faa7259c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa740f56a0 0x7faa740fd150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:46:02.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:02.199+0000 7faa7259c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa740f56a0 0x7faa740fd150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:51730/0 (socket says 172.21.15.134:51730) 2024-01-21T11:46:02.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:02.199+0000 7faa7259c700 1 -- 172.21.15.134:0/2092971538 learned_addr learned my addr 172.21.15.134:0/2092971538 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:46:02.202 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:02.200+0000 7faa7259c700 1 -- 172.21.15.134:0/2092971538 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa68005040 con 0x7faa740f56a0 2024-01-21T11:46:02.202 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:02.200+0000 7faa7259c700 1 --2- 172.21.15.134:0/2092971538 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa740f56a0 0x7faa740fd150 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7faa68013510 tx=0x7faa6800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:46:02.202 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:02.200+0000 7faa62ffd700 1 -- 172.21.15.134:0/2092971538 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faa68009320 con 0x7faa740f56a0 2024-01-21T11:46:02.202 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:02.200+0000 7faa7359e700 1 -- 172.21.15.134:0/2092971538 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa740fb770 con 0x7faa740f56a0 2024-01-21T11:46:02.202 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:02.200+0000 7faa62ffd700 1 -- 172.21.15.134:0/2092971538 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faa68013f00 con 0x7faa740f56a0 2024-01-21T11:46:02.202 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:02.200+0000 7faa7359e700 1 -- 172.21.15.134:0/2092971538 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa740fbc00 con 0x7faa740f56a0 2024-01-21T11:46:02.202 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:02.200+0000 7faa62ffd700 1 -- 172.21.15.134:0/2092971538 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faa68014d90 con 0x7faa740f56a0 2024-01-21T11:46:02.202 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:02.201+0000 7faa62ffd700 1 -- 172.21.15.134:0/2092971538 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7faa680263a0 con 0x7faa740f56a0 2024-01-21T11:46:02.202 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:02.201+0000 7faa62ffd700 1 --2- 172.21.15.134:0/2092971538 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7faa5c02ff60 0x7faa5c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:46:02.202 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:02.201+0000 7faa62ffd700 1 -- 172.21.15.134:0/2092971538 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7faa680410b0 con 0x7faa740f56a0 2024-01-21T11:46:02.203 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:02.201+0000 7faa7359e700 1 -- 172.21.15.134:0/2092971538 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa74059100 con 0x7faa740f56a0 2024-01-21T11:46:02.203 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:02.201+0000 7faa71d9b700 1 --2- 172.21.15.134:0/2092971538 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7faa5c02ff60 0x7faa5c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:46:02.203 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:02.202+0000 7faa71d9b700 1 --2- 172.21.15.134:0/2092971538 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7faa5c02ff60 0x7faa5c032420 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7faa640096a0 tx=0x7faa64006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:46:02.206 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:02.207+0000 7faa62ffd700 1 -- 172.21.15.134:0/2092971538 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7faa68012920 con 0x7faa740f56a0 2024-01-21T11:46:02.411 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:02.411+0000 7faa7359e700 1 -- 172.21.15.134:0/2092971538 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7faa74045ad0 con 0x7faa740f56a0 2024-01-21T11:46:02.412 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:02.412+0000 7faa62ffd700 1 -- 172.21.15.134:0/2092971538 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7faa68012330 con 0x7faa740f56a0 2024-01-21T11:46:02.412 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:46:02.412 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:46:02.414 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:02.413+0000 7faa7359e700 1 -- 172.21.15.134:0/2092971538 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7faa5c02ff60 msgr2=0x7faa5c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:46:02.414 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:02.413+0000 7faa7359e700 1 --2- 172.21.15.134:0/2092971538 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7faa5c02ff60 0x7faa5c032420 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7faa640096a0 tx=0x7faa64006b40).stop 2024-01-21T11:46:02.414 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:02.413+0000 7faa7359e700 1 -- 172.21.15.134:0/2092971538 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa740f56a0 msgr2=0x7faa740fd150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:46:02.414 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:02.413+0000 7faa7359e700 1 --2- 172.21.15.134:0/2092971538 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa740f56a0 0x7faa740fd150 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7faa68013510 tx=0x7faa6800b040).stop 2024-01-21T11:46:02.414 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:02.414+0000 7faa7359e700 1 -- 172.21.15.134:0/2092971538 shutdown_connections 2024-01-21T11:46:02.415 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:02.414+0000 7faa7359e700 1 --2- 172.21.15.134:0/2092971538 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7faa5c02ff60 0x7faa5c032420 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:46:02.415 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:02.414+0000 7faa7359e700 1 --2- 172.21.15.134:0/2092971538 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa740f56a0 0x7faa740fd150 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:46:02.415 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:02.414+0000 7faa7359e700 1 -- 172.21.15.134:0/2092971538 >> 172.21.15.134:0/2092971538 conn(0x7faa740effc0 msgr2=0x7faa740f4af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:46:02.415 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:02.414+0000 7faa7359e700 1 -- 172.21.15.134:0/2092971538 shutdown_connections 2024-01-21T11:46:02.415 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:02.414+0000 7faa7359e700 1 -- 172.21.15.134:0/2092971538 wait complete. 2024-01-21T11:46:02.415 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:46:02.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:46:02 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:46:01.940825+0000 mon.smithi029 (mon.0) 199 : audit [INF] from='mgr.14164 172.21.15.29:0/3252518221' entity='mgr.smithi029.blhxxk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi029.blhxxk/mirror_snapshot_schedule"}]: dispatch 2024-01-21T11:46:02.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:46:02 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:46:01.942964+0000 mon.smithi029 (mon.0) 200 : audit [INF] from='mgr.14164 172.21.15.29:0/3252518221' entity='mgr.smithi029.blhxxk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi029.blhxxk/trash_purge_schedule"}]: dispatch 2024-01-21T11:46:02.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:46:02 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:46:02.412996+0000 mon.smithi029 (mon.0) 201 : audit [DBG] from='client.? 172.21.15.134:0/2092971538' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:46:03.791 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:46:03.791 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:46:03.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:46:03 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:46:01.839930+0000 mgr.smithi029.blhxxk (mgr.14164) 121 : cluster [DBG] pgmap v114: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:46:04.995 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:04.994+0000 7fb5f0f32700 1 -- 172.21.15.134:0/3192453454 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5ec0f5410 msgr2=0x7fb5ec0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:46:04.995 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:04.994+0000 7fb5f0f32700 1 --2- 172.21.15.134:0/3192453454 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5ec0f5410 0x7fb5ec0f5830 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7fb5dc009fb0 tx=0x7fb5dc009b30).stop 2024-01-21T11:46:04.995 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:04.995+0000 7fb5f0f32700 1 -- 172.21.15.134:0/3192453454 shutdown_connections 2024-01-21T11:46:04.995 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:04.995+0000 7fb5f0f32700 1 --2- 172.21.15.134:0/3192453454 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5ec0f5410 0x7fb5ec0f5830 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:46:04.995 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:04.995+0000 7fb5f0f32700 1 -- 172.21.15.134:0/3192453454 >> 172.21.15.134:0/3192453454 conn(0x7fb5ec0f0b70 msgr2=0x7fb5ec0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:46:04.995 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:04.995+0000 7fb5f0f32700 1 -- 172.21.15.134:0/3192453454 shutdown_connections 2024-01-21T11:46:04.995 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:04.995+0000 7fb5f0f32700 1 -- 172.21.15.134:0/3192453454 wait complete. 2024-01-21T11:46:04.995 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:04.995+0000 7fb5f0f32700 1 Processor -- start 2024-01-21T11:46:04.996 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:04.995+0000 7fb5f0f32700 1 -- start start 2024-01-21T11:46:04.996 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:04.995+0000 7fb5f0f32700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5ec0f5410 0x7fb5ec1055c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:46:04.996 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:04.995+0000 7fb5f0f32700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5ec105ab0 con 0x7fb5ec0f5410 2024-01-21T11:46:04.996 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:04.996+0000 7fb5ea59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5ec0f5410 0x7fb5ec1055c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:46:04.996 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:04.996+0000 7fb5ea59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5ec0f5410 0x7fb5ec1055c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:50202/0 (socket says 172.21.15.134:50202) 2024-01-21T11:46:04.996 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:04.996+0000 7fb5ea59c700 1 -- 172.21.15.134:0/846276148 learned_addr learned my addr 172.21.15.134:0/846276148 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:46:04.997 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:04.996+0000 7fb5ea59c700 1 -- 172.21.15.134:0/846276148 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5dc005040 con 0x7fb5ec0f5410 2024-01-21T11:46:04.997 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:04.996+0000 7fb5ea59c700 1 --2- 172.21.15.134:0/846276148 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5ec0f5410 0x7fb5ec1055c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7fb5dc013510 tx=0x7fb5dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:46:04.998 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:04.996+0000 7fb5daffd700 1 -- 172.21.15.134:0/846276148 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb5dc009320 con 0x7fb5ec0f5410 2024-01-21T11:46:04.998 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:04.996+0000 7fb5f0f32700 1 -- 172.21.15.134:0/846276148 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5ec105cb0 con 0x7fb5ec0f5410 2024-01-21T11:46:04.998 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:04.996+0000 7fb5daffd700 1 -- 172.21.15.134:0/846276148 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb5dc013f00 con 0x7fb5ec0f5410 2024-01-21T11:46:04.998 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:04.996+0000 7fb5daffd700 1 -- 172.21.15.134:0/846276148 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb5dc014d90 con 0x7fb5ec0f5410 2024-01-21T11:46:04.998 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:04.996+0000 7fb5f0f32700 1 -- 172.21.15.134:0/846276148 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5ec106140 con 0x7fb5ec0f5410 2024-01-21T11:46:04.998 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:04.997+0000 7fb5daffd700 1 -- 172.21.15.134:0/846276148 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb5dc0095f0 con 0x7fb5ec0f5410 2024-01-21T11:46:04.998 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:04.997+0000 7fb5f0f32700 1 -- 172.21.15.134:0/846276148 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5ec045ad0 con 0x7fb5ec0f5410 2024-01-21T11:46:04.998 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:04.997+0000 7fb5daffd700 1 --2- 172.21.15.134:0/846276148 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fb5d402ff60 0x7fb5d4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:46:04.998 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:04.997+0000 7fb5daffd700 1 -- 172.21.15.134:0/846276148 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb5dc042d90 con 0x7fb5ec0f5410 2024-01-21T11:46:04.998 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:04.998+0000 7fb5e9d9b700 1 --2- 172.21.15.134:0/846276148 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fb5d402ff60 0x7fb5d4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:46:04.998 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:04.998+0000 7fb5e9d9b700 1 --2- 172.21.15.134:0/846276148 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fb5d402ff60 0x7fb5d4032420 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fb5e00096a0 tx=0x7fb5e0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:46:05.001 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:05.002+0000 7fb5daffd700 1 -- 172.21.15.134:0/846276148 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb5dc011290 con 0x7fb5ec0f5410 2024-01-21T11:46:05.205 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:05.206+0000 7fb5f0f32700 1 -- 172.21.15.134:0/846276148 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb5ec106f90 con 0x7fb5ec0f5410 2024-01-21T11:46:05.206 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:05.207+0000 7fb5daffd700 1 -- 172.21.15.134:0/846276148 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb5dc011290 con 0x7fb5ec0f5410 2024-01-21T11:46:05.206 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:46:05.206 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:46:05.208 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:05.208+0000 7fb5f0f32700 1 -- 172.21.15.134:0/846276148 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fb5d402ff60 msgr2=0x7fb5d4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:46:05.208 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:05.208+0000 7fb5f0f32700 1 --2- 172.21.15.134:0/846276148 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fb5d402ff60 0x7fb5d4032420 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fb5e00096a0 tx=0x7fb5e0006b40).stop 2024-01-21T11:46:05.208 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:05.208+0000 7fb5f0f32700 1 -- 172.21.15.134:0/846276148 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5ec0f5410 msgr2=0x7fb5ec1055c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:46:05.208 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:05.208+0000 7fb5f0f32700 1 --2- 172.21.15.134:0/846276148 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5ec0f5410 0x7fb5ec1055c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7fb5dc013510 tx=0x7fb5dc00b040).stop 2024-01-21T11:46:05.208 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:05.209+0000 7fb5f0f32700 1 -- 172.21.15.134:0/846276148 shutdown_connections 2024-01-21T11:46:05.208 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:05.209+0000 7fb5f0f32700 1 --2- 172.21.15.134:0/846276148 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fb5d402ff60 0x7fb5d4032420 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:46:05.208 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:05.209+0000 7fb5f0f32700 1 --2- 172.21.15.134:0/846276148 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5ec0f5410 0x7fb5ec1055c0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:46:05.209 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:05.209+0000 7fb5f0f32700 1 -- 172.21.15.134:0/846276148 >> 172.21.15.134:0/846276148 conn(0x7fb5ec0f0b70 msgr2=0x7fb5ec0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:46:05.209 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:05.209+0000 7fb5f0f32700 1 -- 172.21.15.134:0/846276148 shutdown_connections 2024-01-21T11:46:05.209 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:05.209+0000 7fb5f0f32700 1 -- 172.21.15.134:0/846276148 wait complete. 2024-01-21T11:46:05.209 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:46:05.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:46:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:46:03.840324+0000 mgr.smithi029.blhxxk (mgr.14164) 122 : cluster [DBG] pgmap v115: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:46:05.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:46:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:46:05.207197+0000 mon.smithi029 (mon.0) 202 : audit [DBG] from='client.? 172.21.15.134:0/846276148' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:46:06.550 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:46:06.550 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:46:07.676 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:07.676+0000 7f18c7147700 1 -- 172.21.15.134:0/3625193866 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f18c00f62a0 msgr2=0x7f18c00f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:46:07.676 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:07.676+0000 7f18c7147700 1 --2- 172.21.15.134:0/3625193866 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f18c00f62a0 0x7f18c00f66c0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7f18bc009fb0 tx=0x7f18bc009b30).stop 2024-01-21T11:46:07.676 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:07.676+0000 7f18c7147700 1 -- 172.21.15.134:0/3625193866 shutdown_connections 2024-01-21T11:46:07.676 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:07.676+0000 7f18c7147700 1 --2- 172.21.15.134:0/3625193866 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f18c00f62a0 0x7f18c00f66c0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:46:07.676 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:07.676+0000 7f18c7147700 1 -- 172.21.15.134:0/3625193866 >> 172.21.15.134:0/3625193866 conn(0x7f18c00f1aa0 msgr2=0x7f18c00f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:46:07.677 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:07.676+0000 7f18c7147700 1 -- 172.21.15.134:0/3625193866 shutdown_connections 2024-01-21T11:46:07.677 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:07.676+0000 7f18c7147700 1 -- 172.21.15.134:0/3625193866 wait complete. 2024-01-21T11:46:07.678 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:07.676+0000 7f18c7147700 1 Processor -- start 2024-01-21T11:46:07.678 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:07.677+0000 7f18c7147700 1 -- start start 2024-01-21T11:46:07.678 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:07.677+0000 7f18c7147700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f18c00f62a0 0x7f18c01064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:46:07.678 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:07.677+0000 7f18c7147700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18c01069a0 con 0x7f18c00f62a0 2024-01-21T11:46:07.678 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:07.677+0000 7f18c6145700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f18c00f62a0 0x7f18c01064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:46:07.678 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:07.677+0000 7f18c6145700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f18c00f62a0 0x7f18c01064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:50210/0 (socket says 172.21.15.134:50210) 2024-01-21T11:46:07.678 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:07.677+0000 7f18c6145700 1 -- 172.21.15.134:0/2602582594 learned_addr learned my addr 172.21.15.134:0/2602582594 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:46:07.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:07.677+0000 7f18c6145700 1 -- 172.21.15.134:0/2602582594 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18bc005040 con 0x7f18c00f62a0 2024-01-21T11:46:07.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:07.678+0000 7f18c6145700 1 --2- 172.21.15.134:0/2602582594 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f18c00f62a0 0x7f18c01064b0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f18bc013510 tx=0x7f18bc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:46:07.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:07.678+0000 7f18b6ffd700 1 -- 172.21.15.134:0/2602582594 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f18bc009320 con 0x7f18c00f62a0 2024-01-21T11:46:07.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:07.678+0000 7f18c7147700 1 -- 172.21.15.134:0/2602582594 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18c0106ba0 con 0x7f18c00f62a0 2024-01-21T11:46:07.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:07.678+0000 7f18b6ffd700 1 -- 172.21.15.134:0/2602582594 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f18bc013f00 con 0x7f18c00f62a0 2024-01-21T11:46:07.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:07.678+0000 7f18b6ffd700 1 -- 172.21.15.134:0/2602582594 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f18bc014dc0 con 0x7f18c00f62a0 2024-01-21T11:46:07.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:07.678+0000 7f18c7147700 1 -- 172.21.15.134:0/2602582594 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18c0107030 con 0x7f18c00f62a0 2024-01-21T11:46:07.680 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:07.679+0000 7f18b6ffd700 1 -- 172.21.15.134:0/2602582594 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f18bc0095f0 con 0x7f18c00f62a0 2024-01-21T11:46:07.680 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:07.679+0000 7f18c7147700 1 -- 172.21.15.134:0/2602582594 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18c0045ad0 con 0x7f18c00f62a0 2024-01-21T11:46:07.680 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:07.679+0000 7f18b6ffd700 1 --2- 172.21.15.134:0/2602582594 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f18ac02ff60 0x7f18ac032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:46:07.680 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:07.679+0000 7f18b6ffd700 1 -- 172.21.15.134:0/2602582594 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f18bc041000 con 0x7f18c00f62a0 2024-01-21T11:46:07.680 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:07.679+0000 7f18c5944700 1 --2- 172.21.15.134:0/2602582594 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f18ac02ff60 0x7f18ac032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:46:07.680 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:07.679+0000 7f18c5944700 1 --2- 172.21.15.134:0/2602582594 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f18ac02ff60 0x7f18ac032420 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f18b00096a0 tx=0x7f18b0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:46:07.683 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:07.683+0000 7f18b6ffd700 1 -- 172.21.15.134:0/2602582594 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f18bc01c790 con 0x7f18c00f62a0 2024-01-21T11:46:07.890 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:07.890+0000 7f18c7147700 1 -- 172.21.15.134:0/2602582594 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f18c0107e80 con 0x7f18c00f62a0 2024-01-21T11:46:07.891 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:07.891+0000 7f18b6ffd700 1 -- 172.21.15.134:0/2602582594 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f18bc01c790 con 0x7f18c00f62a0 2024-01-21T11:46:07.891 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:46:07.891 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:46:07.892 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:07.892+0000 7f18c7147700 1 -- 172.21.15.134:0/2602582594 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f18ac02ff60 msgr2=0x7f18ac032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:46:07.893 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:07.892+0000 7f18c7147700 1 --2- 172.21.15.134:0/2602582594 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f18ac02ff60 0x7f18ac032420 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f18b00096a0 tx=0x7f18b0006b40).stop 2024-01-21T11:46:07.893 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:07.892+0000 7f18c7147700 1 -- 172.21.15.134:0/2602582594 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f18c00f62a0 msgr2=0x7f18c01064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:46:07.893 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:07.892+0000 7f18c7147700 1 --2- 172.21.15.134:0/2602582594 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f18c00f62a0 0x7f18c01064b0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f18bc013510 tx=0x7f18bc00b040).stop 2024-01-21T11:46:07.893 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:07.892+0000 7f18c7147700 1 -- 172.21.15.134:0/2602582594 shutdown_connections 2024-01-21T11:46:07.893 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:07.892+0000 7f18c7147700 1 --2- 172.21.15.134:0/2602582594 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f18ac02ff60 0x7f18ac032420 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:46:07.893 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:07.892+0000 7f18c7147700 1 --2- 172.21.15.134:0/2602582594 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f18c00f62a0 0x7f18c01064b0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:46:07.893 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:07.892+0000 7f18c7147700 1 -- 172.21.15.134:0/2602582594 >> 172.21.15.134:0/2602582594 conn(0x7f18c00f1aa0 msgr2=0x7f18c00f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:46:07.893 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:07.892+0000 7f18c7147700 1 -- 172.21.15.134:0/2602582594 shutdown_connections 2024-01-21T11:46:07.893 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:07.892+0000 7f18c7147700 1 -- 172.21.15.134:0/2602582594 wait complete. 2024-01-21T11:46:07.893 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:46:07.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:46:07 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:46:05.840690+0000 mgr.smithi029.blhxxk (mgr.14164) 123 : cluster [DBG] pgmap v116: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:46:08.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:46:08 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:46:07.891779+0000 mon.smithi029 (mon.0) 203 : audit [DBG] from='client.? 172.21.15.134:0/2602582594' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:46:09.288 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:46:09.289 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:46:09.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:46:09 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:46:07.841032+0000 mgr.smithi029.blhxxk (mgr.14164) 124 : cluster [DBG] pgmap v117: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:46:10.391 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:10.391+0000 7fdb090e3700 1 -- 172.21.15.134:0/4021968579 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb040f5410 msgr2=0x7fdb040f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:46:10.391 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:10.391+0000 7fdb090e3700 1 --2- 172.21.15.134:0/4021968579 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb040f5410 0x7fdb040f5830 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7fdaec009fb0 tx=0x7fdaec009b30).stop 2024-01-21T11:46:10.392 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:10.391+0000 7fdb090e3700 1 -- 172.21.15.134:0/4021968579 shutdown_connections 2024-01-21T11:46:10.392 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:10.391+0000 7fdb090e3700 1 --2- 172.21.15.134:0/4021968579 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb040f5410 0x7fdb040f5830 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:46:10.392 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:10.391+0000 7fdb090e3700 1 -- 172.21.15.134:0/4021968579 >> 172.21.15.134:0/4021968579 conn(0x7fdb040f0b70 msgr2=0x7fdb040f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:46:10.392 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:10.391+0000 7fdb090e3700 1 -- 172.21.15.134:0/4021968579 shutdown_connections 2024-01-21T11:46:10.392 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:10.391+0000 7fdb090e3700 1 -- 172.21.15.134:0/4021968579 wait complete. 2024-01-21T11:46:10.393 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:10.392+0000 7fdb090e3700 1 Processor -- start 2024-01-21T11:46:10.393 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:10.392+0000 7fdb090e3700 1 -- start start 2024-01-21T11:46:10.393 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:10.392+0000 7fdb090e3700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb040f5410 0x7fdb040fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:46:10.393 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:10.392+0000 7fdb090e3700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb040fb820 con 0x7fdb040f5410 2024-01-21T11:46:10.393 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:10.392+0000 7fdb02d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb040f5410 0x7fdb040fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:46:10.394 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:10.392+0000 7fdb02d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb040f5410 0x7fdb040fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:50238/0 (socket says 172.21.15.134:50238) 2024-01-21T11:46:10.394 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:10.392+0000 7fdb02d9d700 1 -- 172.21.15.134:0/2228226166 learned_addr learned my addr 172.21.15.134:0/2228226166 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:46:10.394 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:10.392+0000 7fdb02d9d700 1 -- 172.21.15.134:0/2228226166 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaec005040 con 0x7fdb040f5410 2024-01-21T11:46:10.394 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:10.393+0000 7fdb02d9d700 1 --2- 172.21.15.134:0/2228226166 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb040f5410 0x7fdb040fd3a0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7fdaec013510 tx=0x7fdaec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:46:10.394 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:10.393+0000 7fdafb7fe700 1 -- 172.21.15.134:0/2228226166 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdaec009320 con 0x7fdb040f5410 2024-01-21T11:46:10.394 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:10.393+0000 7fdb090e3700 1 -- 172.21.15.134:0/2228226166 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb040fb9c0 con 0x7fdb040f5410 2024-01-21T11:46:10.394 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:10.393+0000 7fdafb7fe700 1 -- 172.21.15.134:0/2228226166 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdaec013f00 con 0x7fdb040f5410 2024-01-21T11:46:10.394 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:10.393+0000 7fdafb7fe700 1 -- 172.21.15.134:0/2228226166 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdaec014d90 con 0x7fdb040f5410 2024-01-21T11:46:10.394 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:10.393+0000 7fdb090e3700 1 -- 172.21.15.134:0/2228226166 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb040fbe50 con 0x7fdb040f5410 2024-01-21T11:46:10.395 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:10.394+0000 7fdafb7fe700 1 -- 172.21.15.134:0/2228226166 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdaec0095f0 con 0x7fdb040f5410 2024-01-21T11:46:10.395 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:10.394+0000 7fdb090e3700 1 -- 172.21.15.134:0/2228226166 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb04059100 con 0x7fdb040f5410 2024-01-21T11:46:10.395 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:10.394+0000 7fdafb7fe700 1 --2- 172.21.15.134:0/2228226166 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fdaf002ff60 0x7fdaf0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:46:10.395 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:10.394+0000 7fdafb7fe700 1 -- 172.21.15.134:0/2228226166 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdaec041000 con 0x7fdb040f5410 2024-01-21T11:46:10.395 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:10.394+0000 7fdb0259c700 1 --2- 172.21.15.134:0/2228226166 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fdaf002ff60 0x7fdaf0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:46:10.395 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:10.394+0000 7fdb0259c700 1 --2- 172.21.15.134:0/2228226166 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fdaf002ff60 0x7fdaf0032420 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fdaf40096a0 tx=0x7fdaf4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:46:10.398 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:10.398+0000 7fdafb7fe700 1 -- 172.21.15.134:0/2228226166 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdaec014420 con 0x7fdb040f5410 2024-01-21T11:46:10.605 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:10.605+0000 7fdb090e3700 1 -- 172.21.15.134:0/2228226166 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdb040fae60 con 0x7fdb040f5410 2024-01-21T11:46:10.606 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:10.606+0000 7fdafb7fe700 1 -- 172.21.15.134:0/2228226166 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdaec010230 con 0x7fdb040f5410 2024-01-21T11:46:10.606 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:46:10.606 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:46:10.608 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:10.607+0000 7fdb090e3700 1 -- 172.21.15.134:0/2228226166 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fdaf002ff60 msgr2=0x7fdaf0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:46:10.608 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:10.607+0000 7fdb090e3700 1 --2- 172.21.15.134:0/2228226166 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fdaf002ff60 0x7fdaf0032420 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fdaf40096a0 tx=0x7fdaf4006b40).stop 2024-01-21T11:46:10.608 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:10.607+0000 7fdb090e3700 1 -- 172.21.15.134:0/2228226166 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb040f5410 msgr2=0x7fdb040fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:46:10.608 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:10.607+0000 7fdb090e3700 1 --2- 172.21.15.134:0/2228226166 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb040f5410 0x7fdb040fd3a0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7fdaec013510 tx=0x7fdaec00b040).stop 2024-01-21T11:46:10.608 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:10.607+0000 7fdb090e3700 1 -- 172.21.15.134:0/2228226166 shutdown_connections 2024-01-21T11:46:10.608 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:10.607+0000 7fdb090e3700 1 --2- 172.21.15.134:0/2228226166 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fdaf002ff60 0x7fdaf0032420 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:46:10.608 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:10.607+0000 7fdb090e3700 1 --2- 172.21.15.134:0/2228226166 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb040f5410 0x7fdb040fd3a0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:46:10.608 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:10.607+0000 7fdb090e3700 1 -- 172.21.15.134:0/2228226166 >> 172.21.15.134:0/2228226166 conn(0x7fdb040f0b70 msgr2=0x7fdb0405fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:46:10.608 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:10.608+0000 7fdb090e3700 1 -- 172.21.15.134:0/2228226166 shutdown_connections 2024-01-21T11:46:10.608 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:10.608+0000 7fdb090e3700 1 -- 172.21.15.134:0/2228226166 wait complete. 2024-01-21T11:46:10.609 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:46:11.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:46:11 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:46:09.841363+0000 mgr.smithi029.blhxxk (mgr.14164) 125 : cluster [DBG] pgmap v118: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:46:11.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:46:11 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:46:10.607018+0000 mon.smithi029 (mon.0) 204 : audit [DBG] from='client.? 172.21.15.134:0/2228226166' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:46:11.986 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:46:11.986 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:46:13.160 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:13.159+0000 7f0ac0962700 1 -- 172.21.15.134:0/3274488507 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0abc06b920 msgr2=0x7f0abc069ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:46:13.160 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:13.159+0000 7f0ac0962700 1 --2- 172.21.15.134:0/3274488507 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0abc06b920 0x7f0abc069ec0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7f0aac009fb0 tx=0x7f0aac009b30).stop 2024-01-21T11:46:13.160 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:13.159+0000 7f0ac0962700 1 -- 172.21.15.134:0/3274488507 shutdown_connections 2024-01-21T11:46:13.160 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:13.159+0000 7f0ac0962700 1 --2- 172.21.15.134:0/3274488507 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0abc06b920 0x7f0abc069ec0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:46:13.160 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:13.159+0000 7f0ac0962700 1 -- 172.21.15.134:0/3274488507 >> 172.21.15.134:0/3274488507 conn(0x7f0abc0f1ac0 msgr2=0x7f0abc0f3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:46:13.160 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:13.159+0000 7f0ac0962700 1 -- 172.21.15.134:0/3274488507 shutdown_connections 2024-01-21T11:46:13.160 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:13.159+0000 7f0ac0962700 1 -- 172.21.15.134:0/3274488507 wait complete. 2024-01-21T11:46:13.160 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:13.159+0000 7f0ac0962700 1 Processor -- start 2024-01-21T11:46:13.160 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:13.160+0000 7f0ac0962700 1 -- start start 2024-01-21T11:46:13.160 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:13.160+0000 7f0ac0962700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0abc06b920 0x7f0abc1086e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:46:13.160 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:13.160+0000 7f0ac0962700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0abc108bd0 con 0x7f0abc06b920 2024-01-21T11:46:13.162 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:13.160+0000 7f0abb7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0abc06b920 0x7f0abc1086e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:46:13.162 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:13.160+0000 7f0abb7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0abc06b920 0x7f0abc1086e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:50264/0 (socket says 172.21.15.134:50264) 2024-01-21T11:46:13.162 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:13.160+0000 7f0abb7fe700 1 -- 172.21.15.134:0/2338292622 learned_addr learned my addr 172.21.15.134:0/2338292622 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:46:13.162 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:13.160+0000 7f0abb7fe700 1 -- 172.21.15.134:0/2338292622 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0aac005040 con 0x7f0abc06b920 2024-01-21T11:46:13.162 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:13.160+0000 7f0abb7fe700 1 --2- 172.21.15.134:0/2338292622 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0abc06b920 0x7f0abc1086e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f0aac013510 tx=0x7f0aac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:46:13.162 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:13.161+0000 7f0aa3fff700 1 -- 172.21.15.134:0/2338292622 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0aac009320 con 0x7f0abc06b920 2024-01-21T11:46:13.162 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:13.161+0000 7f0ac0962700 1 -- 172.21.15.134:0/2338292622 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0abc108dd0 con 0x7f0abc06b920 2024-01-21T11:46:13.162 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:13.161+0000 7f0aa3fff700 1 -- 172.21.15.134:0/2338292622 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0aac013f00 con 0x7f0abc06b920 2024-01-21T11:46:13.162 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:13.161+0000 7f0aa3fff700 1 -- 172.21.15.134:0/2338292622 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0aac014d90 con 0x7f0abc06b920 2024-01-21T11:46:13.163 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:13.161+0000 7f0ac0962700 1 -- 172.21.15.134:0/2338292622 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0abc109260 con 0x7f0abc06b920 2024-01-21T11:46:13.163 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:13.161+0000 7f0aa3fff700 1 -- 172.21.15.134:0/2338292622 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0aac014420 con 0x7f0abc06b920 2024-01-21T11:46:13.163 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:13.162+0000 7f0ac0962700 1 -- 172.21.15.134:0/2338292622 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0abc045ad0 con 0x7f0abc06b920 2024-01-21T11:46:13.163 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:13.162+0000 7f0aa3fff700 1 --2- 172.21.15.134:0/2338292622 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f0aa402ff60 0x7f0aa4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:46:13.163 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:13.162+0000 7f0aa3fff700 1 -- 172.21.15.134:0/2338292622 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0aac041110 con 0x7f0abc06b920 2024-01-21T11:46:13.163 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:13.162+0000 7f0abaffd700 1 --2- 172.21.15.134:0/2338292622 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f0aa402ff60 0x7f0aa4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:46:13.163 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:13.162+0000 7f0abaffd700 1 --2- 172.21.15.134:0/2338292622 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f0aa402ff60 0x7f0aa4032420 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f0ab00096a0 tx=0x7f0ab0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:46:13.167 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:13.167+0000 7f0aa3fff700 1 -- 172.21.15.134:0/2338292622 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0aac0095f0 con 0x7f0abc06b920 2024-01-21T11:46:13.374 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:13.374+0000 7f0ac0962700 1 -- 172.21.15.134:0/2338292622 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0abc10a0b0 con 0x7f0abc06b920 2024-01-21T11:46:13.375 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:13.375+0000 7f0aa3fff700 1 -- 172.21.15.134:0/2338292622 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0aac01db70 con 0x7f0abc06b920 2024-01-21T11:46:13.375 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:46:13.375 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:46:13.376 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:13.376+0000 7f0ac0962700 1 -- 172.21.15.134:0/2338292622 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f0aa402ff60 msgr2=0x7f0aa4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:46:13.377 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:13.376+0000 7f0ac0962700 1 --2- 172.21.15.134:0/2338292622 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f0aa402ff60 0x7f0aa4032420 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f0ab00096a0 tx=0x7f0ab0006b40).stop 2024-01-21T11:46:13.377 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:13.376+0000 7f0ac0962700 1 -- 172.21.15.134:0/2338292622 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0abc06b920 msgr2=0x7f0abc1086e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:46:13.377 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:13.376+0000 7f0ac0962700 1 --2- 172.21.15.134:0/2338292622 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0abc06b920 0x7f0abc1086e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f0aac013510 tx=0x7f0aac00b040).stop 2024-01-21T11:46:13.377 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:13.376+0000 7f0ac0962700 1 -- 172.21.15.134:0/2338292622 shutdown_connections 2024-01-21T11:46:13.377 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:13.376+0000 7f0ac0962700 1 --2- 172.21.15.134:0/2338292622 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f0aa402ff60 0x7f0aa4032420 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:46:13.377 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:13.376+0000 7f0ac0962700 1 --2- 172.21.15.134:0/2338292622 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0abc06b920 0x7f0abc1086e0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:46:13.377 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:13.376+0000 7f0ac0962700 1 -- 172.21.15.134:0/2338292622 >> 172.21.15.134:0/2338292622 conn(0x7f0abc0f1ac0 msgr2=0x7f0abc0f2580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:46:13.377 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:13.376+0000 7f0ac0962700 1 -- 172.21.15.134:0/2338292622 shutdown_connections 2024-01-21T11:46:13.377 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:13.376+0000 7f0ac0962700 1 -- 172.21.15.134:0/2338292622 wait complete. 2024-01-21T11:46:13.378 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:46:13.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:46:13 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:46:11.841717+0000 mgr.smithi029.blhxxk (mgr.14164) 126 : cluster [DBG] pgmap v119: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:46:13.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:46:13 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:46:13.375889+0000 mon.smithi029 (mon.0) 205 : audit [DBG] from='client.? 172.21.15.134:0/2338292622' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:46:14.755 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:46:14.756 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:46:15.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:46:15 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:46:13.842135+0000 mgr.smithi029.blhxxk (mgr.14164) 127 : cluster [DBG] pgmap v120: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:46:15.925 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:15.924+0000 7f5126556700 1 -- 172.21.15.134:0/2006647041 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51200f47c0 msgr2=0x7f51200f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:46:15.925 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:15.924+0000 7f5126556700 1 --2- 172.21.15.134:0/2006647041 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51200f47c0 0x7f51200f4be0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7f5110009fb0 tx=0x7f5110009b30).stop 2024-01-21T11:46:15.925 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:15.925+0000 7f5126556700 1 -- 172.21.15.134:0/2006647041 shutdown_connections 2024-01-21T11:46:15.925 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:15.925+0000 7f5126556700 1 --2- 172.21.15.134:0/2006647041 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51200f47c0 0x7f51200f4be0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:46:15.926 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:15.925+0000 7f5126556700 1 -- 172.21.15.134:0/2006647041 >> 172.21.15.134:0/2006647041 conn(0x7f51200f02e0 msgr2=0x7f51200f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:46:15.926 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:15.925+0000 7f5126556700 1 -- 172.21.15.134:0/2006647041 shutdown_connections 2024-01-21T11:46:15.926 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:15.925+0000 7f5126556700 1 -- 172.21.15.134:0/2006647041 wait complete. 2024-01-21T11:46:15.926 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:15.925+0000 7f5126556700 1 Processor -- start 2024-01-21T11:46:15.926 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:15.925+0000 7f5126556700 1 -- start start 2024-01-21T11:46:15.926 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:15.925+0000 7f5126556700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51200f47c0 0x7f5120106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:46:15.926 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:15.925+0000 7f5126556700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5120106bf0 con 0x7f51200f47c0 2024-01-21T11:46:15.927 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:15.926+0000 7f511ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51200f47c0 0x7f5120106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:46:15.927 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:15.926+0000 7f511ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51200f47c0 0x7f5120106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:40204/0 (socket says 172.21.15.134:40204) 2024-01-21T11:46:15.927 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:15.926+0000 7f511ffff700 1 -- 172.21.15.134:0/2937844698 learned_addr learned my addr 172.21.15.134:0/2937844698 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:46:15.927 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:15.926+0000 7f511ffff700 1 -- 172.21.15.134:0/2937844698 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5110005040 con 0x7f51200f47c0 2024-01-21T11:46:15.927 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:15.926+0000 7f511ffff700 1 --2- 172.21.15.134:0/2937844698 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51200f47c0 0x7f5120106700 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f5110013510 tx=0x7f511000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:46:15.927 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:15.926+0000 7f511cff9700 1 -- 172.21.15.134:0/2937844698 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5110009320 con 0x7f51200f47c0 2024-01-21T11:46:15.927 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:15.926+0000 7f5126556700 1 -- 172.21.15.134:0/2937844698 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5120106df0 con 0x7f51200f47c0 2024-01-21T11:46:15.927 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:15.926+0000 7f511cff9700 1 -- 172.21.15.134:0/2937844698 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5110013f00 con 0x7f51200f47c0 2024-01-21T11:46:15.927 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:15.926+0000 7f511cff9700 1 -- 172.21.15.134:0/2937844698 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5110014d90 con 0x7f51200f47c0 2024-01-21T11:46:15.927 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:15.926+0000 7f5126556700 1 -- 172.21.15.134:0/2937844698 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5120107280 con 0x7f51200f47c0 2024-01-21T11:46:15.928 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:15.927+0000 7f511cff9700 1 -- 172.21.15.134:0/2937844698 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5110014420 con 0x7f51200f47c0 2024-01-21T11:46:15.928 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:15.927+0000 7f5126556700 1 -- 172.21.15.134:0/2937844698 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5120045ad0 con 0x7f51200f47c0 2024-01-21T11:46:15.928 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:15.927+0000 7f511cff9700 1 --2- 172.21.15.134:0/2937844698 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f510802ff60 0x7f5108032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:46:15.928 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:15.927+0000 7f511cff9700 1 -- 172.21.15.134:0/2937844698 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5110041050 con 0x7f51200f47c0 2024-01-21T11:46:15.929 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:15.928+0000 7f511f7fe700 1 --2- 172.21.15.134:0/2937844698 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f510802ff60 0x7f5108032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:46:15.929 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:15.928+0000 7f511f7fe700 1 --2- 172.21.15.134:0/2937844698 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f510802ff60 0x7f5108032420 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f51140096a0 tx=0x7f5114006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:46:15.932 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:15.932+0000 7f511cff9700 1 -- 172.21.15.134:0/2937844698 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f51100095f0 con 0x7f51200f47c0 2024-01-21T11:46:16.136 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:16.136+0000 7f5126556700 1 -- 172.21.15.134:0/2937844698 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f51201080d0 con 0x7f51200f47c0 2024-01-21T11:46:16.137 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:16.137+0000 7f511cff9700 1 -- 172.21.15.134:0/2937844698 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f511001db70 con 0x7f51200f47c0 2024-01-21T11:46:16.137 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:46:16.137 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:46:16.138 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:16.138+0000 7f5126556700 1 -- 172.21.15.134:0/2937844698 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f510802ff60 msgr2=0x7f5108032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:46:16.139 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:16.138+0000 7f5126556700 1 --2- 172.21.15.134:0/2937844698 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f510802ff60 0x7f5108032420 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f51140096a0 tx=0x7f5114006b40).stop 2024-01-21T11:46:16.139 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:16.138+0000 7f5126556700 1 -- 172.21.15.134:0/2937844698 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51200f47c0 msgr2=0x7f5120106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:46:16.139 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:16.138+0000 7f5126556700 1 --2- 172.21.15.134:0/2937844698 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51200f47c0 0x7f5120106700 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f5110013510 tx=0x7f511000b040).stop 2024-01-21T11:46:16.139 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:16.138+0000 7f5126556700 1 -- 172.21.15.134:0/2937844698 shutdown_connections 2024-01-21T11:46:16.139 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:16.138+0000 7f5126556700 1 --2- 172.21.15.134:0/2937844698 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f510802ff60 0x7f5108032420 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:46:16.139 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:16.138+0000 7f5126556700 1 --2- 172.21.15.134:0/2937844698 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51200f47c0 0x7f5120106700 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:46:16.139 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:16.138+0000 7f5126556700 1 -- 172.21.15.134:0/2937844698 >> 172.21.15.134:0/2937844698 conn(0x7f51200f02e0 msgr2=0x7f51200f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:46:16.139 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:16.138+0000 7f5126556700 1 -- 172.21.15.134:0/2937844698 shutdown_connections 2024-01-21T11:46:16.139 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:16.138+0000 7f5126556700 1 -- 172.21.15.134:0/2937844698 wait complete. 2024-01-21T11:46:16.140 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:46:16.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:46:16 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:46:16.137888+0000 mon.smithi029 (mon.0) 206 : audit [DBG] from='client.? 172.21.15.134:0/2937844698' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:46:17.514 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:46:17.514 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:46:17.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:46:17 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:46:15.842430+0000 mgr.smithi029.blhxxk (mgr.14164) 128 : cluster [DBG] pgmap v121: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:46:18.642 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:18.641+0000 7fc4c7be0700 1 -- 172.21.15.134:0/1170608105 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc4c0060280 msgr2=0x7fc4c00f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:46:18.643 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:18.641+0000 7fc4c7be0700 1 --2- 172.21.15.134:0/1170608105 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc4c0060280 0x7fc4c00f90f0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7fc4bc009fb0 tx=0x7fc4bc009b30).stop 2024-01-21T11:46:18.643 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:18.641+0000 7fc4c7be0700 1 -- 172.21.15.134:0/1170608105 shutdown_connections 2024-01-21T11:46:18.643 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:18.641+0000 7fc4c7be0700 1 --2- 172.21.15.134:0/1170608105 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc4c0060280 0x7fc4c00f90f0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:46:18.643 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:18.641+0000 7fc4c7be0700 1 -- 172.21.15.134:0/1170608105 >> 172.21.15.134:0/1170608105 conn(0x7fc4c00f0b70 msgr2=0x7fc4c00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:46:18.643 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:18.641+0000 7fc4c7be0700 1 -- 172.21.15.134:0/1170608105 shutdown_connections 2024-01-21T11:46:18.643 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:18.641+0000 7fc4c7be0700 1 -- 172.21.15.134:0/1170608105 wait complete. 2024-01-21T11:46:18.643 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:18.641+0000 7fc4c7be0700 1 Processor -- start 2024-01-21T11:46:18.643 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:18.641+0000 7fc4c7be0700 1 -- start start 2024-01-21T11:46:18.643 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:18.641+0000 7fc4c7be0700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc4c0060280 0x7fc4c00f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:46:18.643 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:18.641+0000 7fc4c7be0700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4c00f4ff0 con 0x7fc4c0060280 2024-01-21T11:46:18.643 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:18.642+0000 7fc4c597c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc4c0060280 0x7fc4c00f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:46:18.643 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:18.642+0000 7fc4c597c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc4c0060280 0x7fc4c00f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:40228/0 (socket says 172.21.15.134:40228) 2024-01-21T11:46:18.644 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:18.642+0000 7fc4c597c700 1 -- 172.21.15.134:0/162835791 learned_addr learned my addr 172.21.15.134:0/162835791 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:46:18.644 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:18.642+0000 7fc4c597c700 1 -- 172.21.15.134:0/162835791 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4bc005040 con 0x7fc4c0060280 2024-01-21T11:46:18.644 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:18.642+0000 7fc4c597c700 1 --2- 172.21.15.134:0/162835791 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc4c0060280 0x7fc4c00f6b70 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7fc4bc013510 tx=0x7fc4bc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:46:18.644 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:18.642+0000 7fc4b67fc700 1 -- 172.21.15.134:0/162835791 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc4bc009320 con 0x7fc4c0060280 2024-01-21T11:46:18.644 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:18.642+0000 7fc4c7be0700 1 -- 172.21.15.134:0/162835791 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4c00f5190 con 0x7fc4c0060280 2024-01-21T11:46:18.644 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:18.642+0000 7fc4b67fc700 1 -- 172.21.15.134:0/162835791 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc4bc013f00 con 0x7fc4c0060280 2024-01-21T11:46:18.644 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:18.642+0000 7fc4b67fc700 1 -- 172.21.15.134:0/162835791 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc4bc014dc0 con 0x7fc4c0060280 2024-01-21T11:46:18.644 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:18.643+0000 7fc4c7be0700 1 -- 172.21.15.134:0/162835791 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4c00f5620 con 0x7fc4c0060280 2024-01-21T11:46:18.645 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:18.643+0000 7fc4b67fc700 1 -- 172.21.15.134:0/162835791 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc4bc0095f0 con 0x7fc4c0060280 2024-01-21T11:46:18.645 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:18.643+0000 7fc4b67fc700 1 --2- 172.21.15.134:0/162835791 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fc4ac02ff60 0x7fc4ac032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:46:18.645 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:18.643+0000 7fc4c7be0700 1 -- 172.21.15.134:0/162835791 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4c0059100 con 0x7fc4c0060280 2024-01-21T11:46:18.645 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:18.644+0000 7fc4b67fc700 1 -- 172.21.15.134:0/162835791 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc4bc041000 con 0x7fc4c0060280 2024-01-21T11:46:18.645 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:18.644+0000 7fc4c517b700 1 --2- 172.21.15.134:0/162835791 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fc4ac02ff60 0x7fc4ac032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:46:18.645 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:18.644+0000 7fc4c517b700 1 --2- 172.21.15.134:0/162835791 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fc4ac02ff60 0x7fc4ac032420 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7fc4b00096a0 tx=0x7fc4b0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:46:18.648 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:18.648+0000 7fc4b67fc700 1 -- 172.21.15.134:0/162835791 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc4bc06d2b0 con 0x7fc4c0060280 2024-01-21T11:46:18.853 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:18.853+0000 7fc4c7be0700 1 -- 172.21.15.134:0/162835791 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc4c0045ad0 con 0x7fc4c0060280 2024-01-21T11:46:18.854 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:18.854+0000 7fc4b67fc700 1 -- 172.21.15.134:0/162835791 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc4bc01c790 con 0x7fc4c0060280 2024-01-21T11:46:18.854 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:46:18.854 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:46:18.855 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:18.855+0000 7fc4c7be0700 1 -- 172.21.15.134:0/162835791 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fc4ac02ff60 msgr2=0x7fc4ac032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:46:18.856 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:18.855+0000 7fc4c7be0700 1 --2- 172.21.15.134:0/162835791 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fc4ac02ff60 0x7fc4ac032420 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7fc4b00096a0 tx=0x7fc4b0006b40).stop 2024-01-21T11:46:18.856 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:18.855+0000 7fc4c7be0700 1 -- 172.21.15.134:0/162835791 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc4c0060280 msgr2=0x7fc4c00f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:46:18.856 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:18.855+0000 7fc4c7be0700 1 --2- 172.21.15.134:0/162835791 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc4c0060280 0x7fc4c00f6b70 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7fc4bc013510 tx=0x7fc4bc00b040).stop 2024-01-21T11:46:18.856 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:18.855+0000 7fc4c7be0700 1 -- 172.21.15.134:0/162835791 shutdown_connections 2024-01-21T11:46:18.856 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:18.855+0000 7fc4c7be0700 1 --2- 172.21.15.134:0/162835791 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fc4ac02ff60 0x7fc4ac032420 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:46:18.856 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:18.855+0000 7fc4c7be0700 1 --2- 172.21.15.134:0/162835791 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc4c0060280 0x7fc4c00f6b70 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:46:18.856 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:18.855+0000 7fc4c7be0700 1 -- 172.21.15.134:0/162835791 >> 172.21.15.134:0/162835791 conn(0x7fc4c00f0b70 msgr2=0x7fc4c00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:46:18.856 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:18.855+0000 7fc4c7be0700 1 -- 172.21.15.134:0/162835791 shutdown_connections 2024-01-21T11:46:18.856 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:18.855+0000 7fc4c7be0700 1 -- 172.21.15.134:0/162835791 wait complete. 2024-01-21T11:46:18.856 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:46:19.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:46:19 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:46:17.842768+0000 mgr.smithi029.blhxxk (mgr.14164) 129 : cluster [DBG] pgmap v122: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:46:19.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:46:19 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:46:18.854939+0000 mon.smithi029 (mon.0) 207 : audit [DBG] from='client.? 172.21.15.134:0/162835791' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:46:20.234 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:46:20.234 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:46:21.342 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:21.341+0000 7faa64b88700 1 -- 172.21.15.134:0/2285686547 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa600f5410 msgr2=0x7faa600f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:46:21.342 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:21.341+0000 7faa64b88700 1 --2- 172.21.15.134:0/2285686547 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa600f5410 0x7faa600f5830 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7faa54009fb0 tx=0x7faa54009b30).stop 2024-01-21T11:46:21.342 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:21.341+0000 7faa64b88700 1 -- 172.21.15.134:0/2285686547 shutdown_connections 2024-01-21T11:46:21.342 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:21.341+0000 7faa64b88700 1 --2- 172.21.15.134:0/2285686547 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa600f5410 0x7faa600f5830 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:46:21.342 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:21.341+0000 7faa64b88700 1 -- 172.21.15.134:0/2285686547 >> 172.21.15.134:0/2285686547 conn(0x7faa600f0b70 msgr2=0x7faa600f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:46:21.342 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:21.341+0000 7faa64b88700 1 -- 172.21.15.134:0/2285686547 shutdown_connections 2024-01-21T11:46:21.342 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:21.341+0000 7faa64b88700 1 -- 172.21.15.134:0/2285686547 wait complete. 2024-01-21T11:46:21.345 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:21.342+0000 7faa64b88700 1 Processor -- start 2024-01-21T11:46:21.345 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:21.342+0000 7faa64b88700 1 -- start start 2024-01-21T11:46:21.345 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:21.342+0000 7faa64b88700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa600f5410 0x7faa60109400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:46:21.345 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:21.342+0000 7faa64b88700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa60060240 con 0x7faa600f5410 2024-01-21T11:46:21.345 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:21.342+0000 7faa5e59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa600f5410 0x7faa60109400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:46:21.346 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:21.342+0000 7faa5e59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa600f5410 0x7faa60109400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:40248/0 (socket says 172.21.15.134:40248) 2024-01-21T11:46:21.346 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:21.342+0000 7faa5e59c700 1 -- 172.21.15.134:0/1108041428 learned_addr learned my addr 172.21.15.134:0/1108041428 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:46:21.346 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:21.343+0000 7faa5e59c700 1 -- 172.21.15.134:0/1108041428 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa54005040 con 0x7faa600f5410 2024-01-21T11:46:21.346 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:21.343+0000 7faa5e59c700 1 --2- 172.21.15.134:0/1108041428 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa600f5410 0x7faa60109400 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7faa54013510 tx=0x7faa5400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:46:21.346 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:21.343+0000 7faa4effd700 1 -- 172.21.15.134:0/1108041428 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faa54009320 con 0x7faa600f5410 2024-01-21T11:46:21.346 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:21.343+0000 7faa64b88700 1 -- 172.21.15.134:0/1108041428 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa60060440 con 0x7faa600f5410 2024-01-21T11:46:21.346 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:21.343+0000 7faa4effd700 1 -- 172.21.15.134:0/1108041428 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faa54013f00 con 0x7faa600f5410 2024-01-21T11:46:21.346 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:21.343+0000 7faa4effd700 1 -- 172.21.15.134:0/1108041428 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faa54014d90 con 0x7faa600f5410 2024-01-21T11:46:21.346 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:21.343+0000 7faa64b88700 1 -- 172.21.15.134:0/1108041428 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa60109ab0 con 0x7faa600f5410 2024-01-21T11:46:21.346 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:21.344+0000 7faa64b88700 1 -- 172.21.15.134:0/1108041428 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa60045ad0 con 0x7faa600f5410 2024-01-21T11:46:21.346 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:21.344+0000 7faa4effd700 1 -- 172.21.15.134:0/1108041428 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7faa540095f0 con 0x7faa600f5410 2024-01-21T11:46:21.346 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:21.344+0000 7faa4effd700 1 --2- 172.21.15.134:0/1108041428 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7faa4802ff60 0x7faa48032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:46:21.346 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:21.344+0000 7faa4effd700 1 -- 172.21.15.134:0/1108041428 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7faa54040fe0 con 0x7faa600f5410 2024-01-21T11:46:21.346 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:21.344+0000 7faa5dd9b700 1 --2- 172.21.15.134:0/1108041428 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7faa4802ff60 0x7faa48032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:46:21.347 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:21.345+0000 7faa5dd9b700 1 --2- 172.21.15.134:0/1108041428 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7faa4802ff60 0x7faa48032420 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7faa500096a0 tx=0x7faa50006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:46:21.349 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:21.349+0000 7faa4effd700 1 -- 172.21.15.134:0/1108041428 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7faa54010230 con 0x7faa600f5410 2024-01-21T11:46:21.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:21.555+0000 7faa64b88700 1 -- 172.21.15.134:0/1108041428 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7faa600fae60 con 0x7faa600f5410 2024-01-21T11:46:21.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:21.556+0000 7faa4effd700 1 -- 172.21.15.134:0/1108041428 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7faa54010230 con 0x7faa600f5410 2024-01-21T11:46:21.556 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:46:21.556 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:46:21.558 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:21.557+0000 7faa64b88700 1 -- 172.21.15.134:0/1108041428 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7faa4802ff60 msgr2=0x7faa48032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:46:21.558 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:21.557+0000 7faa64b88700 1 --2- 172.21.15.134:0/1108041428 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7faa4802ff60 0x7faa48032420 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7faa500096a0 tx=0x7faa50006b40).stop 2024-01-21T11:46:21.558 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:21.557+0000 7faa64b88700 1 -- 172.21.15.134:0/1108041428 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa600f5410 msgr2=0x7faa60109400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:46:21.558 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:21.557+0000 7faa64b88700 1 --2- 172.21.15.134:0/1108041428 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa600f5410 0x7faa60109400 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7faa54013510 tx=0x7faa5400b040).stop 2024-01-21T11:46:21.558 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:21.557+0000 7faa64b88700 1 -- 172.21.15.134:0/1108041428 shutdown_connections 2024-01-21T11:46:21.558 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:21.557+0000 7faa64b88700 1 --2- 172.21.15.134:0/1108041428 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7faa4802ff60 0x7faa48032420 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:46:21.558 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:21.557+0000 7faa64b88700 1 --2- 172.21.15.134:0/1108041428 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa600f5410 0x7faa60109400 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:46:21.558 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:21.557+0000 7faa64b88700 1 -- 172.21.15.134:0/1108041428 >> 172.21.15.134:0/1108041428 conn(0x7faa600f0b70 msgr2=0x7faa600f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:46:21.558 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:21.557+0000 7faa64b88700 1 -- 172.21.15.134:0/1108041428 shutdown_connections 2024-01-21T11:46:21.558 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:21.557+0000 7faa64b88700 1 -- 172.21.15.134:0/1108041428 wait complete. 2024-01-21T11:46:21.559 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:46:21.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:46:21 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:46:19.843073+0000 mgr.smithi029.blhxxk (mgr.14164) 130 : cluster [DBG] pgmap v123: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:46:21.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:46:21 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:46:21.556888+0000 mon.smithi029 (mon.0) 208 : audit [DBG] from='client.? 172.21.15.134:0/1108041428' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:46:22.962 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:46:22.963 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:46:23.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:46:23 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:46:21.843392+0000 mgr.smithi029.blhxxk (mgr.14164) 131 : cluster [DBG] pgmap v124: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:46:24.062 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:24.060+0000 7f4e5bfff700 1 -- 172.21.15.134:0/362865547 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e5c0f51a0 msgr2=0x7f4e5c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:46:24.062 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:24.060+0000 7f4e5bfff700 1 --2- 172.21.15.134:0/362865547 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e5c0f51a0 0x7f4e5c0f55c0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7f4e4c009fb0 tx=0x7f4e4c009b30).stop 2024-01-21T11:46:24.062 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:24.060+0000 7f4e5bfff700 1 -- 172.21.15.134:0/362865547 shutdown_connections 2024-01-21T11:46:24.062 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:24.060+0000 7f4e5bfff700 1 --2- 172.21.15.134:0/362865547 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e5c0f51a0 0x7f4e5c0f55c0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:46:24.062 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:24.060+0000 7f4e5bfff700 1 -- 172.21.15.134:0/362865547 >> 172.21.15.134:0/362865547 conn(0x7f4e5c0f09a0 msgr2=0x7f4e5c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:46:24.062 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:24.061+0000 7f4e5bfff700 1 -- 172.21.15.134:0/362865547 shutdown_connections 2024-01-21T11:46:24.062 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:24.061+0000 7f4e5bfff700 1 -- 172.21.15.134:0/362865547 wait complete. 2024-01-21T11:46:24.063 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:24.061+0000 7f4e5bfff700 1 Processor -- start 2024-01-21T11:46:24.063 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:24.061+0000 7f4e5bfff700 1 -- start start 2024-01-21T11:46:24.063 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:24.061+0000 7f4e5bfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e5c0f51a0 0x7f4e5c1064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:46:24.063 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:24.061+0000 7f4e5bfff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e5c1069d0 con 0x7f4e5c0f51a0 2024-01-21T11:46:24.063 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:24.062+0000 7f4e5affd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e5c0f51a0 0x7f4e5c1064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:46:24.063 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:24.062+0000 7f4e5affd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e5c0f51a0 0x7f4e5c1064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:40258/0 (socket says 172.21.15.134:40258) 2024-01-21T11:46:24.063 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:24.062+0000 7f4e5affd700 1 -- 172.21.15.134:0/4135255539 learned_addr learned my addr 172.21.15.134:0/4135255539 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:46:24.063 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:24.062+0000 7f4e5affd700 1 -- 172.21.15.134:0/4135255539 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e4c005040 con 0x7f4e5c0f51a0 2024-01-21T11:46:24.063 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:24.062+0000 7f4e5affd700 1 --2- 172.21.15.134:0/4135255539 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e5c0f51a0 0x7f4e5c1064e0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f4e4c009830 tx=0x7f4e4c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:46:24.063 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:24.062+0000 7f4e437fe700 1 -- 172.21.15.134:0/4135255539 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4e4c009320 con 0x7f4e5c0f51a0 2024-01-21T11:46:24.064 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:24.062+0000 7f4e5bfff700 1 -- 172.21.15.134:0/4135255539 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e5c106bd0 con 0x7f4e5c0f51a0 2024-01-21T11:46:24.064 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:24.062+0000 7f4e437fe700 1 -- 172.21.15.134:0/4135255539 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4e4c0136c0 con 0x7f4e5c0f51a0 2024-01-21T11:46:24.064 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:24.062+0000 7f4e437fe700 1 -- 172.21.15.134:0/4135255539 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4e4c01ee90 con 0x7f4e5c0f51a0 2024-01-21T11:46:24.064 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:24.062+0000 7f4e5bfff700 1 -- 172.21.15.134:0/4135255539 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e5c107060 con 0x7f4e5c0f51a0 2024-01-21T11:46:24.064 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:24.063+0000 7f4e437fe700 1 -- 172.21.15.134:0/4135255539 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4e4c01e420 con 0x7f4e5c0f51a0 2024-01-21T11:46:24.065 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:24.063+0000 7f4e5bfff700 1 -- 172.21.15.134:0/4135255539 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e5c045ad0 con 0x7f4e5c0f51a0 2024-01-21T11:46:24.065 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:24.063+0000 7f4e437fe700 1 --2- 172.21.15.134:0/4135255539 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f4e4402ff10 0x7f4e440323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:46:24.065 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:24.063+0000 7f4e437fe700 1 -- 172.21.15.134:0/4135255539 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4e4c040f40 con 0x7f4e5c0f51a0 2024-01-21T11:46:24.065 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:24.064+0000 7f4e5a7fc700 1 --2- 172.21.15.134:0/4135255539 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f4e4402ff10 0x7f4e440323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:46:24.065 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:24.064+0000 7f4e5a7fc700 1 --2- 172.21.15.134:0/4135255539 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f4e4402ff10 0x7f4e440323d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f4e500096a0 tx=0x7f4e50006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:46:24.068 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:24.068+0000 7f4e437fe700 1 -- 172.21.15.134:0/4135255539 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4e4c06d2b0 con 0x7f4e5c0f51a0 2024-01-21T11:46:24.276 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:24.276+0000 7f4e5bfff700 1 -- 172.21.15.134:0/4135255539 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4e5c107eb0 con 0x7f4e5c0f51a0 2024-01-21T11:46:24.276 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:24.276+0000 7f4e437fe700 1 -- 172.21.15.134:0/4135255539 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4e4c0095f0 con 0x7f4e5c0f51a0 2024-01-21T11:46:24.277 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:46:24.277 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:46:24.278 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:24.278+0000 7f4e5bfff700 1 -- 172.21.15.134:0/4135255539 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f4e4402ff10 msgr2=0x7f4e440323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:46:24.278 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:24.278+0000 7f4e5bfff700 1 --2- 172.21.15.134:0/4135255539 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f4e4402ff10 0x7f4e440323d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f4e500096a0 tx=0x7f4e50006b40).stop 2024-01-21T11:46:24.278 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:24.278+0000 7f4e5bfff700 1 -- 172.21.15.134:0/4135255539 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e5c0f51a0 msgr2=0x7f4e5c1064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:46:24.278 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:24.278+0000 7f4e5bfff700 1 --2- 172.21.15.134:0/4135255539 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e5c0f51a0 0x7f4e5c1064e0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f4e4c009830 tx=0x7f4e4c00b040).stop 2024-01-21T11:46:24.279 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:24.278+0000 7f4e5bfff700 1 -- 172.21.15.134:0/4135255539 shutdown_connections 2024-01-21T11:46:24.279 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:24.278+0000 7f4e5bfff700 1 --2- 172.21.15.134:0/4135255539 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f4e4402ff10 0x7f4e440323d0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:46:24.279 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:24.278+0000 7f4e5bfff700 1 --2- 172.21.15.134:0/4135255539 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e5c0f51a0 0x7f4e5c1064e0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:46:24.279 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:24.278+0000 7f4e5bfff700 1 -- 172.21.15.134:0/4135255539 >> 172.21.15.134:0/4135255539 conn(0x7f4e5c0f09a0 msgr2=0x7f4e5c05fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:46:24.279 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:24.278+0000 7f4e5bfff700 1 -- 172.21.15.134:0/4135255539 shutdown_connections 2024-01-21T11:46:24.279 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:24.278+0000 7f4e5bfff700 1 -- 172.21.15.134:0/4135255539 wait complete. 2024-01-21T11:46:24.279 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:46:24.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:46:24 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:46:24.277602+0000 mon.smithi029 (mon.0) 209 : audit [DBG] from='client.? 172.21.15.134:0/4135255539' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:46:25.659 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:46:25.660 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:46:25.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:46:25 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:46:23.843696+0000 mgr.smithi029.blhxxk (mgr.14164) 132 : cluster [DBG] pgmap v125: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:46:26.805 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:26.803+0000 7fc15bfe7700 1 -- 172.21.15.134:0/3465644858 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1540f5410 msgr2=0x7fc1540f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:46:26.805 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:26.803+0000 7fc15bfe7700 1 --2- 172.21.15.134:0/3465644858 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1540f5410 0x7fc1540f5830 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7fc144009fb0 tx=0x7fc144009b30).stop 2024-01-21T11:46:26.805 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:26.803+0000 7fc15bfe7700 1 -- 172.21.15.134:0/3465644858 shutdown_connections 2024-01-21T11:46:26.805 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:26.803+0000 7fc15bfe7700 1 --2- 172.21.15.134:0/3465644858 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1540f5410 0x7fc1540f5830 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:46:26.806 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:26.803+0000 7fc15bfe7700 1 -- 172.21.15.134:0/3465644858 >> 172.21.15.134:0/3465644858 conn(0x7fc1540f0b70 msgr2=0x7fc1540f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:46:26.806 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:26.803+0000 7fc15bfe7700 1 -- 172.21.15.134:0/3465644858 shutdown_connections 2024-01-21T11:46:26.806 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:26.803+0000 7fc15bfe7700 1 -- 172.21.15.134:0/3465644858 wait complete. 2024-01-21T11:46:26.806 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:26.804+0000 7fc15bfe7700 1 Processor -- start 2024-01-21T11:46:26.806 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:26.804+0000 7fc15bfe7700 1 -- start start 2024-01-21T11:46:26.806 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:26.804+0000 7fc15bfe7700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1540f5410 0x7fc154106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:46:26.806 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:26.804+0000 7fc15bfe7700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc154106c20 con 0x7fc1540f5410 2024-01-21T11:46:26.806 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:26.804+0000 7fc159d83700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1540f5410 0x7fc154106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:46:26.806 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:26.804+0000 7fc159d83700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1540f5410 0x7fc154106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:41262/0 (socket says 172.21.15.134:41262) 2024-01-21T11:46:26.806 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:26.804+0000 7fc159d83700 1 -- 172.21.15.134:0/2070551483 learned_addr learned my addr 172.21.15.134:0/2070551483 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:46:26.806 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:26.804+0000 7fc159d83700 1 -- 172.21.15.134:0/2070551483 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc144005040 con 0x7fc1540f5410 2024-01-21T11:46:26.806 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:26.804+0000 7fc159d83700 1 --2- 172.21.15.134:0/2070551483 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1540f5410 0x7fc154106730 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7fc144013510 tx=0x7fc14400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:46:26.806 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:26.805+0000 7fc14a7fc700 1 -- 172.21.15.134:0/2070551483 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc144009320 con 0x7fc1540f5410 2024-01-21T11:46:26.806 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:26.805+0000 7fc15bfe7700 1 -- 172.21.15.134:0/2070551483 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc154106e20 con 0x7fc1540f5410 2024-01-21T11:46:26.807 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:26.805+0000 7fc14a7fc700 1 -- 172.21.15.134:0/2070551483 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc144013f00 con 0x7fc1540f5410 2024-01-21T11:46:26.807 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:26.805+0000 7fc14a7fc700 1 -- 172.21.15.134:0/2070551483 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc144014d90 con 0x7fc1540f5410 2024-01-21T11:46:26.807 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:26.805+0000 7fc15bfe7700 1 -- 172.21.15.134:0/2070551483 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1541072b0 con 0x7fc1540f5410 2024-01-21T11:46:26.807 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:26.805+0000 7fc15bfe7700 1 -- 172.21.15.134:0/2070551483 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc154045ad0 con 0x7fc1540f5410 2024-01-21T11:46:26.808 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:26.806+0000 7fc14a7fc700 1 -- 172.21.15.134:0/2070551483 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc144014420 con 0x7fc1540f5410 2024-01-21T11:46:26.808 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:26.806+0000 7fc14a7fc700 1 --2- 172.21.15.134:0/2070551483 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fc14002ff60 0x7fc140032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:46:26.808 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:26.806+0000 7fc14a7fc700 1 -- 172.21.15.134:0/2070551483 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc144041050 con 0x7fc1540f5410 2024-01-21T11:46:26.808 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:26.806+0000 7fc159582700 1 --2- 172.21.15.134:0/2070551483 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fc14002ff60 0x7fc140032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:46:26.808 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:26.806+0000 7fc159582700 1 --2- 172.21.15.134:0/2070551483 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fc14002ff60 0x7fc140032420 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7fc1500096a0 tx=0x7fc150006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:46:26.810 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:26.810+0000 7fc14a7fc700 1 -- 172.21.15.134:0/2070551483 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc1440095f0 con 0x7fc1540f5410 2024-01-21T11:46:27.020 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:27.019+0000 7fc15bfe7700 1 -- 172.21.15.134:0/2070551483 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc1540fae60 con 0x7fc1540f5410 2024-01-21T11:46:27.020 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:27.020+0000 7fc14a7fc700 1 -- 172.21.15.134:0/2070551483 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc14401db70 con 0x7fc1540f5410 2024-01-21T11:46:27.020 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:46:27.021 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:46:27.023 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:27.022+0000 7fc15bfe7700 1 -- 172.21.15.134:0/2070551483 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fc14002ff60 msgr2=0x7fc140032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:46:27.023 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:27.022+0000 7fc15bfe7700 1 --2- 172.21.15.134:0/2070551483 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fc14002ff60 0x7fc140032420 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7fc1500096a0 tx=0x7fc150006b40).stop 2024-01-21T11:46:27.023 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:27.022+0000 7fc15bfe7700 1 -- 172.21.15.134:0/2070551483 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1540f5410 msgr2=0x7fc154106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:46:27.023 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:27.022+0000 7fc15bfe7700 1 --2- 172.21.15.134:0/2070551483 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1540f5410 0x7fc154106730 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7fc144013510 tx=0x7fc14400b040).stop 2024-01-21T11:46:27.023 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:27.022+0000 7fc15bfe7700 1 -- 172.21.15.134:0/2070551483 shutdown_connections 2024-01-21T11:46:27.023 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:27.022+0000 7fc15bfe7700 1 --2- 172.21.15.134:0/2070551483 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fc14002ff60 0x7fc140032420 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:46:27.023 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:27.022+0000 7fc15bfe7700 1 --2- 172.21.15.134:0/2070551483 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1540f5410 0x7fc154106730 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:46:27.023 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:27.022+0000 7fc15bfe7700 1 -- 172.21.15.134:0/2070551483 >> 172.21.15.134:0/2070551483 conn(0x7fc1540f0b70 msgr2=0x7fc1540f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:46:27.023 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:27.022+0000 7fc15bfe7700 1 -- 172.21.15.134:0/2070551483 shutdown_connections 2024-01-21T11:46:27.023 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:27.022+0000 7fc15bfe7700 1 -- 172.21.15.134:0/2070551483 wait complete. 2024-01-21T11:46:27.023 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:46:27.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:46:27 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:46:25.843953+0000 mgr.smithi029.blhxxk (mgr.14164) 133 : cluster [DBG] pgmap v126: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:46:27.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:46:27 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:46:27.021386+0000 mon.smithi029 (mon.0) 210 : audit [DBG] from='client.? 172.21.15.134:0/2070551483' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:46:28.396 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:46:28.396 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:46:29.559 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:29.557+0000 7f9f9a5e2700 1 -- 172.21.15.134:0/3669993382 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f940f6510 msgr2=0x7f9f940f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:46:29.560 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:29.557+0000 7f9f9a5e2700 1 --2- 172.21.15.134:0/3669993382 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f940f6510 0x7f9f940f6930 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7f9f84009fb0 tx=0x7f9f84009b30).stop 2024-01-21T11:46:29.560 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:29.557+0000 7f9f9a5e2700 1 -- 172.21.15.134:0/3669993382 shutdown_connections 2024-01-21T11:46:29.560 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:29.557+0000 7f9f9a5e2700 1 --2- 172.21.15.134:0/3669993382 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f940f6510 0x7f9f940f6930 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:46:29.560 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:29.557+0000 7f9f9a5e2700 1 -- 172.21.15.134:0/3669993382 >> 172.21.15.134:0/3669993382 conn(0x7f9f940f1c70 msgr2=0x7f9f940f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:46:29.560 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:29.557+0000 7f9f9a5e2700 1 -- 172.21.15.134:0/3669993382 shutdown_connections 2024-01-21T11:46:29.560 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:29.557+0000 7f9f9a5e2700 1 -- 172.21.15.134:0/3669993382 wait complete. 2024-01-21T11:46:29.560 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:29.558+0000 7f9f9a5e2700 1 Processor -- start 2024-01-21T11:46:29.560 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:29.558+0000 7f9f9a5e2700 1 -- start start 2024-01-21T11:46:29.560 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:29.558+0000 7f9f9a5e2700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f940f6510 0x7f9f941066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:46:29.561 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:29.558+0000 7f9f9a5e2700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f94106bc0 con 0x7f9f940f6510 2024-01-21T11:46:29.561 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:29.558+0000 7f9f93fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f940f6510 0x7f9f941066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:46:29.561 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:29.558+0000 7f9f93fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f940f6510 0x7f9f941066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:41288/0 (socket says 172.21.15.134:41288) 2024-01-21T11:46:29.561 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:29.558+0000 7f9f93fff700 1 -- 172.21.15.134:0/1225166055 learned_addr learned my addr 172.21.15.134:0/1225166055 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:46:29.561 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:29.559+0000 7f9f93fff700 1 -- 172.21.15.134:0/1225166055 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f84005040 con 0x7f9f940f6510 2024-01-21T11:46:29.561 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:29.559+0000 7f9f93fff700 1 --2- 172.21.15.134:0/1225166055 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f940f6510 0x7f9f941066d0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f9f84013510 tx=0x7f9f8400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:46:29.561 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:29.559+0000 7f9f90ff9700 1 -- 172.21.15.134:0/1225166055 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9f84009320 con 0x7f9f940f6510 2024-01-21T11:46:29.561 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:29.559+0000 7f9f90ff9700 1 -- 172.21.15.134:0/1225166055 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9f84013f00 con 0x7f9f940f6510 2024-01-21T11:46:29.561 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:29.559+0000 7f9f9a5e2700 1 -- 172.21.15.134:0/1225166055 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f94106dc0 con 0x7f9f940f6510 2024-01-21T11:46:29.561 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:29.559+0000 7f9f90ff9700 1 -- 172.21.15.134:0/1225166055 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9f84014d90 con 0x7f9f940f6510 2024-01-21T11:46:29.562 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:29.559+0000 7f9f9a5e2700 1 -- 172.21.15.134:0/1225166055 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f94107250 con 0x7f9f940f6510 2024-01-21T11:46:29.562 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:29.560+0000 7f9f90ff9700 1 -- 172.21.15.134:0/1225166055 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9f84014420 con 0x7f9f940f6510 2024-01-21T11:46:29.563 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:29.560+0000 7f9f9a5e2700 1 -- 172.21.15.134:0/1225166055 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f94045ad0 con 0x7f9f940f6510 2024-01-21T11:46:29.563 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:29.560+0000 7f9f90ff9700 1 --2- 172.21.15.134:0/1225166055 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f9f7c02ff60 0x7f9f7c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:46:29.563 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:29.560+0000 7f9f90ff9700 1 -- 172.21.15.134:0/1225166055 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9f84041080 con 0x7f9f940f6510 2024-01-21T11:46:29.563 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:29.560+0000 7f9f937fe700 1 --2- 172.21.15.134:0/1225166055 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f9f7c02ff60 0x7f9f7c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:46:29.563 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:29.561+0000 7f9f937fe700 1 --2- 172.21.15.134:0/1225166055 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f9f7c02ff60 0x7f9f7c032420 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f9f880096a0 tx=0x7f9f88006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:46:29.567 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:29.565+0000 7f9f90ff9700 1 -- 172.21.15.134:0/1225166055 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9f840095f0 con 0x7f9f940f6510 2024-01-21T11:46:29.771 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:29.771+0000 7f9f9a5e2700 1 -- 172.21.15.134:0/1225166055 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9f94108100 con 0x7f9f940f6510 2024-01-21T11:46:29.772 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:29.772+0000 7f9f90ff9700 1 -- 172.21.15.134:0/1225166055 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9f8401db70 con 0x7f9f940f6510 2024-01-21T11:46:29.772 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:46:29.773 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:46:29.774 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:29.773+0000 7f9f9a5e2700 1 -- 172.21.15.134:0/1225166055 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f9f7c02ff60 msgr2=0x7f9f7c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:46:29.774 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:29.773+0000 7f9f9a5e2700 1 --2- 172.21.15.134:0/1225166055 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f9f7c02ff60 0x7f9f7c032420 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f9f880096a0 tx=0x7f9f88006b40).stop 2024-01-21T11:46:29.774 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:29.773+0000 7f9f9a5e2700 1 -- 172.21.15.134:0/1225166055 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f940f6510 msgr2=0x7f9f941066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:46:29.774 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:29.773+0000 7f9f9a5e2700 1 --2- 172.21.15.134:0/1225166055 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f940f6510 0x7f9f941066d0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f9f84013510 tx=0x7f9f8400b040).stop 2024-01-21T11:46:29.774 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:29.774+0000 7f9f9a5e2700 1 -- 172.21.15.134:0/1225166055 shutdown_connections 2024-01-21T11:46:29.774 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:29.774+0000 7f9f9a5e2700 1 --2- 172.21.15.134:0/1225166055 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f9f7c02ff60 0x7f9f7c032420 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:46:29.774 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:29.774+0000 7f9f9a5e2700 1 --2- 172.21.15.134:0/1225166055 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f940f6510 0x7f9f941066d0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:46:29.774 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:29.774+0000 7f9f9a5e2700 1 -- 172.21.15.134:0/1225166055 >> 172.21.15.134:0/1225166055 conn(0x7f9f940f1c70 msgr2=0x7f9f940f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:46:29.774 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:29.774+0000 7f9f9a5e2700 1 -- 172.21.15.134:0/1225166055 shutdown_connections 2024-01-21T11:46:29.775 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:29.774+0000 7f9f9a5e2700 1 -- 172.21.15.134:0/1225166055 wait complete. 2024-01-21T11:46:29.775 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:46:29.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:46:29 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:46:27.844215+0000 mgr.smithi029.blhxxk (mgr.14164) 134 : cluster [DBG] pgmap v127: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:46:30.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:46:30 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:46:29.773193+0000 mon.smithi029 (mon.0) 211 : audit [DBG] from='client.? 172.21.15.134:0/1225166055' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:46:31.148 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:46:31.148 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:46:31.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:46:31 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:46:29.844485+0000 mgr.smithi029.blhxxk (mgr.14164) 135 : cluster [DBG] pgmap v128: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:46:32.257 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:32.255+0000 7fc950383700 1 -- 172.21.15.134:0/2460758138 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9480f5410 msgr2=0x7fc9480f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:46:32.257 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:32.255+0000 7fc950383700 1 --2- 172.21.15.134:0/2460758138 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9480f5410 0x7fc9480f5830 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7fc938009fb0 tx=0x7fc938009b30).stop 2024-01-21T11:46:32.257 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:32.256+0000 7fc950383700 1 -- 172.21.15.134:0/2460758138 shutdown_connections 2024-01-21T11:46:32.257 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:32.256+0000 7fc950383700 1 --2- 172.21.15.134:0/2460758138 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9480f5410 0x7fc9480f5830 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:46:32.257 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:32.256+0000 7fc950383700 1 -- 172.21.15.134:0/2460758138 >> 172.21.15.134:0/2460758138 conn(0x7fc9480f0b70 msgr2=0x7fc9480f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:46:32.257 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:32.256+0000 7fc950383700 1 -- 172.21.15.134:0/2460758138 shutdown_connections 2024-01-21T11:46:32.257 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:32.256+0000 7fc950383700 1 -- 172.21.15.134:0/2460758138 wait complete. 2024-01-21T11:46:32.257 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:32.256+0000 7fc950383700 1 Processor -- start 2024-01-21T11:46:32.257 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:32.256+0000 7fc950383700 1 -- start start 2024-01-21T11:46:32.257 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:32.257+0000 7fc950383700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9480f5410 0x7fc948105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:46:32.257 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:32.257+0000 7fc950383700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc948105b20 con 0x7fc9480f5410 2024-01-21T11:46:32.259 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:32.257+0000 7fc94e11f700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9480f5410 0x7fc948105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:46:32.259 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:32.257+0000 7fc94e11f700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9480f5410 0x7fc948105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:41306/0 (socket says 172.21.15.134:41306) 2024-01-21T11:46:32.259 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:32.257+0000 7fc94e11f700 1 -- 172.21.15.134:0/3093931384 learned_addr learned my addr 172.21.15.134:0/3093931384 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:46:32.260 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:32.257+0000 7fc94e11f700 1 -- 172.21.15.134:0/3093931384 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc938005040 con 0x7fc9480f5410 2024-01-21T11:46:32.260 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:32.257+0000 7fc94e11f700 1 --2- 172.21.15.134:0/3093931384 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9480f5410 0x7fc948105630 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7fc938013510 tx=0x7fc93800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:46:32.260 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:32.257+0000 7fc93effd700 1 -- 172.21.15.134:0/3093931384 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc938009320 con 0x7fc9480f5410 2024-01-21T11:46:32.260 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:32.258+0000 7fc950383700 1 -- 172.21.15.134:0/3093931384 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc948105d20 con 0x7fc9480f5410 2024-01-21T11:46:32.260 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:32.258+0000 7fc93effd700 1 -- 172.21.15.134:0/3093931384 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc938013f00 con 0x7fc9480f5410 2024-01-21T11:46:32.260 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:32.258+0000 7fc93effd700 1 -- 172.21.15.134:0/3093931384 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc938014d90 con 0x7fc9480f5410 2024-01-21T11:46:32.260 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:32.258+0000 7fc950383700 1 -- 172.21.15.134:0/3093931384 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9481061b0 con 0x7fc9480f5410 2024-01-21T11:46:32.260 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:32.258+0000 7fc93effd700 1 -- 172.21.15.134:0/3093931384 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc9380095f0 con 0x7fc9480f5410 2024-01-21T11:46:32.260 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:32.258+0000 7fc950383700 1 -- 172.21.15.134:0/3093931384 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc948045ad0 con 0x7fc9480f5410 2024-01-21T11:46:32.260 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:32.258+0000 7fc93effd700 1 --2- 172.21.15.134:0/3093931384 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fc93402ff60 0x7fc934032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:46:32.260 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:32.259+0000 7fc93effd700 1 -- 172.21.15.134:0/3093931384 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc938042d90 con 0x7fc9480f5410 2024-01-21T11:46:32.260 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:32.259+0000 7fc94d91e700 1 --2- 172.21.15.134:0/3093931384 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fc93402ff60 0x7fc934032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:46:32.260 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:32.259+0000 7fc94d91e700 1 --2- 172.21.15.134:0/3093931384 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fc93402ff60 0x7fc934032420 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7fc9440096a0 tx=0x7fc944006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:46:32.263 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:32.263+0000 7fc93effd700 1 -- 172.21.15.134:0/3093931384 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc938014420 con 0x7fc9480f5410 2024-01-21T11:46:32.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:32.470+0000 7fc950383700 1 -- 172.21.15.134:0/3093931384 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc9480fae60 con 0x7fc9480f5410 2024-01-21T11:46:32.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:32.470+0000 7fc93effd700 1 -- 172.21.15.134:0/3093931384 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc938011290 con 0x7fc9480f5410 2024-01-21T11:46:32.471 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:46:32.471 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:46:32.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:32.472+0000 7fc950383700 1 -- 172.21.15.134:0/3093931384 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fc93402ff60 msgr2=0x7fc934032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:46:32.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:32.472+0000 7fc950383700 1 --2- 172.21.15.134:0/3093931384 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fc93402ff60 0x7fc934032420 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7fc9440096a0 tx=0x7fc944006b40).stop 2024-01-21T11:46:32.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:32.472+0000 7fc950383700 1 -- 172.21.15.134:0/3093931384 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9480f5410 msgr2=0x7fc948105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:46:32.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:32.472+0000 7fc950383700 1 --2- 172.21.15.134:0/3093931384 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9480f5410 0x7fc948105630 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7fc938013510 tx=0x7fc93800b040).stop 2024-01-21T11:46:32.473 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:32.472+0000 7fc950383700 1 -- 172.21.15.134:0/3093931384 shutdown_connections 2024-01-21T11:46:32.473 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:32.472+0000 7fc950383700 1 --2- 172.21.15.134:0/3093931384 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fc93402ff60 0x7fc934032420 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:46:32.473 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:32.472+0000 7fc950383700 1 --2- 172.21.15.134:0/3093931384 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9480f5410 0x7fc948105630 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:46:32.473 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:32.472+0000 7fc950383700 1 -- 172.21.15.134:0/3093931384 >> 172.21.15.134:0/3093931384 conn(0x7fc9480f0b70 msgr2=0x7fc9480f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:46:32.473 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:32.472+0000 7fc950383700 1 -- 172.21.15.134:0/3093931384 shutdown_connections 2024-01-21T11:46:32.473 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:32.472+0000 7fc950383700 1 -- 172.21.15.134:0/3093931384 wait complete. 2024-01-21T11:46:32.473 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:46:32.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:46:32 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:46:32.471668+0000 mon.smithi029 (mon.0) 212 : audit [DBG] from='client.? 172.21.15.134:0/3093931384' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:46:33.852 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:46:33.852 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:46:33.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:46:33 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:46:31.844826+0000 mgr.smithi029.blhxxk (mgr.14164) 136 : cluster [DBG] pgmap v129: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:46:35.006 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:35.003+0000 7f9725376700 1 -- 172.21.15.134:0/956485574 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97200f6510 msgr2=0x7f97200f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:46:35.006 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:35.003+0000 7f9725376700 1 --2- 172.21.15.134:0/956485574 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97200f6510 0x7f97200f6930 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7f9714009fb0 tx=0x7f9714009b30).stop 2024-01-21T11:46:35.006 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:35.003+0000 7f9725376700 1 -- 172.21.15.134:0/956485574 shutdown_connections 2024-01-21T11:46:35.007 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:35.003+0000 7f9725376700 1 --2- 172.21.15.134:0/956485574 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97200f6510 0x7f97200f6930 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:46:35.007 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:35.003+0000 7f9725376700 1 -- 172.21.15.134:0/956485574 >> 172.21.15.134:0/956485574 conn(0x7f97200f1c70 msgr2=0x7f97200f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:46:35.007 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:35.003+0000 7f9725376700 1 -- 172.21.15.134:0/956485574 shutdown_connections 2024-01-21T11:46:35.007 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:35.003+0000 7f9725376700 1 -- 172.21.15.134:0/956485574 wait complete. 2024-01-21T11:46:35.007 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:35.004+0000 7f9725376700 1 Processor -- start 2024-01-21T11:46:35.007 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:35.004+0000 7f9725376700 1 -- start start 2024-01-21T11:46:35.007 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:35.004+0000 7f9725376700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97200f6510 0x7f9720106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:46:35.007 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:35.004+0000 7f9725376700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9720106c20 con 0x7f97200f6510 2024-01-21T11:46:35.007 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:35.004+0000 7f971effd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97200f6510 0x7f9720106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:46:35.007 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:35.004+0000 7f971effd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97200f6510 0x7f9720106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:34092/0 (socket says 172.21.15.134:34092) 2024-01-21T11:46:35.007 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:35.004+0000 7f971effd700 1 -- 172.21.15.134:0/914466459 learned_addr learned my addr 172.21.15.134:0/914466459 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:46:35.007 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:35.004+0000 7f971effd700 1 -- 172.21.15.134:0/914466459 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9714005040 con 0x7f97200f6510 2024-01-21T11:46:35.007 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:35.004+0000 7f971effd700 1 --2- 172.21.15.134:0/914466459 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97200f6510 0x7f9720106730 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f9714013510 tx=0x7f971400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:46:35.007 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:35.005+0000 7f97077fe700 1 -- 172.21.15.134:0/914466459 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9714009320 con 0x7f97200f6510 2024-01-21T11:46:35.007 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:35.005+0000 7f9725376700 1 -- 172.21.15.134:0/914466459 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9720106e20 con 0x7f97200f6510 2024-01-21T11:46:35.008 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:35.005+0000 7f97077fe700 1 -- 172.21.15.134:0/914466459 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9714013f00 con 0x7f97200f6510 2024-01-21T11:46:35.008 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:35.005+0000 7f97077fe700 1 -- 172.21.15.134:0/914466459 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9714014d90 con 0x7f97200f6510 2024-01-21T11:46:35.008 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:35.005+0000 7f9725376700 1 -- 172.21.15.134:0/914466459 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97201072b0 con 0x7f97200f6510 2024-01-21T11:46:35.008 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:35.005+0000 7f97077fe700 1 -- 172.21.15.134:0/914466459 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f97140263a0 con 0x7f97200f6510 2024-01-21T11:46:35.008 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:35.006+0000 7f97077fe700 1 --2- 172.21.15.134:0/914466459 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f9708030000 0x7f97080324c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:46:35.008 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:35.006+0000 7f9725376700 1 -- 172.21.15.134:0/914466459 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9720045ad0 con 0x7f97200f6510 2024-01-21T11:46:35.008 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:35.006+0000 7f97077fe700 1 -- 172.21.15.134:0/914466459 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9714041060 con 0x7f97200f6510 2024-01-21T11:46:35.008 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:35.006+0000 7f971e7fc700 1 --2- 172.21.15.134:0/914466459 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f9708030000 0x7f97080324c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:46:35.008 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:35.006+0000 7f971e7fc700 1 --2- 172.21.15.134:0/914466459 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f9708030000 0x7f97080324c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f97100096a0 tx=0x7f9710006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:46:35.010 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:35.011+0000 7f97077fe700 1 -- 172.21.15.134:0/914466459 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9714012920 con 0x7f97200f6510 2024-01-21T11:46:35.220 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:35.220+0000 7f9725376700 1 -- 172.21.15.134:0/914466459 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f97200fc050 con 0x7f97200f6510 2024-01-21T11:46:35.220 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:35.221+0000 7f97077fe700 1 -- 172.21.15.134:0/914466459 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f971402b020 con 0x7f97200f6510 2024-01-21T11:46:35.221 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:46:35.221 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:46:35.222 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:35.222+0000 7f9725376700 1 -- 172.21.15.134:0/914466459 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f9708030000 msgr2=0x7f97080324c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:46:35.222 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:35.222+0000 7f9725376700 1 --2- 172.21.15.134:0/914466459 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f9708030000 0x7f97080324c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f97100096a0 tx=0x7f9710006b40).stop 2024-01-21T11:46:35.222 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:35.222+0000 7f9725376700 1 -- 172.21.15.134:0/914466459 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97200f6510 msgr2=0x7f9720106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:46:35.223 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:35.222+0000 7f9725376700 1 --2- 172.21.15.134:0/914466459 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97200f6510 0x7f9720106730 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f9714013510 tx=0x7f971400b040).stop 2024-01-21T11:46:35.223 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:35.222+0000 7f9725376700 1 -- 172.21.15.134:0/914466459 shutdown_connections 2024-01-21T11:46:35.223 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:35.222+0000 7f9725376700 1 --2- 172.21.15.134:0/914466459 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f9708030000 0x7f97080324c0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:46:35.223 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:35.222+0000 7f9725376700 1 --2- 172.21.15.134:0/914466459 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97200f6510 0x7f9720106730 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:46:35.223 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:35.222+0000 7f9725376700 1 -- 172.21.15.134:0/914466459 >> 172.21.15.134:0/914466459 conn(0x7f97200f1c70 msgr2=0x7f97200f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:46:35.223 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:35.222+0000 7f9725376700 1 -- 172.21.15.134:0/914466459 shutdown_connections 2024-01-21T11:46:35.223 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:35.222+0000 7f9725376700 1 -- 172.21.15.134:0/914466459 wait complete. 2024-01-21T11:46:35.223 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:46:35.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:46:35 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:46:33.845118+0000 mgr.smithi029.blhxxk (mgr.14164) 137 : cluster [DBG] pgmap v130: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:46:35.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:46:35 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:46:35.221767+0000 mon.smithi029 (mon.0) 213 : audit [DBG] from='client.? 172.21.15.134:0/914466459' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:46:36.599 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:46:36.599 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:46:37.753 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:37.752+0000 7efd33cf5700 1 -- 172.21.15.134:0/2808795368 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd2c0f5410 msgr2=0x7efd2c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:46:37.753 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:37.752+0000 7efd33cf5700 1 --2- 172.21.15.134:0/2808795368 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd2c0f5410 0x7efd2c0f5830 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7efd20009fb0 tx=0x7efd20009b30).stop 2024-01-21T11:46:37.753 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:37.752+0000 7efd33cf5700 1 -- 172.21.15.134:0/2808795368 shutdown_connections 2024-01-21T11:46:37.753 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:37.752+0000 7efd33cf5700 1 --2- 172.21.15.134:0/2808795368 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd2c0f5410 0x7efd2c0f5830 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:46:37.753 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:37.752+0000 7efd33cf5700 1 -- 172.21.15.134:0/2808795368 >> 172.21.15.134:0/2808795368 conn(0x7efd2c0f0b70 msgr2=0x7efd2c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:46:37.753 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:37.752+0000 7efd33cf5700 1 -- 172.21.15.134:0/2808795368 shutdown_connections 2024-01-21T11:46:37.753 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:37.752+0000 7efd33cf5700 1 -- 172.21.15.134:0/2808795368 wait complete. 2024-01-21T11:46:37.755 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:37.753+0000 7efd33cf5700 1 Processor -- start 2024-01-21T11:46:37.755 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:37.753+0000 7efd33cf5700 1 -- start start 2024-01-21T11:46:37.755 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:37.753+0000 7efd33cf5700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd2c0f5410 0x7efd2c109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:46:37.755 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:37.753+0000 7efd33cf5700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd2c060240 con 0x7efd2c0f5410 2024-01-21T11:46:37.755 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:37.753+0000 7efd31a91700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd2c0f5410 0x7efd2c109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:46:37.755 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:37.753+0000 7efd31a91700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd2c0f5410 0x7efd2c109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:34114/0 (socket says 172.21.15.134:34114) 2024-01-21T11:46:37.755 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:37.753+0000 7efd31a91700 1 -- 172.21.15.134:0/4236371800 learned_addr learned my addr 172.21.15.134:0/4236371800 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:46:37.755 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:37.753+0000 7efd31a91700 1 -- 172.21.15.134:0/4236371800 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd20005040 con 0x7efd2c0f5410 2024-01-21T11:46:37.756 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:37.754+0000 7efd31a91700 1 --2- 172.21.15.134:0/4236371800 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd2c0f5410 0x7efd2c109430 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7efd20013510 tx=0x7efd2000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:46:37.756 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:37.754+0000 7efd1e7fc700 1 -- 172.21.15.134:0/4236371800 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efd20009320 con 0x7efd2c0f5410 2024-01-21T11:46:37.756 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:37.754+0000 7efd33cf5700 1 -- 172.21.15.134:0/4236371800 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd2c060440 con 0x7efd2c0f5410 2024-01-21T11:46:37.756 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:37.754+0000 7efd1e7fc700 1 -- 172.21.15.134:0/4236371800 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efd20013f00 con 0x7efd2c0f5410 2024-01-21T11:46:37.756 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:37.754+0000 7efd1e7fc700 1 -- 172.21.15.134:0/4236371800 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efd20014d90 con 0x7efd2c0f5410 2024-01-21T11:46:37.756 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:37.754+0000 7efd33cf5700 1 -- 172.21.15.134:0/4236371800 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd2c109ae0 con 0x7efd2c0f5410 2024-01-21T11:46:37.756 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:37.755+0000 7efd1e7fc700 1 -- 172.21.15.134:0/4236371800 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efd200095f0 con 0x7efd2c0f5410 2024-01-21T11:46:37.756 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:37.755+0000 7efd33cf5700 1 -- 172.21.15.134:0/4236371800 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd2c045ad0 con 0x7efd2c0f5410 2024-01-21T11:46:37.756 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:37.755+0000 7efd1e7fc700 1 --2- 172.21.15.134:0/4236371800 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7efd1802ff60 0x7efd18032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:46:37.756 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:37.755+0000 7efd1e7fc700 1 -- 172.21.15.134:0/4236371800 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efd20041080 con 0x7efd2c0f5410 2024-01-21T11:46:37.756 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:37.755+0000 7efd31290700 1 --2- 172.21.15.134:0/4236371800 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7efd1802ff60 0x7efd18032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:46:37.757 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:37.756+0000 7efd31290700 1 --2- 172.21.15.134:0/4236371800 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7efd1802ff60 0x7efd18032420 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7efd280096a0 tx=0x7efd28006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:46:37.760 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:37.760+0000 7efd1e7fc700 1 -- 172.21.15.134:0/4236371800 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efd200268c0 con 0x7efd2c0f5410 2024-01-21T11:46:37.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:46:37 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:46:35.845409+0000 mgr.smithi029.blhxxk (mgr.14164) 138 : cluster [DBG] pgmap v131: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:46:37.967 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:37.966+0000 7efd33cf5700 1 -- 172.21.15.134:0/4236371800 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efd2c0fae60 con 0x7efd2c0f5410 2024-01-21T11:46:37.967 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:37.967+0000 7efd1e7fc700 1 -- 172.21.15.134:0/4236371800 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7efd20010230 con 0x7efd2c0f5410 2024-01-21T11:46:37.967 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:46:37.968 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:46:37.969 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:37.968+0000 7efd33cf5700 1 -- 172.21.15.134:0/4236371800 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7efd1802ff60 msgr2=0x7efd18032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:46:37.969 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:37.968+0000 7efd33cf5700 1 --2- 172.21.15.134:0/4236371800 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7efd1802ff60 0x7efd18032420 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7efd280096a0 tx=0x7efd28006b40).stop 2024-01-21T11:46:37.969 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:37.968+0000 7efd33cf5700 1 -- 172.21.15.134:0/4236371800 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd2c0f5410 msgr2=0x7efd2c109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:46:37.969 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:37.968+0000 7efd33cf5700 1 --2- 172.21.15.134:0/4236371800 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd2c0f5410 0x7efd2c109430 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7efd20013510 tx=0x7efd2000b040).stop 2024-01-21T11:46:37.969 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:37.969+0000 7efd33cf5700 1 -- 172.21.15.134:0/4236371800 shutdown_connections 2024-01-21T11:46:37.969 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:37.969+0000 7efd33cf5700 1 --2- 172.21.15.134:0/4236371800 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7efd1802ff60 0x7efd18032420 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:46:37.970 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:37.969+0000 7efd33cf5700 1 --2- 172.21.15.134:0/4236371800 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd2c0f5410 0x7efd2c109430 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:46:37.970 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:37.969+0000 7efd33cf5700 1 -- 172.21.15.134:0/4236371800 >> 172.21.15.134:0/4236371800 conn(0x7efd2c0f0b70 msgr2=0x7efd2c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:46:37.970 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:37.969+0000 7efd33cf5700 1 -- 172.21.15.134:0/4236371800 shutdown_connections 2024-01-21T11:46:37.970 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:37.969+0000 7efd33cf5700 1 -- 172.21.15.134:0/4236371800 wait complete. 2024-01-21T11:46:37.970 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:46:38.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:46:38 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:46:37.968171+0000 mon.smithi029 (mon.0) 214 : audit [DBG] from='client.? 172.21.15.134:0/4236371800' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:46:39.343 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:46:39.343 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:46:39.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:46:39 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:46:37.845741+0000 mgr.smithi029.blhxxk (mgr.14164) 139 : cluster [DBG] pgmap v132: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:46:40.490 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:40.488+0000 7fd64750d700 1 -- 172.21.15.134:0/3732631373 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6400f5410 msgr2=0x7fd6400f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:46:40.490 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:40.488+0000 7fd64750d700 1 --2- 172.21.15.134:0/3732631373 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6400f5410 0x7fd6400f5830 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7fd630009fb0 tx=0x7fd630009b30).stop 2024-01-21T11:46:40.490 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:40.488+0000 7fd64750d700 1 -- 172.21.15.134:0/3732631373 shutdown_connections 2024-01-21T11:46:40.490 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:40.488+0000 7fd64750d700 1 --2- 172.21.15.134:0/3732631373 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6400f5410 0x7fd6400f5830 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:46:40.490 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:40.488+0000 7fd64750d700 1 -- 172.21.15.134:0/3732631373 >> 172.21.15.134:0/3732631373 conn(0x7fd6400f0b70 msgr2=0x7fd6400f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:46:40.490 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:40.488+0000 7fd64750d700 1 -- 172.21.15.134:0/3732631373 shutdown_connections 2024-01-21T11:46:40.490 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:40.488+0000 7fd64750d700 1 -- 172.21.15.134:0/3732631373 wait complete. 2024-01-21T11:46:40.490 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:40.489+0000 7fd64750d700 1 Processor -- start 2024-01-21T11:46:40.490 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:40.489+0000 7fd64750d700 1 -- start start 2024-01-21T11:46:40.491 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:40.489+0000 7fd64750d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6400f5410 0x7fd640109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:46:40.491 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:40.489+0000 7fd64750d700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd640060220 con 0x7fd6400f5410 2024-01-21T11:46:40.491 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:40.489+0000 7fd6452a9700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6400f5410 0x7fd640109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:46:40.491 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:40.489+0000 7fd6452a9700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6400f5410 0x7fd640109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:34124/0 (socket says 172.21.15.134:34124) 2024-01-21T11:46:40.491 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:40.489+0000 7fd6452a9700 1 -- 172.21.15.134:0/2022321846 learned_addr learned my addr 172.21.15.134:0/2022321846 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:46:40.491 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:40.489+0000 7fd6452a9700 1 -- 172.21.15.134:0/2022321846 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd630005040 con 0x7fd6400f5410 2024-01-21T11:46:40.491 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:40.489+0000 7fd6452a9700 1 --2- 172.21.15.134:0/2022321846 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6400f5410 0x7fd640109450 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7fd630014000 tx=0x7fd63000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:46:40.491 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:40.490+0000 7fd635ffb700 1 -- 172.21.15.134:0/2022321846 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd6300095f0 con 0x7fd6400f5410 2024-01-21T11:46:40.491 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:40.490+0000 7fd64750d700 1 -- 172.21.15.134:0/2022321846 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd640060420 con 0x7fd6400f5410 2024-01-21T11:46:40.491 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:40.490+0000 7fd635ffb700 1 -- 172.21.15.134:0/2022321846 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd630007e70 con 0x7fd6400f5410 2024-01-21T11:46:40.491 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:40.490+0000 7fd635ffb700 1 -- 172.21.15.134:0/2022321846 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd630014ea0 con 0x7fd6400f5410 2024-01-21T11:46:40.491 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:40.490+0000 7fd64750d700 1 -- 172.21.15.134:0/2022321846 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd640109b00 con 0x7fd6400f5410 2024-01-21T11:46:40.492 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:40.491+0000 7fd64750d700 1 -- 172.21.15.134:0/2022321846 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd640059100 con 0x7fd6400f5410 2024-01-21T11:46:40.492 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:40.491+0000 7fd635ffb700 1 -- 172.21.15.134:0/2022321846 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd630025070 con 0x7fd6400f5410 2024-01-21T11:46:40.492 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:40.491+0000 7fd635ffb700 1 --2- 172.21.15.134:0/2022321846 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fd62c02ff60 0x7fd62c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:46:40.492 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:40.491+0000 7fd635ffb700 1 -- 172.21.15.134:0/2022321846 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd630040a30 con 0x7fd6400f5410 2024-01-21T11:46:40.492 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:40.491+0000 7fd644aa8700 1 --2- 172.21.15.134:0/2022321846 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fd62c02ff60 0x7fd62c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:46:40.492 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:40.492+0000 7fd644aa8700 1 --2- 172.21.15.134:0/2022321846 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fd62c02ff60 0x7fd62c032420 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7fd63c0096a0 tx=0x7fd63c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:46:40.496 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:40.496+0000 7fd635ffb700 1 -- 172.21.15.134:0/2022321846 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd6300079c0 con 0x7fd6400f5410 2024-01-21T11:46:40.700 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:40.700+0000 7fd64750d700 1 -- 172.21.15.134:0/2022321846 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd640045ad0 con 0x7fd6400f5410 2024-01-21T11:46:40.701 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:40.701+0000 7fd635ffb700 1 -- 172.21.15.134:0/2022321846 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd63000f300 con 0x7fd6400f5410 2024-01-21T11:46:40.701 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:46:40.701 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:46:40.703 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:40.702+0000 7fd64750d700 1 -- 172.21.15.134:0/2022321846 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fd62c02ff60 msgr2=0x7fd62c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:46:40.703 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:40.702+0000 7fd64750d700 1 --2- 172.21.15.134:0/2022321846 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fd62c02ff60 0x7fd62c032420 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7fd63c0096a0 tx=0x7fd63c006b40).stop 2024-01-21T11:46:40.703 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:40.702+0000 7fd64750d700 1 -- 172.21.15.134:0/2022321846 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6400f5410 msgr2=0x7fd640109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:46:40.703 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:40.702+0000 7fd64750d700 1 --2- 172.21.15.134:0/2022321846 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6400f5410 0x7fd640109450 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7fd630014000 tx=0x7fd63000b040).stop 2024-01-21T11:46:40.703 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:40.702+0000 7fd64750d700 1 -- 172.21.15.134:0/2022321846 shutdown_connections 2024-01-21T11:46:40.703 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:40.702+0000 7fd64750d700 1 --2- 172.21.15.134:0/2022321846 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fd62c02ff60 0x7fd62c032420 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:46:40.703 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:40.702+0000 7fd64750d700 1 --2- 172.21.15.134:0/2022321846 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6400f5410 0x7fd640109450 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:46:40.703 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:40.703+0000 7fd64750d700 1 -- 172.21.15.134:0/2022321846 >> 172.21.15.134:0/2022321846 conn(0x7fd6400f0b70 msgr2=0x7fd6400f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:46:40.703 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:40.703+0000 7fd64750d700 1 -- 172.21.15.134:0/2022321846 shutdown_connections 2024-01-21T11:46:40.703 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:40.703+0000 7fd64750d700 1 -- 172.21.15.134:0/2022321846 wait complete. 2024-01-21T11:46:40.704 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:46:41.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:46:41 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:46:39.846012+0000 mgr.smithi029.blhxxk (mgr.14164) 140 : cluster [DBG] pgmap v133: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:46:41.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:46:41 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:46:40.701846+0000 mon.smithi029 (mon.0) 215 : audit [DBG] from='client.? 172.21.15.134:0/2022321846' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:46:42.080 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:46:42.081 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:46:43.212 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:43.212+0000 7ff24759e700 1 -- 172.21.15.134:0/590913248 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2480f62a0 msgr2=0x7ff2480f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:46:43.212 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:43.212+0000 7ff24759e700 1 --2- 172.21.15.134:0/590913248 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2480f62a0 0x7ff2480f66c0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7ff23c009fb0 tx=0x7ff23c009b30).stop 2024-01-21T11:46:43.212 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:43.212+0000 7ff24759e700 1 -- 172.21.15.134:0/590913248 shutdown_connections 2024-01-21T11:46:43.212 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:43.212+0000 7ff24759e700 1 --2- 172.21.15.134:0/590913248 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2480f62a0 0x7ff2480f66c0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:46:43.213 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:43.212+0000 7ff24759e700 1 -- 172.21.15.134:0/590913248 >> 172.21.15.134:0/590913248 conn(0x7ff2480f1aa0 msgr2=0x7ff2480f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:46:43.213 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:43.212+0000 7ff24759e700 1 -- 172.21.15.134:0/590913248 shutdown_connections 2024-01-21T11:46:43.213 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:43.212+0000 7ff24759e700 1 -- 172.21.15.134:0/590913248 wait complete. 2024-01-21T11:46:43.214 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:43.212+0000 7ff24759e700 1 Processor -- start 2024-01-21T11:46:43.214 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:43.212+0000 7ff24759e700 1 -- start start 2024-01-21T11:46:43.214 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:43.213+0000 7ff24759e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2480f62a0 0x7ff2481064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:46:43.214 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:43.213+0000 7ff24759e700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2481069a0 con 0x7ff2480f62a0 2024-01-21T11:46:43.214 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:43.213+0000 7ff24659c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2480f62a0 0x7ff2481064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:46:43.214 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:43.213+0000 7ff24659c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2480f62a0 0x7ff2481064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:34134/0 (socket says 172.21.15.134:34134) 2024-01-21T11:46:43.215 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:43.213+0000 7ff24659c700 1 -- 172.21.15.134:0/2837866114 learned_addr learned my addr 172.21.15.134:0/2837866114 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:46:43.215 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:43.213+0000 7ff24659c700 1 -- 172.21.15.134:0/2837866114 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff23c005040 con 0x7ff2480f62a0 2024-01-21T11:46:43.215 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:43.213+0000 7ff24659c700 1 --2- 172.21.15.134:0/2837866114 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2480f62a0 0x7ff2481064b0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7ff23c013510 tx=0x7ff23c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:46:43.215 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:43.214+0000 7ff236ffd700 1 -- 172.21.15.134:0/2837866114 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff23c009320 con 0x7ff2480f62a0 2024-01-21T11:46:43.215 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:43.214+0000 7ff24759e700 1 -- 172.21.15.134:0/2837866114 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff248106ba0 con 0x7ff2480f62a0 2024-01-21T11:46:43.215 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:43.214+0000 7ff236ffd700 1 -- 172.21.15.134:0/2837866114 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff23c013f00 con 0x7ff2480f62a0 2024-01-21T11:46:43.215 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:43.214+0000 7ff236ffd700 1 -- 172.21.15.134:0/2837866114 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff23c014dc0 con 0x7ff2480f62a0 2024-01-21T11:46:43.215 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:43.214+0000 7ff24759e700 1 -- 172.21.15.134:0/2837866114 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff248107030 con 0x7ff2480f62a0 2024-01-21T11:46:43.216 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:43.214+0000 7ff236ffd700 1 -- 172.21.15.134:0/2837866114 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff23c0095f0 con 0x7ff2480f62a0 2024-01-21T11:46:43.216 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:43.214+0000 7ff24759e700 1 -- 172.21.15.134:0/2837866114 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff248045ad0 con 0x7ff2480f62a0 2024-01-21T11:46:43.216 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:43.215+0000 7ff236ffd700 1 --2- 172.21.15.134:0/2837866114 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7ff23002ff60 0x7ff230032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:46:43.216 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:43.215+0000 7ff236ffd700 1 -- 172.21.15.134:0/2837866114 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff23c0410c0 con 0x7ff2480f62a0 2024-01-21T11:46:43.216 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:43.215+0000 7ff245d9b700 1 --2- 172.21.15.134:0/2837866114 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7ff23002ff60 0x7ff230032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:46:43.217 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:43.215+0000 7ff245d9b700 1 --2- 172.21.15.134:0/2837866114 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7ff23002ff60 0x7ff230032420 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7ff2380096a0 tx=0x7ff238006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:46:43.219 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:43.220+0000 7ff236ffd700 1 -- 172.21.15.134:0/2837866114 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff23c01c790 con 0x7ff2480f62a0 2024-01-21T11:46:43.427 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:43.427+0000 7ff24759e700 1 -- 172.21.15.134:0/2837866114 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff248107e80 con 0x7ff2480f62a0 2024-01-21T11:46:43.428 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:43.428+0000 7ff236ffd700 1 -- 172.21.15.134:0/2837866114 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff23c01c790 con 0x7ff2480f62a0 2024-01-21T11:46:43.428 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:46:43.428 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:46:43.429 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:43.429+0000 7ff24759e700 1 -- 172.21.15.134:0/2837866114 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7ff23002ff60 msgr2=0x7ff230032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:46:43.430 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:43.429+0000 7ff24759e700 1 --2- 172.21.15.134:0/2837866114 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7ff23002ff60 0x7ff230032420 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7ff2380096a0 tx=0x7ff238006b40).stop 2024-01-21T11:46:43.430 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:43.429+0000 7ff24759e700 1 -- 172.21.15.134:0/2837866114 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2480f62a0 msgr2=0x7ff2481064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:46:43.430 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:43.429+0000 7ff24759e700 1 --2- 172.21.15.134:0/2837866114 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2480f62a0 0x7ff2481064b0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7ff23c013510 tx=0x7ff23c00b040).stop 2024-01-21T11:46:43.430 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:43.429+0000 7ff24759e700 1 -- 172.21.15.134:0/2837866114 shutdown_connections 2024-01-21T11:46:43.430 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:43.429+0000 7ff24759e700 1 --2- 172.21.15.134:0/2837866114 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7ff23002ff60 0x7ff230032420 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:46:43.430 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:43.429+0000 7ff24759e700 1 --2- 172.21.15.134:0/2837866114 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2480f62a0 0x7ff2481064b0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:46:43.430 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:43.429+0000 7ff24759e700 1 -- 172.21.15.134:0/2837866114 >> 172.21.15.134:0/2837866114 conn(0x7ff2480f1aa0 msgr2=0x7ff2480f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:46:43.430 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:43.429+0000 7ff24759e700 1 -- 172.21.15.134:0/2837866114 shutdown_connections 2024-01-21T11:46:43.430 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:43.429+0000 7ff24759e700 1 -- 172.21.15.134:0/2837866114 wait complete. 2024-01-21T11:46:43.430 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:46:43.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:46:43 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:46:41.846339+0000 mgr.smithi029.blhxxk (mgr.14164) 141 : cluster [DBG] pgmap v134: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:46:43.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:46:43 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:46:43.428799+0000 mon.smithi029 (mon.0) 216 : audit [DBG] from='client.? 172.21.15.134:0/2837866114' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:46:44.851 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:46:44.851 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:46:45.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:46:45 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:46:43.846710+0000 mgr.smithi029.blhxxk (mgr.14164) 142 : cluster [DBG] pgmap v135: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:46:45.971 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:45.970+0000 7fbac0b90700 1 -- 172.21.15.134:0/1825895433 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbabc0f5410 msgr2=0x7fbabc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:46:45.971 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:45.970+0000 7fbac0b90700 1 --2- 172.21.15.134:0/1825895433 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbabc0f5410 0x7fbabc0f5830 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7fbab0009fb0 tx=0x7fbab0009b30).stop 2024-01-21T11:46:45.971 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:45.970+0000 7fbac0b90700 1 -- 172.21.15.134:0/1825895433 shutdown_connections 2024-01-21T11:46:45.971 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:45.970+0000 7fbac0b90700 1 --2- 172.21.15.134:0/1825895433 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbabc0f5410 0x7fbabc0f5830 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:46:45.971 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:45.970+0000 7fbac0b90700 1 -- 172.21.15.134:0/1825895433 >> 172.21.15.134:0/1825895433 conn(0x7fbabc0f0b70 msgr2=0x7fbabc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:46:45.971 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:45.970+0000 7fbac0b90700 1 -- 172.21.15.134:0/1825895433 shutdown_connections 2024-01-21T11:46:45.972 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:45.970+0000 7fbac0b90700 1 -- 172.21.15.134:0/1825895433 wait complete. 2024-01-21T11:46:45.972 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:45.971+0000 7fbac0b90700 1 Processor -- start 2024-01-21T11:46:45.972 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:45.971+0000 7fbac0b90700 1 -- start start 2024-01-21T11:46:45.972 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:45.971+0000 7fbac0b90700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbabc0f5410 0x7fbabc109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:46:45.972 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:45.971+0000 7fbac0b90700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbabc060240 con 0x7fbabc0f5410 2024-01-21T11:46:45.972 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:45.971+0000 7fbaba59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbabc0f5410 0x7fbabc109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:46:45.972 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:45.971+0000 7fbaba59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbabc0f5410 0x7fbabc109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:43438/0 (socket says 172.21.15.134:43438) 2024-01-21T11:46:45.972 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:45.971+0000 7fbaba59c700 1 -- 172.21.15.134:0/1459405704 learned_addr learned my addr 172.21.15.134:0/1459405704 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:46:45.972 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:45.971+0000 7fbaba59c700 1 -- 172.21.15.134:0/1459405704 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbab0005040 con 0x7fbabc0f5410 2024-01-21T11:46:45.972 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:45.972+0000 7fbaba59c700 1 --2- 172.21.15.134:0/1459405704 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbabc0f5410 0x7fbabc109430 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7fbab0013510 tx=0x7fbab000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:46:45.973 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:45.972+0000 7fbaaaffd700 1 -- 172.21.15.134:0/1459405704 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbab0009320 con 0x7fbabc0f5410 2024-01-21T11:46:45.973 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:45.972+0000 7fbac0b90700 1 -- 172.21.15.134:0/1459405704 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbabc060440 con 0x7fbabc0f5410 2024-01-21T11:46:45.973 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:45.972+0000 7fbaaaffd700 1 -- 172.21.15.134:0/1459405704 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbab0013f00 con 0x7fbabc0f5410 2024-01-21T11:46:45.973 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:45.972+0000 7fbaaaffd700 1 -- 172.21.15.134:0/1459405704 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbab0014d90 con 0x7fbabc0f5410 2024-01-21T11:46:45.973 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:45.972+0000 7fbac0b90700 1 -- 172.21.15.134:0/1459405704 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbabc109ae0 con 0x7fbabc0f5410 2024-01-21T11:46:45.974 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:45.973+0000 7fbaaaffd700 1 -- 172.21.15.134:0/1459405704 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbab00095f0 con 0x7fbabc0f5410 2024-01-21T11:46:45.974 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:45.973+0000 7fbaaaffd700 1 --2- 172.21.15.134:0/1459405704 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fbaa402ff60 0x7fbaa4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:46:45.974 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:45.973+0000 7fbaaaffd700 1 -- 172.21.15.134:0/1459405704 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbab0041080 con 0x7fbabc0f5410 2024-01-21T11:46:45.974 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:45.973+0000 7fbac0b90700 1 -- 172.21.15.134:0/1459405704 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbabc045ad0 con 0x7fbabc0f5410 2024-01-21T11:46:45.974 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:45.973+0000 7fbab9d9b700 1 --2- 172.21.15.134:0/1459405704 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fbaa402ff60 0x7fbaa4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:46:45.974 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:45.974+0000 7fbab9d9b700 1 --2- 172.21.15.134:0/1459405704 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fbaa402ff60 0x7fbaa4032420 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7fbaac0096a0 tx=0x7fbaac006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:46:45.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:45.978+0000 7fbaaaffd700 1 -- 172.21.15.134:0/1459405704 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbab0014420 con 0x7fbabc0f5410 2024-01-21T11:46:46.183 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:46.184+0000 7fbac0b90700 1 -- 172.21.15.134:0/1459405704 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbabc0faf50 con 0x7fbabc0f5410 2024-01-21T11:46:46.184 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:46.184+0000 7fbaaaffd700 1 -- 172.21.15.134:0/1459405704 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbab0010230 con 0x7fbabc0f5410 2024-01-21T11:46:46.184 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:46:46.184 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:46:46.185 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:46.186+0000 7fbac0b90700 1 -- 172.21.15.134:0/1459405704 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fbaa402ff60 msgr2=0x7fbaa4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:46:46.185 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:46.186+0000 7fbac0b90700 1 --2- 172.21.15.134:0/1459405704 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fbaa402ff60 0x7fbaa4032420 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7fbaac0096a0 tx=0x7fbaac006b40).stop 2024-01-21T11:46:46.185 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:46.186+0000 7fbac0b90700 1 -- 172.21.15.134:0/1459405704 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbabc0f5410 msgr2=0x7fbabc109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:46:46.186 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:46.186+0000 7fbac0b90700 1 --2- 172.21.15.134:0/1459405704 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbabc0f5410 0x7fbabc109430 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7fbab0013510 tx=0x7fbab000b040).stop 2024-01-21T11:46:46.186 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:46.186+0000 7fbac0b90700 1 -- 172.21.15.134:0/1459405704 shutdown_connections 2024-01-21T11:46:46.186 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:46.186+0000 7fbac0b90700 1 --2- 172.21.15.134:0/1459405704 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fbaa402ff60 0x7fbaa4032420 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:46:46.186 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:46.186+0000 7fbac0b90700 1 --2- 172.21.15.134:0/1459405704 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbabc0f5410 0x7fbabc109430 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:46:46.186 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:46.186+0000 7fbac0b90700 1 -- 172.21.15.134:0/1459405704 >> 172.21.15.134:0/1459405704 conn(0x7fbabc0f0b70 msgr2=0x7fbabc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:46:46.186 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:46.186+0000 7fbac0b90700 1 -- 172.21.15.134:0/1459405704 shutdown_connections 2024-01-21T11:46:46.186 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:46.186+0000 7fbac0b90700 1 -- 172.21.15.134:0/1459405704 wait complete. 2024-01-21T11:46:46.186 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:46:46.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:46:46 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:46:46.184749+0000 mon.smithi029 (mon.0) 217 : audit [DBG] from='client.? 172.21.15.134:0/1459405704' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:46:47.566 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:46:47.566 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:46:47.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:46:47 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:46:45.847080+0000 mgr.smithi029.blhxxk (mgr.14164) 143 : cluster [DBG] pgmap v136: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:46:48.690 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:48.686+0000 7f40520b7700 1 -- 172.21.15.134:0/2105543800 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f404c0f5410 msgr2=0x7f404c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:46:48.690 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:48.686+0000 7f40520b7700 1 --2- 172.21.15.134:0/2105543800 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f404c0f5410 0x7f404c0f5830 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7f4040009fb0 tx=0x7f4040009b30).stop 2024-01-21T11:46:48.690 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:48.686+0000 7f40520b7700 1 -- 172.21.15.134:0/2105543800 shutdown_connections 2024-01-21T11:46:48.690 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:48.686+0000 7f40520b7700 1 --2- 172.21.15.134:0/2105543800 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f404c0f5410 0x7f404c0f5830 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:46:48.690 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:48.686+0000 7f40520b7700 1 -- 172.21.15.134:0/2105543800 >> 172.21.15.134:0/2105543800 conn(0x7f404c0f0b70 msgr2=0x7f404c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:46:48.690 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:48.687+0000 7f40520b7700 1 -- 172.21.15.134:0/2105543800 shutdown_connections 2024-01-21T11:46:48.690 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:48.687+0000 7f40520b7700 1 -- 172.21.15.134:0/2105543800 wait complete. 2024-01-21T11:46:48.690 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:48.687+0000 7f40520b7700 1 Processor -- start 2024-01-21T11:46:48.691 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:48.687+0000 7f40520b7700 1 -- start start 2024-01-21T11:46:48.691 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:48.687+0000 7f40520b7700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f404c0f5410 0x7f404c10b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:46:48.691 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:48.687+0000 7f40520b7700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f404c107210 con 0x7f404c0f5410 2024-01-21T11:46:48.691 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:48.688+0000 7f404b7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f404c0f5410 0x7f404c10b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:46:48.691 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:48.688+0000 7f404b7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f404c0f5410 0x7f404c10b820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:43454/0 (socket says 172.21.15.134:43454) 2024-01-21T11:46:48.691 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:48.688+0000 7f404b7fe700 1 -- 172.21.15.134:0/1849246853 learned_addr learned my addr 172.21.15.134:0/1849246853 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:46:48.691 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:48.688+0000 7f404b7fe700 1 -- 172.21.15.134:0/1849246853 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4040005040 con 0x7f404c0f5410 2024-01-21T11:46:48.691 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:48.688+0000 7f404b7fe700 1 --2- 172.21.15.134:0/1849246853 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f404c0f5410 0x7f404c10b820 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f4040013510 tx=0x7f404000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:46:48.691 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:48.688+0000 7f4033fff700 1 -- 172.21.15.134:0/1849246853 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4040009320 con 0x7f404c0f5410 2024-01-21T11:46:48.691 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:48.688+0000 7f40520b7700 1 -- 172.21.15.134:0/1849246853 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f404c060220 con 0x7f404c0f5410 2024-01-21T11:46:48.691 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:48.688+0000 7f4033fff700 1 -- 172.21.15.134:0/1849246853 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4040013f00 con 0x7f404c0f5410 2024-01-21T11:46:48.692 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:48.688+0000 7f4033fff700 1 -- 172.21.15.134:0/1849246853 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4040014d90 con 0x7f404c0f5410 2024-01-21T11:46:48.692 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:48.688+0000 7f40520b7700 1 -- 172.21.15.134:0/1849246853 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f404c10bd10 con 0x7f404c0f5410 2024-01-21T11:46:48.692 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:48.689+0000 7f4033fff700 1 -- 172.21.15.134:0/1849246853 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f40400095f0 con 0x7f404c0f5410 2024-01-21T11:46:48.692 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:48.689+0000 7f40520b7700 1 -- 172.21.15.134:0/1849246853 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f404c045ad0 con 0x7f404c0f5410 2024-01-21T11:46:48.692 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:48.689+0000 7f4033fff700 1 --2- 172.21.15.134:0/1849246853 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f403402ffb0 0x7f4034032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:46:48.692 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:48.689+0000 7f4033fff700 1 -- 172.21.15.134:0/1849246853 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4040041050 con 0x7f404c0f5410 2024-01-21T11:46:48.692 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:48.689+0000 7f404affd700 1 --2- 172.21.15.134:0/1849246853 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f403402ffb0 0x7f4034032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:46:48.692 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:48.690+0000 7f404affd700 1 --2- 172.21.15.134:0/1849246853 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f403402ffb0 0x7f4034032470 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f403c0096a0 tx=0x7f403c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:46:48.694 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:48.694+0000 7f4033fff700 1 -- 172.21.15.134:0/1849246853 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f404001d7f0 con 0x7f404c0f5410 2024-01-21T11:46:48.902 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:48.901+0000 7f40520b7700 1 -- 172.21.15.134:0/1849246853 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f404c0fae60 con 0x7f404c0f5410 2024-01-21T11:46:48.902 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:48.902+0000 7f4033fff700 1 -- 172.21.15.134:0/1849246853 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4040010230 con 0x7f404c0f5410 2024-01-21T11:46:48.902 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:46:48.903 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:46:48.905 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:48.904+0000 7f40520b7700 1 -- 172.21.15.134:0/1849246853 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f403402ffb0 msgr2=0x7f4034032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:46:48.905 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:48.904+0000 7f40520b7700 1 --2- 172.21.15.134:0/1849246853 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f403402ffb0 0x7f4034032470 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f403c0096a0 tx=0x7f403c006b40).stop 2024-01-21T11:46:48.905 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:48.904+0000 7f40520b7700 1 -- 172.21.15.134:0/1849246853 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f404c0f5410 msgr2=0x7f404c10b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:46:48.905 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:48.904+0000 7f40520b7700 1 --2- 172.21.15.134:0/1849246853 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f404c0f5410 0x7f404c10b820 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f4040013510 tx=0x7f404000b040).stop 2024-01-21T11:46:48.905 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:48.904+0000 7f40520b7700 1 -- 172.21.15.134:0/1849246853 shutdown_connections 2024-01-21T11:46:48.905 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:48.904+0000 7f40520b7700 1 --2- 172.21.15.134:0/1849246853 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f403402ffb0 0x7f4034032470 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:46:48.905 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:48.904+0000 7f40520b7700 1 --2- 172.21.15.134:0/1849246853 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f404c0f5410 0x7f404c10b820 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:46:48.905 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:48.904+0000 7f40520b7700 1 -- 172.21.15.134:0/1849246853 >> 172.21.15.134:0/1849246853 conn(0x7f404c0f0b70 msgr2=0x7f404c0f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:46:48.905 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:48.904+0000 7f40520b7700 1 -- 172.21.15.134:0/1849246853 shutdown_connections 2024-01-21T11:46:48.905 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:48.904+0000 7f40520b7700 1 -- 172.21.15.134:0/1849246853 wait complete. 2024-01-21T11:46:48.905 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:46:48.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:46:48 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:46:47.847367+0000 mgr.smithi029.blhxxk (mgr.14164) 144 : cluster [DBG] pgmap v137: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:46:49.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:46:49 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:46:48.903239+0000 mon.smithi029 (mon.0) 218 : audit [DBG] from='client.? 172.21.15.134:0/1849246853' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:46:50.283 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:46:50.283 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:46:51.458 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:51.456+0000 7fa22b5d5700 1 -- 172.21.15.134:0/3064797337 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2240f5410 msgr2=0x7fa2240f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:46:51.458 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:51.456+0000 7fa22b5d5700 1 --2- 172.21.15.134:0/3064797337 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2240f5410 0x7fa2240f5830 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7fa214009fb0 tx=0x7fa214009b30).stop 2024-01-21T11:46:51.458 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:51.456+0000 7fa22b5d5700 1 -- 172.21.15.134:0/3064797337 shutdown_connections 2024-01-21T11:46:51.458 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:51.456+0000 7fa22b5d5700 1 --2- 172.21.15.134:0/3064797337 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2240f5410 0x7fa2240f5830 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:46:51.458 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:51.456+0000 7fa22b5d5700 1 -- 172.21.15.134:0/3064797337 >> 172.21.15.134:0/3064797337 conn(0x7fa2240f0b70 msgr2=0x7fa2240f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:46:51.458 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:51.456+0000 7fa22b5d5700 1 -- 172.21.15.134:0/3064797337 shutdown_connections 2024-01-21T11:46:51.459 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:51.456+0000 7fa22b5d5700 1 -- 172.21.15.134:0/3064797337 wait complete. 2024-01-21T11:46:51.459 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:51.457+0000 7fa22b5d5700 1 Processor -- start 2024-01-21T11:46:51.459 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:51.457+0000 7fa22b5d5700 1 -- start start 2024-01-21T11:46:51.459 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:51.457+0000 7fa22b5d5700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2240f5410 0x7fa22410b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:46:51.459 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:51.457+0000 7fa22b5d5700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa224107240 con 0x7fa2240f5410 2024-01-21T11:46:51.459 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:51.457+0000 7fa229371700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2240f5410 0x7fa22410b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:46:51.459 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:51.457+0000 7fa229371700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2240f5410 0x7fa22410b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:43466/0 (socket says 172.21.15.134:43466) 2024-01-21T11:46:51.459 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:51.457+0000 7fa229371700 1 -- 172.21.15.134:0/3715815892 learned_addr learned my addr 172.21.15.134:0/3715815892 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:46:51.459 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:51.457+0000 7fa229371700 1 -- 172.21.15.134:0/3715815892 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa214005040 con 0x7fa2240f5410 2024-01-21T11:46:51.459 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:51.458+0000 7fa229371700 1 --2- 172.21.15.134:0/3715815892 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2240f5410 0x7fa22410b850 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7fa214013510 tx=0x7fa21400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:46:51.459 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:51.458+0000 7fa219ffb700 1 -- 172.21.15.134:0/3715815892 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa214009320 con 0x7fa2240f5410 2024-01-21T11:46:51.459 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:51.458+0000 7fa219ffb700 1 -- 172.21.15.134:0/3715815892 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa214013f00 con 0x7fa2240f5410 2024-01-21T11:46:51.459 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:51.458+0000 7fa22b5d5700 1 -- 172.21.15.134:0/3715815892 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa224060220 con 0x7fa2240f5410 2024-01-21T11:46:51.459 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:51.458+0000 7fa219ffb700 1 -- 172.21.15.134:0/3715815892 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa214014d90 con 0x7fa2240f5410 2024-01-21T11:46:51.459 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:51.458+0000 7fa22b5d5700 1 -- 172.21.15.134:0/3715815892 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa22410bd40 con 0x7fa2240f5410 2024-01-21T11:46:51.460 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:51.459+0000 7fa219ffb700 1 -- 172.21.15.134:0/3715815892 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa214014420 con 0x7fa2240f5410 2024-01-21T11:46:51.460 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:51.459+0000 7fa22b5d5700 1 -- 172.21.15.134:0/3715815892 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa224045ad0 con 0x7fa2240f5410 2024-01-21T11:46:51.460 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:51.459+0000 7fa219ffb700 1 --2- 172.21.15.134:0/3715815892 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fa21002ff60 0x7fa210032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:46:51.460 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:51.459+0000 7fa219ffb700 1 -- 172.21.15.134:0/3715815892 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa214041050 con 0x7fa2240f5410 2024-01-21T11:46:51.461 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:51.459+0000 7fa228b70700 1 --2- 172.21.15.134:0/3715815892 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fa21002ff60 0x7fa210032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:46:51.461 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:51.460+0000 7fa228b70700 1 --2- 172.21.15.134:0/3715815892 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fa21002ff60 0x7fa210032420 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7fa2200096a0 tx=0x7fa220006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:46:51.464 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:51.464+0000 7fa219ffb700 1 -- 172.21.15.134:0/3715815892 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa2140095f0 con 0x7fa2240f5410 2024-01-21T11:46:51.677 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:51.677+0000 7fa22b5d5700 1 -- 172.21.15.134:0/3715815892 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa2240fae60 con 0x7fa2240f5410 2024-01-21T11:46:51.678 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:51.678+0000 7fa219ffb700 1 -- 172.21.15.134:0/3715815892 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa21401db70 con 0x7fa2240f5410 2024-01-21T11:46:51.678 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:46:51.678 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:46:51.680 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:51.680+0000 7fa22b5d5700 1 -- 172.21.15.134:0/3715815892 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fa21002ff60 msgr2=0x7fa210032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:46:51.680 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:51.680+0000 7fa22b5d5700 1 --2- 172.21.15.134:0/3715815892 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fa21002ff60 0x7fa210032420 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7fa2200096a0 tx=0x7fa220006b40).stop 2024-01-21T11:46:51.680 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:51.680+0000 7fa22b5d5700 1 -- 172.21.15.134:0/3715815892 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2240f5410 msgr2=0x7fa22410b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:46:51.680 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:51.680+0000 7fa22b5d5700 1 --2- 172.21.15.134:0/3715815892 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2240f5410 0x7fa22410b850 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7fa214013510 tx=0x7fa21400b040).stop 2024-01-21T11:46:51.680 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:51.680+0000 7fa22b5d5700 1 -- 172.21.15.134:0/3715815892 shutdown_connections 2024-01-21T11:46:51.680 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:51.680+0000 7fa22b5d5700 1 --2- 172.21.15.134:0/3715815892 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fa21002ff60 0x7fa210032420 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:46:51.680 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:51.680+0000 7fa22b5d5700 1 --2- 172.21.15.134:0/3715815892 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2240f5410 0x7fa22410b850 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:46:51.680 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:51.680+0000 7fa22b5d5700 1 -- 172.21.15.134:0/3715815892 >> 172.21.15.134:0/3715815892 conn(0x7fa2240f0b70 msgr2=0x7fa2240f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:46:51.681 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:51.680+0000 7fa22b5d5700 1 -- 172.21.15.134:0/3715815892 shutdown_connections 2024-01-21T11:46:51.681 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:51.680+0000 7fa22b5d5700 1 -- 172.21.15.134:0/3715815892 wait complete. 2024-01-21T11:46:51.681 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:46:51.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:46:51 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:46:49.847677+0000 mgr.smithi029.blhxxk (mgr.14164) 145 : cluster [DBG] pgmap v138: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:46:52.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:46:52 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:46:51.679170+0000 mon.smithi029 (mon.0) 219 : audit [DBG] from='client.? 172.21.15.134:0/3715815892' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:46:53.061 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:46:53.062 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:46:53.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:46:53 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:46:51.848031+0000 mgr.smithi029.blhxxk (mgr.14164) 146 : cluster [DBG] pgmap v139: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:46:54.174 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:54.173+0000 7f94f986d700 1 -- 172.21.15.134:0/1667239053 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94f40f47c0 msgr2=0x7f94f40f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:46:54.174 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:54.173+0000 7f94f986d700 1 --2- 172.21.15.134:0/1667239053 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94f40f47c0 0x7f94f40f4be0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7f94e8009fb0 tx=0x7f94e8009b30).stop 2024-01-21T11:46:54.174 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:54.173+0000 7f94f986d700 1 -- 172.21.15.134:0/1667239053 shutdown_connections 2024-01-21T11:46:54.174 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:54.173+0000 7f94f986d700 1 --2- 172.21.15.134:0/1667239053 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94f40f47c0 0x7f94f40f4be0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:46:54.174 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:54.173+0000 7f94f986d700 1 -- 172.21.15.134:0/1667239053 >> 172.21.15.134:0/1667239053 conn(0x7f94f40f02e0 msgr2=0x7f94f40f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:46:54.174 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:54.173+0000 7f94f986d700 1 -- 172.21.15.134:0/1667239053 shutdown_connections 2024-01-21T11:46:54.174 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:54.173+0000 7f94f986d700 1 -- 172.21.15.134:0/1667239053 wait complete. 2024-01-21T11:46:54.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:54.174+0000 7f94f986d700 1 Processor -- start 2024-01-21T11:46:54.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:54.174+0000 7f94f986d700 1 -- start start 2024-01-21T11:46:54.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:54.174+0000 7f94f986d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94f40f47c0 0x7f94f4106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:46:54.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:54.174+0000 7f94f986d700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94f4106bf0 con 0x7f94f40f47c0 2024-01-21T11:46:54.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:54.174+0000 7f94f2ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94f40f47c0 0x7f94f4106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:46:54.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:54.174+0000 7f94f2ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94f40f47c0 0x7f94f4106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:43478/0 (socket says 172.21.15.134:43478) 2024-01-21T11:46:54.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:54.174+0000 7f94f2ffd700 1 -- 172.21.15.134:0/562971122 learned_addr learned my addr 172.21.15.134:0/562971122 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:46:54.177 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:54.175+0000 7f94f2ffd700 1 -- 172.21.15.134:0/562971122 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94e8005040 con 0x7f94f40f47c0 2024-01-21T11:46:54.177 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:54.175+0000 7f94f2ffd700 1 --2- 172.21.15.134:0/562971122 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94f40f47c0 0x7f94f4106700 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7f94e8013510 tx=0x7f94e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:46:54.177 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:54.175+0000 7f94db7fe700 1 -- 172.21.15.134:0/562971122 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f94e8009320 con 0x7f94f40f47c0 2024-01-21T11:46:54.177 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:54.175+0000 7f94f986d700 1 -- 172.21.15.134:0/562971122 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94f4106df0 con 0x7f94f40f47c0 2024-01-21T11:46:54.177 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:54.175+0000 7f94db7fe700 1 -- 172.21.15.134:0/562971122 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f94e8013f00 con 0x7f94f40f47c0 2024-01-21T11:46:54.177 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:54.175+0000 7f94f986d700 1 -- 172.21.15.134:0/562971122 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94f4107280 con 0x7f94f40f47c0 2024-01-21T11:46:54.177 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:54.175+0000 7f94db7fe700 1 -- 172.21.15.134:0/562971122 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f94e8014d90 con 0x7f94f40f47c0 2024-01-21T11:46:54.177 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:54.176+0000 7f94db7fe700 1 -- 172.21.15.134:0/562971122 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f94e80263a0 con 0x7f94f40f47c0 2024-01-21T11:46:54.177 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:54.176+0000 7f94db7fe700 1 --2- 172.21.15.134:0/562971122 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f94dc02ff60 0x7f94dc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:46:54.177 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:54.176+0000 7f94db7fe700 1 -- 172.21.15.134:0/562971122 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f94e80411d0 con 0x7f94f40f47c0 2024-01-21T11:46:54.177 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:54.176+0000 7f94d97fa700 1 -- 172.21.15.134:0/562971122 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94cc004fa0 con 0x7f94f40f47c0 2024-01-21T11:46:54.177 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:54.176+0000 7f94f27fc700 1 --2- 172.21.15.134:0/562971122 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f94dc02ff60 0x7f94dc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:46:54.178 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:54.177+0000 7f94f27fc700 1 --2- 172.21.15.134:0/562971122 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f94dc02ff60 0x7f94dc032420 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f94e40096a0 tx=0x7f94e4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:46:54.181 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:54.181+0000 7f94db7fe700 1 -- 172.21.15.134:0/562971122 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f94e8012920 con 0x7f94f40f47c0 2024-01-21T11:46:54.388 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:54.388+0000 7f94d97fa700 1 -- 172.21.15.134:0/562971122 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f94cc005670 con 0x7f94f40f47c0 2024-01-21T11:46:54.389 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:54.389+0000 7f94db7fe700 1 -- 172.21.15.134:0/562971122 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f94e802b020 con 0x7f94f40f47c0 2024-01-21T11:46:54.389 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:46:54.389 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:46:54.390 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:54.390+0000 7f94d97fa700 1 -- 172.21.15.134:0/562971122 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f94dc02ff60 msgr2=0x7f94dc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:46:54.391 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:54.390+0000 7f94d97fa700 1 --2- 172.21.15.134:0/562971122 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f94dc02ff60 0x7f94dc032420 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f94e40096a0 tx=0x7f94e4006b40).stop 2024-01-21T11:46:54.391 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:54.390+0000 7f94d97fa700 1 -- 172.21.15.134:0/562971122 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94f40f47c0 msgr2=0x7f94f4106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:46:54.391 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:54.390+0000 7f94d97fa700 1 --2- 172.21.15.134:0/562971122 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94f40f47c0 0x7f94f4106700 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7f94e8013510 tx=0x7f94e800b040).stop 2024-01-21T11:46:54.391 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:54.390+0000 7f94d97fa700 1 -- 172.21.15.134:0/562971122 shutdown_connections 2024-01-21T11:46:54.391 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:54.390+0000 7f94d97fa700 1 --2- 172.21.15.134:0/562971122 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f94dc02ff60 0x7f94dc032420 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:46:54.391 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:54.390+0000 7f94d97fa700 1 --2- 172.21.15.134:0/562971122 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94f40f47c0 0x7f94f4106700 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:46:54.391 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:54.390+0000 7f94d97fa700 1 -- 172.21.15.134:0/562971122 >> 172.21.15.134:0/562971122 conn(0x7f94f40f02e0 msgr2=0x7f94f40f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:46:54.391 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:54.390+0000 7f94d97fa700 1 -- 172.21.15.134:0/562971122 shutdown_connections 2024-01-21T11:46:54.391 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:54.391+0000 7f94d97fa700 1 -- 172.21.15.134:0/562971122 wait complete. 2024-01-21T11:46:54.391 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:46:54.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:46:54 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:46:54.389827+0000 mon.smithi029 (mon.0) 220 : audit [DBG] from='client.? 172.21.15.134:0/562971122' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:46:55.778 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:46:55.778 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:46:55.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:46:55 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:46:53.848365+0000 mgr.smithi029.blhxxk (mgr.14164) 147 : cluster [DBG] pgmap v140: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:46:56.942 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:56.940+0000 7fecc4bf9700 1 -- 172.21.15.134:0/3679134450 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fecc00f4740 msgr2=0x7fecc00f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:46:56.942 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:56.940+0000 7fecc4bf9700 1 --2- 172.21.15.134:0/3679134450 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fecc00f4740 0x7fecc00f4b60 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7feca8009fb0 tx=0x7feca8009b30).stop 2024-01-21T11:46:56.942 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:56.941+0000 7fecc4bf9700 1 -- 172.21.15.134:0/3679134450 shutdown_connections 2024-01-21T11:46:56.942 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:56.941+0000 7fecc4bf9700 1 --2- 172.21.15.134:0/3679134450 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fecc00f4740 0x7fecc00f4b60 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:46:56.942 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:56.941+0000 7fecc4bf9700 1 -- 172.21.15.134:0/3679134450 >> 172.21.15.134:0/3679134450 conn(0x7fecc00f0260 msgr2=0x7fecc00f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:46:56.942 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:56.941+0000 7fecc4bf9700 1 -- 172.21.15.134:0/3679134450 shutdown_connections 2024-01-21T11:46:56.942 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:56.941+0000 7fecc4bf9700 1 -- 172.21.15.134:0/3679134450 wait complete. 2024-01-21T11:46:56.942 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:56.941+0000 7fecc4bf9700 1 Processor -- start 2024-01-21T11:46:56.942 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:56.941+0000 7fecc4bf9700 1 -- start start 2024-01-21T11:46:56.942 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:56.941+0000 7fecc4bf9700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fecc00f4740 0x7fecc00f72f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:46:56.942 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:56.941+0000 7fecc4bf9700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecc00f5770 con 0x7fecc00f4740 2024-01-21T11:46:56.942 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:56.941+0000 7fecbe59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fecc00f4740 0x7fecc00f72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:46:56.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:56.942+0000 7fecbe59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fecc00f4740 0x7fecc00f72f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:60196/0 (socket says 172.21.15.134:60196) 2024-01-21T11:46:56.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:56.942+0000 7fecbe59c700 1 -- 172.21.15.134:0/850701859 learned_addr learned my addr 172.21.15.134:0/850701859 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:46:56.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:56.942+0000 7fecbe59c700 1 -- 172.21.15.134:0/850701859 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feca8005040 con 0x7fecc00f4740 2024-01-21T11:46:56.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:56.942+0000 7fecbe59c700 1 --2- 172.21.15.134:0/850701859 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fecc00f4740 0x7fecc00f72f0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7feca8013510 tx=0x7feca800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:46:56.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:56.942+0000 7fecb6ffd700 1 -- 172.21.15.134:0/850701859 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feca8009320 con 0x7fecc00f4740 2024-01-21T11:46:56.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:56.942+0000 7fecc4bf9700 1 -- 172.21.15.134:0/850701859 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecc00f5910 con 0x7fecc00f4740 2024-01-21T11:46:56.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:56.942+0000 7fecb6ffd700 1 -- 172.21.15.134:0/850701859 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7feca8013f00 con 0x7fecc00f4740 2024-01-21T11:46:56.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:56.942+0000 7fecb6ffd700 1 -- 172.21.15.134:0/850701859 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feca8014d90 con 0x7fecc00f4740 2024-01-21T11:46:56.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:56.942+0000 7fecc4bf9700 1 -- 172.21.15.134:0/850701859 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecc00f5da0 con 0x7fecc00f4740 2024-01-21T11:46:56.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:56.943+0000 7fecb6ffd700 1 -- 172.21.15.134:0/850701859 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7feca8014420 con 0x7fecc00f4740 2024-01-21T11:46:56.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:56.943+0000 7fecc4bf9700 1 -- 172.21.15.134:0/850701859 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecc0059100 con 0x7fecc00f4740 2024-01-21T11:46:56.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:56.943+0000 7fecb6ffd700 1 --2- 172.21.15.134:0/850701859 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fecac02ff60 0x7fecac032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:46:56.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:56.943+0000 7fecb6ffd700 1 -- 172.21.15.134:0/850701859 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7feca80410c0 con 0x7fecc00f4740 2024-01-21T11:46:56.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:56.943+0000 7fecbdd9b700 1 --2- 172.21.15.134:0/850701859 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fecac02ff60 0x7fecac032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:46:56.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:56.944+0000 7fecbdd9b700 1 --2- 172.21.15.134:0/850701859 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fecac02ff60 0x7fecac032420 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7fecb00096a0 tx=0x7fecb0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:46:56.948 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:56.948+0000 7fecb6ffd700 1 -- 172.21.15.134:0/850701859 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7feca80095f0 con 0x7fecc00f4740 2024-01-21T11:46:57.155 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:57.155+0000 7fecc4bf9700 1 -- 172.21.15.134:0/850701859 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fecc00fbf40 con 0x7fecc00f4740 2024-01-21T11:46:57.156 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:57.155+0000 7fecb6ffd700 1 -- 172.21.15.134:0/850701859 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7feca801db70 con 0x7fecc00f4740 2024-01-21T11:46:57.156 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:46:57.156 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:46:57.157 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:57.157+0000 7fecc4bf9700 1 -- 172.21.15.134:0/850701859 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fecac02ff60 msgr2=0x7fecac032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:46:57.157 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:57.157+0000 7fecc4bf9700 1 --2- 172.21.15.134:0/850701859 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fecac02ff60 0x7fecac032420 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7fecb00096a0 tx=0x7fecb0006b40).stop 2024-01-21T11:46:57.158 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:57.157+0000 7fecc4bf9700 1 -- 172.21.15.134:0/850701859 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fecc00f4740 msgr2=0x7fecc00f72f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:46:57.158 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:57.157+0000 7fecc4bf9700 1 --2- 172.21.15.134:0/850701859 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fecc00f4740 0x7fecc00f72f0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7feca8013510 tx=0x7feca800b040).stop 2024-01-21T11:46:57.158 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:57.157+0000 7fecc4bf9700 1 -- 172.21.15.134:0/850701859 shutdown_connections 2024-01-21T11:46:57.158 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:57.157+0000 7fecc4bf9700 1 --2- 172.21.15.134:0/850701859 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fecac02ff60 0x7fecac032420 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:46:57.158 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:57.157+0000 7fecc4bf9700 1 --2- 172.21.15.134:0/850701859 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fecc00f4740 0x7fecc00f72f0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:46:57.158 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:57.157+0000 7fecc4bf9700 1 -- 172.21.15.134:0/850701859 >> 172.21.15.134:0/850701859 conn(0x7fecc00f0260 msgr2=0x7fecc00f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:46:57.158 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:57.157+0000 7fecc4bf9700 1 -- 172.21.15.134:0/850701859 shutdown_connections 2024-01-21T11:46:57.158 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:57.157+0000 7fecc4bf9700 1 -- 172.21.15.134:0/850701859 wait complete. 2024-01-21T11:46:57.158 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:46:57.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:46:57 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:46:55.848650+0000 mgr.smithi029.blhxxk (mgr.14164) 148 : cluster [DBG] pgmap v141: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:46:57.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:46:57 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:46:57.156541+0000 mon.smithi029 (mon.0) 221 : audit [DBG] from='client.? 172.21.15.134:0/850701859' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:46:58.529 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:46:58.529 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:46:59.703 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:59.702+0000 7f6f0f58a700 1 -- 172.21.15.134:0/966312321 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f080f51a0 msgr2=0x7f6f080f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:46:59.703 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:59.702+0000 7f6f0f58a700 1 --2- 172.21.15.134:0/966312321 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f080f51a0 0x7f6f080f55c0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7f6f04009fb0 tx=0x7f6f04009b30).stop 2024-01-21T11:46:59.703 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:59.702+0000 7f6f0f58a700 1 -- 172.21.15.134:0/966312321 shutdown_connections 2024-01-21T11:46:59.703 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:59.702+0000 7f6f0f58a700 1 --2- 172.21.15.134:0/966312321 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f080f51a0 0x7f6f080f55c0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:46:59.703 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:59.702+0000 7f6f0f58a700 1 -- 172.21.15.134:0/966312321 >> 172.21.15.134:0/966312321 conn(0x7f6f080f09a0 msgr2=0x7f6f080f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:46:59.704 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:59.702+0000 7f6f0f58a700 1 -- 172.21.15.134:0/966312321 shutdown_connections 2024-01-21T11:46:59.704 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:59.702+0000 7f6f0f58a700 1 -- 172.21.15.134:0/966312321 wait complete. 2024-01-21T11:46:59.704 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:59.702+0000 7f6f0f58a700 1 Processor -- start 2024-01-21T11:46:59.704 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:59.702+0000 7f6f0f58a700 1 -- start start 2024-01-21T11:46:59.704 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:59.703+0000 7f6f0f58a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f080f51a0 0x7f6f081053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:46:59.704 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:59.703+0000 7f6f0f58a700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f081058a0 con 0x7f6f080f51a0 2024-01-21T11:46:59.704 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:59.703+0000 7f6f0e588700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f080f51a0 0x7f6f081053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:46:59.704 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:59.703+0000 7f6f0e588700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f080f51a0 0x7f6f081053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:60218/0 (socket says 172.21.15.134:60218) 2024-01-21T11:46:59.704 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:59.703+0000 7f6f0e588700 1 -- 172.21.15.134:0/5382945 learned_addr learned my addr 172.21.15.134:0/5382945 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:46:59.704 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:59.703+0000 7f6f0e588700 1 -- 172.21.15.134:0/5382945 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f04005040 con 0x7f6f080f51a0 2024-01-21T11:46:59.704 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:59.703+0000 7f6f0e588700 1 --2- 172.21.15.134:0/5382945 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f080f51a0 0x7f6f081053b0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f6f04013510 tx=0x7f6f0400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:46:59.705 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:59.703+0000 7f6efeffd700 1 -- 172.21.15.134:0/5382945 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f04009320 con 0x7f6f080f51a0 2024-01-21T11:46:59.705 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:59.703+0000 7f6f0f58a700 1 -- 172.21.15.134:0/5382945 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f08105aa0 con 0x7f6f080f51a0 2024-01-21T11:46:59.706 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:59.703+0000 7f6efeffd700 1 -- 172.21.15.134:0/5382945 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6f04013f00 con 0x7f6f080f51a0 2024-01-21T11:46:59.706 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:59.703+0000 7f6efeffd700 1 -- 172.21.15.134:0/5382945 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f04014dc0 con 0x7f6f080f51a0 2024-01-21T11:46:59.706 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:59.704+0000 7f6f0f58a700 1 -- 172.21.15.134:0/5382945 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f08105f30 con 0x7f6f080f51a0 2024-01-21T11:46:59.706 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:59.704+0000 7f6efeffd700 1 -- 172.21.15.134:0/5382945 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6f0401d3d0 con 0x7f6f080f51a0 2024-01-21T11:46:59.706 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:59.704+0000 7f6efeffd700 1 --2- 172.21.15.134:0/5382945 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f6ef402ff60 0x7f6ef4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:46:59.706 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:59.704+0000 7f6efeffd700 1 -- 172.21.15.134:0/5382945 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6f04041000 con 0x7f6f080f51a0 2024-01-21T11:46:59.706 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:59.704+0000 7f6f0f58a700 1 -- 172.21.15.134:0/5382945 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f08045ad0 con 0x7f6f080f51a0 2024-01-21T11:46:59.706 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:59.705+0000 7f6f0dd87700 1 --2- 172.21.15.134:0/5382945 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f6ef402ff60 0x7f6ef4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:46:59.706 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:59.705+0000 7f6f0dd87700 1 --2- 172.21.15.134:0/5382945 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f6ef402ff60 0x7f6ef4032420 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f6ef80096a0 tx=0x7f6ef8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:46:59.709 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:59.709+0000 7f6efeffd700 1 -- 172.21.15.134:0/5382945 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6f04012920 con 0x7f6f080f51a0 2024-01-21T11:46:59.915 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:59.915+0000 7f6f0f58a700 1 -- 172.21.15.134:0/5382945 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6f08106c70 con 0x7f6f080f51a0 2024-01-21T11:46:59.916 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:59.916+0000 7f6efeffd700 1 -- 172.21.15.134:0/5382945 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6f04012330 con 0x7f6f080f51a0 2024-01-21T11:46:59.916 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:46:59.917 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:46:59.918 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:59.918+0000 7f6f0f58a700 1 -- 172.21.15.134:0/5382945 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f6ef402ff60 msgr2=0x7f6ef4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:46:59.918 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:59.918+0000 7f6f0f58a700 1 --2- 172.21.15.134:0/5382945 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f6ef402ff60 0x7f6ef4032420 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f6ef80096a0 tx=0x7f6ef8006b40).stop 2024-01-21T11:46:59.918 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:59.918+0000 7f6f0f58a700 1 -- 172.21.15.134:0/5382945 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f080f51a0 msgr2=0x7f6f081053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:46:59.918 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:59.918+0000 7f6f0f58a700 1 --2- 172.21.15.134:0/5382945 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f080f51a0 0x7f6f081053b0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f6f04013510 tx=0x7f6f0400b040).stop 2024-01-21T11:46:59.918 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:59.918+0000 7f6f0f58a700 1 -- 172.21.15.134:0/5382945 shutdown_connections 2024-01-21T11:46:59.918 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:59.918+0000 7f6f0f58a700 1 --2- 172.21.15.134:0/5382945 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f6ef402ff60 0x7f6ef4032420 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:46:59.918 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:59.918+0000 7f6f0f58a700 1 --2- 172.21.15.134:0/5382945 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f080f51a0 0x7f6f081053b0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:46:59.919 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:59.918+0000 7f6f0f58a700 1 -- 172.21.15.134:0/5382945 >> 172.21.15.134:0/5382945 conn(0x7f6f080f09a0 msgr2=0x7f6f080f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:46:59.919 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:59.918+0000 7f6f0f58a700 1 -- 172.21.15.134:0/5382945 shutdown_connections 2024-01-21T11:46:59.919 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:46:59.918+0000 7f6f0f58a700 1 -- 172.21.15.134:0/5382945 wait complete. 2024-01-21T11:46:59.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:46:59 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:46:57.848989+0000 mgr.smithi029.blhxxk (mgr.14164) 149 : cluster [DBG] pgmap v142: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:46:59.919 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:47:00.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:00 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:46:59.917296+0000 mon.smithi029 (mon.0) 222 : audit [DBG] from='client.? 172.21.15.134:0/5382945' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:47:01.313 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:47:01.314 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:47:01.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:01 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:46:59.849346+0000 mgr.smithi029.blhxxk (mgr.14164) 150 : cluster [DBG] pgmap v143: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:47:02.466 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:02.465+0000 7fb4379ba700 1 -- 172.21.15.134:0/2587983099 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4300f5410 msgr2=0x7fb4300f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:47:02.467 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:02.465+0000 7fb4379ba700 1 --2- 172.21.15.134:0/2587983099 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4300f5410 0x7fb4300f5830 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7fb420009fb0 tx=0x7fb420009b30).stop 2024-01-21T11:47:02.467 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:02.465+0000 7fb4379ba700 1 -- 172.21.15.134:0/2587983099 shutdown_connections 2024-01-21T11:47:02.467 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:02.465+0000 7fb4379ba700 1 --2- 172.21.15.134:0/2587983099 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4300f5410 0x7fb4300f5830 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:47:02.467 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:02.465+0000 7fb4379ba700 1 -- 172.21.15.134:0/2587983099 >> 172.21.15.134:0/2587983099 conn(0x7fb4300f0b70 msgr2=0x7fb4300f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:47:02.467 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:02.465+0000 7fb4379ba700 1 -- 172.21.15.134:0/2587983099 shutdown_connections 2024-01-21T11:47:02.467 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:02.465+0000 7fb4379ba700 1 -- 172.21.15.134:0/2587983099 wait complete. 2024-01-21T11:47:02.467 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:02.465+0000 7fb4379ba700 1 Processor -- start 2024-01-21T11:47:02.467 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:02.465+0000 7fb4379ba700 1 -- start start 2024-01-21T11:47:02.467 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:02.465+0000 7fb4379ba700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4300f5410 0x7fb430109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:47:02.467 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:02.465+0000 7fb4379ba700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb430060240 con 0x7fb4300f5410 2024-01-21T11:47:02.467 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:02.466+0000 7fb435756700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4300f5410 0x7fb430109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:47:02.467 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:02.466+0000 7fb435756700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4300f5410 0x7fb430109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:60234/0 (socket says 172.21.15.134:60234) 2024-01-21T11:47:02.468 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:02.466+0000 7fb435756700 1 -- 172.21.15.134:0/2192829366 learned_addr learned my addr 172.21.15.134:0/2192829366 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:47:02.468 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:02.466+0000 7fb435756700 1 -- 172.21.15.134:0/2192829366 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb420005040 con 0x7fb4300f5410 2024-01-21T11:47:02.468 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:02.466+0000 7fb435756700 1 --2- 172.21.15.134:0/2192829366 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4300f5410 0x7fb430109430 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7fb420013510 tx=0x7fb42000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:47:02.468 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:02.466+0000 7fb425ffb700 1 -- 172.21.15.134:0/2192829366 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb420009320 con 0x7fb4300f5410 2024-01-21T11:47:02.468 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:02.466+0000 7fb4379ba700 1 -- 172.21.15.134:0/2192829366 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb430060440 con 0x7fb4300f5410 2024-01-21T11:47:02.468 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:02.466+0000 7fb425ffb700 1 -- 172.21.15.134:0/2192829366 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb420013f00 con 0x7fb4300f5410 2024-01-21T11:47:02.468 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:02.466+0000 7fb425ffb700 1 -- 172.21.15.134:0/2192829366 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb420014d90 con 0x7fb4300f5410 2024-01-21T11:47:02.468 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:02.466+0000 7fb4379ba700 1 -- 172.21.15.134:0/2192829366 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb430109ae0 con 0x7fb4300f5410 2024-01-21T11:47:02.469 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:02.467+0000 7fb425ffb700 1 -- 172.21.15.134:0/2192829366 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb4200095f0 con 0x7fb4300f5410 2024-01-21T11:47:02.469 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:02.467+0000 7fb4379ba700 1 -- 172.21.15.134:0/2192829366 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb430045ad0 con 0x7fb4300f5410 2024-01-21T11:47:02.469 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:02.467+0000 7fb425ffb700 1 --2- 172.21.15.134:0/2192829366 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fb41c02ff60 0x7fb41c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:47:02.469 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:02.467+0000 7fb425ffb700 1 -- 172.21.15.134:0/2192829366 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb420041000 con 0x7fb4300f5410 2024-01-21T11:47:02.469 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:02.468+0000 7fb434f55700 1 --2- 172.21.15.134:0/2192829366 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fb41c02ff60 0x7fb41c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:47:02.469 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:02.468+0000 7fb434f55700 1 --2- 172.21.15.134:0/2192829366 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fb41c02ff60 0x7fb41c032420 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7fb42c0096a0 tx=0x7fb42c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:47:02.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:02.472+0000 7fb425ffb700 1 -- 172.21.15.134:0/2192829366 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb42001d7f0 con 0x7fb4300f5410 2024-01-21T11:47:02.676 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:02.675+0000 7fb4379ba700 1 -- 172.21.15.134:0/2192829366 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb4300fae60 con 0x7fb4300f5410 2024-01-21T11:47:02.676 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:02.676+0000 7fb425ffb700 1 -- 172.21.15.134:0/2192829366 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb420010230 con 0x7fb4300f5410 2024-01-21T11:47:02.676 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:47:02.677 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:47:02.678 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:02.678+0000 7fb4379ba700 1 -- 172.21.15.134:0/2192829366 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fb41c02ff60 msgr2=0x7fb41c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:47:02.678 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:02.678+0000 7fb4379ba700 1 --2- 172.21.15.134:0/2192829366 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fb41c02ff60 0x7fb41c032420 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7fb42c0096a0 tx=0x7fb42c006b40).stop 2024-01-21T11:47:02.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:02.678+0000 7fb4379ba700 1 -- 172.21.15.134:0/2192829366 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4300f5410 msgr2=0x7fb430109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:47:02.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:02.678+0000 7fb4379ba700 1 --2- 172.21.15.134:0/2192829366 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4300f5410 0x7fb430109430 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7fb420013510 tx=0x7fb42000b040).stop 2024-01-21T11:47:02.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:02.678+0000 7fb4379ba700 1 -- 172.21.15.134:0/2192829366 shutdown_connections 2024-01-21T11:47:02.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:02.678+0000 7fb4379ba700 1 --2- 172.21.15.134:0/2192829366 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fb41c02ff60 0x7fb41c032420 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:47:02.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:02.678+0000 7fb4379ba700 1 --2- 172.21.15.134:0/2192829366 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4300f5410 0x7fb430109430 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:47:02.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:02.678+0000 7fb4379ba700 1 -- 172.21.15.134:0/2192829366 >> 172.21.15.134:0/2192829366 conn(0x7fb4300f0b70 msgr2=0x7fb4300f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:47:02.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:02.678+0000 7fb4379ba700 1 -- 172.21.15.134:0/2192829366 shutdown_connections 2024-01-21T11:47:02.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:02.678+0000 7fb4379ba700 1 -- 172.21.15.134:0/2192829366 wait complete. 2024-01-21T11:47:02.679 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:47:02.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:02 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:47:01.941893+0000 mon.smithi029 (mon.0) 223 : audit [INF] from='mgr.14164 172.21.15.29:0/3252518221' entity='mgr.smithi029.blhxxk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi029.blhxxk/mirror_snapshot_schedule"}]: dispatch 2024-01-21T11:47:02.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:02 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:47:01.943919+0000 mon.smithi029 (mon.0) 224 : audit [INF] from='mgr.14164 172.21.15.29:0/3252518221' entity='mgr.smithi029.blhxxk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi029.blhxxk/trash_purge_schedule"}]: dispatch 2024-01-21T11:47:03.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:03 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:47:01.849714+0000 mgr.smithi029.blhxxk (mgr.14164) 151 : cluster [DBG] pgmap v144: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:47:03.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:03 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:47:02.677221+0000 mon.smithi029 (mon.0) 225 : audit [DBG] from='client.? 172.21.15.134:0/2192829366' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:47:04.047 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:47:04.048 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:47:05.175 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:05.174+0000 7f5097515700 1 -- 172.21.15.134:0/575628580 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50900f51a0 msgr2=0x7f50900f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:47:05.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:05.174+0000 7f5097515700 1 --2- 172.21.15.134:0/575628580 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50900f51a0 0x7f50900f55c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7f508c009fb0 tx=0x7f508c009b30).stop 2024-01-21T11:47:05.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:05.174+0000 7f5097515700 1 -- 172.21.15.134:0/575628580 shutdown_connections 2024-01-21T11:47:05.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:05.174+0000 7f5097515700 1 --2- 172.21.15.134:0/575628580 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50900f51a0 0x7f50900f55c0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:47:05.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:05.174+0000 7f5097515700 1 -- 172.21.15.134:0/575628580 >> 172.21.15.134:0/575628580 conn(0x7f50900f09a0 msgr2=0x7f50900f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:47:05.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:05.174+0000 7f5097515700 1 -- 172.21.15.134:0/575628580 shutdown_connections 2024-01-21T11:47:05.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:05.174+0000 7f5097515700 1 -- 172.21.15.134:0/575628580 wait complete. 2024-01-21T11:47:05.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:05.174+0000 7f5097515700 1 Processor -- start 2024-01-21T11:47:05.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:05.174+0000 7f5097515700 1 -- start start 2024-01-21T11:47:05.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:05.175+0000 7f5097515700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50900f51a0 0x7f50901091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:47:05.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:05.175+0000 7f5097515700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5090060240 con 0x7f50900f51a0 2024-01-21T11:47:05.177 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:05.175+0000 7f5096513700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50900f51a0 0x7f50901091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:47:05.177 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:05.175+0000 7f5096513700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50900f51a0 0x7f50901091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:33798/0 (socket says 172.21.15.134:33798) 2024-01-21T11:47:05.177 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:05.175+0000 7f5096513700 1 -- 172.21.15.134:0/2886026716 learned_addr learned my addr 172.21.15.134:0/2886026716 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:47:05.177 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:05.175+0000 7f5096513700 1 -- 172.21.15.134:0/2886026716 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f508c005040 con 0x7f50900f51a0 2024-01-21T11:47:05.177 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:05.175+0000 7f5096513700 1 --2- 172.21.15.134:0/2886026716 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50900f51a0 0x7f50901091b0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f508c013510 tx=0x7f508c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:47:05.177 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:05.175+0000 7f5086ffd700 1 -- 172.21.15.134:0/2886026716 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f508c009320 con 0x7f50900f51a0 2024-01-21T11:47:05.177 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:05.175+0000 7f5097515700 1 -- 172.21.15.134:0/2886026716 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5090060440 con 0x7f50900f51a0 2024-01-21T11:47:05.177 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:05.175+0000 7f5086ffd700 1 -- 172.21.15.134:0/2886026716 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f508c013f00 con 0x7f50900f51a0 2024-01-21T11:47:05.177 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:05.176+0000 7f5086ffd700 1 -- 172.21.15.134:0/2886026716 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f508c014dc0 con 0x7f50900f51a0 2024-01-21T11:47:05.177 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:05.176+0000 7f5097515700 1 -- 172.21.15.134:0/2886026716 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5090109860 con 0x7f50900f51a0 2024-01-21T11:47:05.178 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:05.176+0000 7f5086ffd700 1 -- 172.21.15.134:0/2886026716 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f508c014420 con 0x7f50900f51a0 2024-01-21T11:47:05.178 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:05.176+0000 7f5097515700 1 -- 172.21.15.134:0/2886026716 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5090045ad0 con 0x7f50900f51a0 2024-01-21T11:47:05.178 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:05.176+0000 7f5086ffd700 1 --2- 172.21.15.134:0/2886026716 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f507c02ff60 0x7f507c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:47:05.178 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:05.176+0000 7f5086ffd700 1 -- 172.21.15.134:0/2886026716 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f508c041050 con 0x7f50900f51a0 2024-01-21T11:47:05.178 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:05.177+0000 7f5095d12700 1 --2- 172.21.15.134:0/2886026716 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f507c02ff60 0x7f507c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:47:05.179 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:05.177+0000 7f5095d12700 1 --2- 172.21.15.134:0/2886026716 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f507c02ff60 0x7f507c032420 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f50800096a0 tx=0x7f5080006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:47:05.181 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:05.181+0000 7f5086ffd700 1 -- 172.21.15.134:0/2886026716 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f508c0095f0 con 0x7f50900f51a0 2024-01-21T11:47:05.387 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:05.387+0000 7f5097515700 1 -- 172.21.15.134:0/2886026716 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f509010a080 con 0x7f50900f51a0 2024-01-21T11:47:05.388 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:05.388+0000 7f5086ffd700 1 -- 172.21.15.134:0/2886026716 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f508c01cb10 con 0x7f50900f51a0 2024-01-21T11:47:05.388 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:47:05.388 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:47:05.390 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:05.389+0000 7f5097515700 1 -- 172.21.15.134:0/2886026716 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f507c02ff60 msgr2=0x7f507c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:47:05.390 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:05.389+0000 7f5097515700 1 --2- 172.21.15.134:0/2886026716 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f507c02ff60 0x7f507c032420 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f50800096a0 tx=0x7f5080006b40).stop 2024-01-21T11:47:05.390 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:05.389+0000 7f5097515700 1 -- 172.21.15.134:0/2886026716 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50900f51a0 msgr2=0x7f50901091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:47:05.390 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:05.389+0000 7f5097515700 1 --2- 172.21.15.134:0/2886026716 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50900f51a0 0x7f50901091b0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f508c013510 tx=0x7f508c00b040).stop 2024-01-21T11:47:05.390 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:05.389+0000 7f5097515700 1 -- 172.21.15.134:0/2886026716 shutdown_connections 2024-01-21T11:47:05.390 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:05.389+0000 7f5097515700 1 --2- 172.21.15.134:0/2886026716 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f507c02ff60 0x7f507c032420 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:47:05.390 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:05.389+0000 7f5097515700 1 --2- 172.21.15.134:0/2886026716 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50900f51a0 0x7f50901091b0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:47:05.390 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:05.389+0000 7f5097515700 1 -- 172.21.15.134:0/2886026716 >> 172.21.15.134:0/2886026716 conn(0x7f50900f09a0 msgr2=0x7f50900f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:47:05.390 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:05.390+0000 7f5097515700 1 -- 172.21.15.134:0/2886026716 shutdown_connections 2024-01-21T11:47:05.390 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:05.390+0000 7f5097515700 1 -- 172.21.15.134:0/2886026716 wait complete. 2024-01-21T11:47:05.391 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:47:05.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:47:03.850116+0000 mgr.smithi029.blhxxk (mgr.14164) 152 : cluster [DBG] pgmap v145: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:47:05.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:47:05.389139+0000 mon.smithi029 (mon.0) 226 : audit [DBG] from='client.? 172.21.15.134:0/2886026716' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:47:06.756 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:47:06.757 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:47:07.848 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:07.847+0000 7fdad1ba5700 1 -- 172.21.15.134:0/4275917207 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdacc060220 msgr2=0x7fdacc0fb550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:47:07.848 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:07.847+0000 7fdad1ba5700 1 --2- 172.21.15.134:0/4275917207 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdacc060220 0x7fdacc0fb550 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7fdab4009fb0 tx=0x7fdab4009b30).stop 2024-01-21T11:47:07.848 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:07.847+0000 7fdad1ba5700 1 -- 172.21.15.134:0/4275917207 shutdown_connections 2024-01-21T11:47:07.848 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:07.847+0000 7fdad1ba5700 1 --2- 172.21.15.134:0/4275917207 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdacc060220 0x7fdacc0fb550 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:47:07.848 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:07.847+0000 7fdad1ba5700 1 -- 172.21.15.134:0/4275917207 >> 172.21.15.134:0/4275917207 conn(0x7fdacc0f0b70 msgr2=0x7fdacc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:47:07.849 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:07.847+0000 7fdad1ba5700 1 -- 172.21.15.134:0/4275917207 shutdown_connections 2024-01-21T11:47:07.849 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:07.847+0000 7fdad1ba5700 1 -- 172.21.15.134:0/4275917207 wait complete. 2024-01-21T11:47:07.849 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:07.848+0000 7fdad1ba5700 1 Processor -- start 2024-01-21T11:47:07.849 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:07.848+0000 7fdad1ba5700 1 -- start start 2024-01-21T11:47:07.851 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:07.848+0000 7fdad1ba5700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdacc060220 0x7fdacc1088b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:47:07.851 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:07.848+0000 7fdad1ba5700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdacc108da0 con 0x7fdacc060220 2024-01-21T11:47:07.851 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:07.849+0000 7fdacb7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdacc060220 0x7fdacc1088b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:47:07.851 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:07.849+0000 7fdacb7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdacc060220 0x7fdacc1088b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:33812/0 (socket says 172.21.15.134:33812) 2024-01-21T11:47:07.851 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:07.849+0000 7fdacb7fe700 1 -- 172.21.15.134:0/3404372183 learned_addr learned my addr 172.21.15.134:0/3404372183 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:47:07.851 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:07.849+0000 7fdacb7fe700 1 -- 172.21.15.134:0/3404372183 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdab4005040 con 0x7fdacc060220 2024-01-21T11:47:07.851 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:07.849+0000 7fdacb7fe700 1 --2- 172.21.15.134:0/3404372183 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdacc060220 0x7fdacc1088b0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7fdab4009830 tx=0x7fdab400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:47:07.852 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:07.849+0000 7fdac3fff700 1 -- 172.21.15.134:0/3404372183 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdab4009320 con 0x7fdacc060220 2024-01-21T11:47:07.852 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:07.849+0000 7fdad1ba5700 1 -- 172.21.15.134:0/3404372183 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdacc108fa0 con 0x7fdacc060220 2024-01-21T11:47:07.852 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:07.849+0000 7fdad1ba5700 1 -- 172.21.15.134:0/3404372183 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdacc109430 con 0x7fdacc060220 2024-01-21T11:47:07.852 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:07.849+0000 7fdac3fff700 1 -- 172.21.15.134:0/3404372183 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdab40136c0 con 0x7fdacc060220 2024-01-21T11:47:07.852 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:07.849+0000 7fdac3fff700 1 -- 172.21.15.134:0/3404372183 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdab4014420 con 0x7fdacc060220 2024-01-21T11:47:07.852 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:07.850+0000 7fdac3fff700 1 -- 172.21.15.134:0/3404372183 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdab4014640 con 0x7fdacc060220 2024-01-21T11:47:07.852 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:07.850+0000 7fdac3fff700 1 --2- 172.21.15.134:0/3404372183 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fdab802ff60 0x7fdab8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:47:07.852 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:07.850+0000 7fdac3fff700 1 -- 172.21.15.134:0/3404372183 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdab4040f00 con 0x7fdacc060220 2024-01-21T11:47:07.852 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:07.850+0000 7fdad1ba5700 1 -- 172.21.15.134:0/3404372183 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdab0004fa0 con 0x7fdacc060220 2024-01-21T11:47:07.852 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:07.850+0000 7fdacaffd700 1 --2- 172.21.15.134:0/3404372183 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fdab802ff60 0x7fdab8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:47:07.852 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:07.851+0000 7fdacaffd700 1 --2- 172.21.15.134:0/3404372183 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fdab802ff60 0x7fdab8032420 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7fdabc0096a0 tx=0x7fdabc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:47:07.855 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:07.855+0000 7fdac3fff700 1 -- 172.21.15.134:0/3404372183 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdab4012d60 con 0x7fdacc060220 2024-01-21T11:47:07.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:07 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:47:05.850492+0000 mgr.smithi029.blhxxk (mgr.14164) 153 : cluster [DBG] pgmap v146: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:47:08.061 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:08.060+0000 7fdad1ba5700 1 -- 172.21.15.134:0/3404372183 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdab0005670 con 0x7fdacc060220 2024-01-21T11:47:08.061 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:08.061+0000 7fdac3fff700 1 -- 172.21.15.134:0/3404372183 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdab4012980 con 0x7fdacc060220 2024-01-21T11:47:08.061 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:47:08.061 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:47:08.063 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:08.062+0000 7fdad1ba5700 1 -- 172.21.15.134:0/3404372183 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fdab802ff60 msgr2=0x7fdab8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:47:08.063 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:08.062+0000 7fdad1ba5700 1 --2- 172.21.15.134:0/3404372183 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fdab802ff60 0x7fdab8032420 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7fdabc0096a0 tx=0x7fdabc006b40).stop 2024-01-21T11:47:08.063 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:08.062+0000 7fdad1ba5700 1 -- 172.21.15.134:0/3404372183 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdacc060220 msgr2=0x7fdacc1088b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:47:08.063 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:08.062+0000 7fdad1ba5700 1 --2- 172.21.15.134:0/3404372183 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdacc060220 0x7fdacc1088b0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7fdab4009830 tx=0x7fdab400b040).stop 2024-01-21T11:47:08.063 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:08.062+0000 7fdad1ba5700 1 -- 172.21.15.134:0/3404372183 shutdown_connections 2024-01-21T11:47:08.063 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:08.062+0000 7fdad1ba5700 1 --2- 172.21.15.134:0/3404372183 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fdab802ff60 0x7fdab8032420 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:47:08.063 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:08.062+0000 7fdad1ba5700 1 --2- 172.21.15.134:0/3404372183 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdacc060220 0x7fdacc1088b0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:47:08.063 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:08.062+0000 7fdad1ba5700 1 -- 172.21.15.134:0/3404372183 >> 172.21.15.134:0/3404372183 conn(0x7fdacc0f0b70 msgr2=0x7fdacc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:47:08.063 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:08.062+0000 7fdad1ba5700 1 -- 172.21.15.134:0/3404372183 shutdown_connections 2024-01-21T11:47:08.063 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:08.062+0000 7fdad1ba5700 1 -- 172.21.15.134:0/3404372183 wait complete. 2024-01-21T11:47:08.064 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:47:08.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:08 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:47:08.061852+0000 mon.smithi029 (mon.0) 227 : audit [DBG] from='client.? 172.21.15.134:0/3404372183' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:47:09.493 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:47:09.493 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:47:09.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:09 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:47:07.850740+0000 mgr.smithi029.blhxxk (mgr.14164) 154 : cluster [DBG] pgmap v147: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:47:10.603 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:10.603+0000 7fe0cd163700 1 -- 172.21.15.134:0/2104259921 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0c80f5410 msgr2=0x7fe0c80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:47:10.604 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:10.603+0000 7fe0cd163700 1 --2- 172.21.15.134:0/2104259921 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0c80f5410 0x7fe0c80f5830 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7fe0bc009fb0 tx=0x7fe0bc009b30).stop 2024-01-21T11:47:10.604 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:10.603+0000 7fe0cd163700 1 -- 172.21.15.134:0/2104259921 shutdown_connections 2024-01-21T11:47:10.604 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:10.603+0000 7fe0cd163700 1 --2- 172.21.15.134:0/2104259921 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0c80f5410 0x7fe0c80f5830 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:47:10.604 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:10.603+0000 7fe0cd163700 1 -- 172.21.15.134:0/2104259921 >> 172.21.15.134:0/2104259921 conn(0x7fe0c80f0b70 msgr2=0x7fe0c80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:47:10.604 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:10.603+0000 7fe0cd163700 1 -- 172.21.15.134:0/2104259921 shutdown_connections 2024-01-21T11:47:10.604 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:10.603+0000 7fe0cd163700 1 -- 172.21.15.134:0/2104259921 wait complete. 2024-01-21T11:47:10.605 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:10.604+0000 7fe0cd163700 1 Processor -- start 2024-01-21T11:47:10.605 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:10.604+0000 7fe0cd163700 1 -- start start 2024-01-21T11:47:10.606 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:10.604+0000 7fe0cd163700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0c80f5410 0x7fe0c81055d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:47:10.606 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:10.604+0000 7fe0cd163700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0c8105ac0 con 0x7fe0c80f5410 2024-01-21T11:47:10.606 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:10.604+0000 7fe0c6d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0c80f5410 0x7fe0c81055d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:47:10.606 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:10.604+0000 7fe0c6d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0c80f5410 0x7fe0c81055d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:33836/0 (socket says 172.21.15.134:33836) 2024-01-21T11:47:10.606 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:10.604+0000 7fe0c6d9d700 1 -- 172.21.15.134:0/644380197 learned_addr learned my addr 172.21.15.134:0/644380197 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:47:10.606 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:10.605+0000 7fe0c6d9d700 1 -- 172.21.15.134:0/644380197 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0bc005040 con 0x7fe0c80f5410 2024-01-21T11:47:10.606 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:10.605+0000 7fe0c6d9d700 1 --2- 172.21.15.134:0/644380197 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0c80f5410 0x7fe0c81055d0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7fe0bc013510 tx=0x7fe0bc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:47:10.606 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:10.605+0000 7fe0af7fe700 1 -- 172.21.15.134:0/644380197 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe0bc009320 con 0x7fe0c80f5410 2024-01-21T11:47:10.606 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:10.605+0000 7fe0cd163700 1 -- 172.21.15.134:0/644380197 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0c8105cc0 con 0x7fe0c80f5410 2024-01-21T11:47:10.606 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:10.605+0000 7fe0af7fe700 1 -- 172.21.15.134:0/644380197 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe0bc013f00 con 0x7fe0c80f5410 2024-01-21T11:47:10.606 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:10.605+0000 7fe0af7fe700 1 -- 172.21.15.134:0/644380197 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe0bc014d90 con 0x7fe0c80f5410 2024-01-21T11:47:10.606 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:10.605+0000 7fe0cd163700 1 -- 172.21.15.134:0/644380197 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0c8106150 con 0x7fe0c80f5410 2024-01-21T11:47:10.607 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:10.606+0000 7fe0af7fe700 1 -- 172.21.15.134:0/644380197 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe0bc0095f0 con 0x7fe0c80f5410 2024-01-21T11:47:10.607 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:10.606+0000 7fe0cd163700 1 -- 172.21.15.134:0/644380197 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0c8045ad0 con 0x7fe0c80f5410 2024-01-21T11:47:10.607 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:10.606+0000 7fe0af7fe700 1 --2- 172.21.15.134:0/644380197 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fe0b002ff60 0x7fe0b0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:47:10.607 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:10.606+0000 7fe0af7fe700 1 -- 172.21.15.134:0/644380197 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe0bc040fe0 con 0x7fe0c80f5410 2024-01-21T11:47:10.607 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:10.606+0000 7fe0c659c700 1 --2- 172.21.15.134:0/644380197 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fe0b002ff60 0x7fe0b0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:47:10.607 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:10.607+0000 7fe0c659c700 1 --2- 172.21.15.134:0/644380197 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fe0b002ff60 0x7fe0b0032420 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fe0b80096a0 tx=0x7fe0b8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:47:10.610 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:10.611+0000 7fe0af7fe700 1 -- 172.21.15.134:0/644380197 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe0bc01d7f0 con 0x7fe0c80f5410 2024-01-21T11:47:10.819 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:10.819+0000 7fe0cd163700 1 -- 172.21.15.134:0/644380197 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe0c8107000 con 0x7fe0c80f5410 2024-01-21T11:47:10.820 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:10.820+0000 7fe0af7fe700 1 -- 172.21.15.134:0/644380197 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe0bc010230 con 0x7fe0c80f5410 2024-01-21T11:47:10.820 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:47:10.820 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:47:10.821 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:10.821+0000 7fe0cd163700 1 -- 172.21.15.134:0/644380197 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fe0b002ff60 msgr2=0x7fe0b0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:47:10.822 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:10.821+0000 7fe0cd163700 1 --2- 172.21.15.134:0/644380197 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fe0b002ff60 0x7fe0b0032420 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fe0b80096a0 tx=0x7fe0b8006b40).stop 2024-01-21T11:47:10.822 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:10.821+0000 7fe0cd163700 1 -- 172.21.15.134:0/644380197 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0c80f5410 msgr2=0x7fe0c81055d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:47:10.822 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:10.821+0000 7fe0cd163700 1 --2- 172.21.15.134:0/644380197 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0c80f5410 0x7fe0c81055d0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7fe0bc013510 tx=0x7fe0bc00b040).stop 2024-01-21T11:47:10.822 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:10.821+0000 7fe0cd163700 1 -- 172.21.15.134:0/644380197 shutdown_connections 2024-01-21T11:47:10.822 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:10.821+0000 7fe0cd163700 1 --2- 172.21.15.134:0/644380197 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fe0b002ff60 0x7fe0b0032420 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:47:10.822 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:10.821+0000 7fe0cd163700 1 --2- 172.21.15.134:0/644380197 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0c80f5410 0x7fe0c81055d0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:47:10.822 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:10.821+0000 7fe0cd163700 1 -- 172.21.15.134:0/644380197 >> 172.21.15.134:0/644380197 conn(0x7fe0c80f0b70 msgr2=0x7fe0c80f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:47:10.822 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:10.821+0000 7fe0cd163700 1 -- 172.21.15.134:0/644380197 shutdown_connections 2024-01-21T11:47:10.822 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:10.821+0000 7fe0cd163700 1 -- 172.21.15.134:0/644380197 wait complete. 2024-01-21T11:47:10.822 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:47:11.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:11 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:47:09.851050+0000 mgr.smithi029.blhxxk (mgr.14164) 155 : cluster [DBG] pgmap v148: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:47:11.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:11 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:47:10.820860+0000 mon.smithi029 (mon.0) 228 : audit [DBG] from='client.? 172.21.15.134:0/644380197' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:47:12.205 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:47:12.205 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:47:13.305 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:13.303+0000 7f0e249a0700 1 -- 172.21.15.134:0/978250545 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0e200f6510 msgr2=0x7f0e200f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:47:13.305 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:13.303+0000 7f0e249a0700 1 --2- 172.21.15.134:0/978250545 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0e200f6510 0x7f0e200f6930 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7f0e10009fb0 tx=0x7f0e10009b30).stop 2024-01-21T11:47:13.305 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:13.303+0000 7f0e249a0700 1 -- 172.21.15.134:0/978250545 shutdown_connections 2024-01-21T11:47:13.305 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:13.303+0000 7f0e249a0700 1 --2- 172.21.15.134:0/978250545 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0e200f6510 0x7f0e200f6930 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:47:13.305 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:13.303+0000 7f0e249a0700 1 -- 172.21.15.134:0/978250545 >> 172.21.15.134:0/978250545 conn(0x7f0e200f1c70 msgr2=0x7f0e200f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:47:13.305 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:13.303+0000 7f0e249a0700 1 -- 172.21.15.134:0/978250545 shutdown_connections 2024-01-21T11:47:13.305 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:13.303+0000 7f0e249a0700 1 -- 172.21.15.134:0/978250545 wait complete. 2024-01-21T11:47:13.305 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:13.304+0000 7f0e249a0700 1 Processor -- start 2024-01-21T11:47:13.305 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:13.304+0000 7f0e249a0700 1 -- start start 2024-01-21T11:47:13.305 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:13.304+0000 7f0e249a0700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0e200f6510 0x7f0e20106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:47:13.305 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:13.304+0000 7f0e249a0700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e20106c20 con 0x7f0e200f6510 2024-01-21T11:47:13.305 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:13.304+0000 7f0e1e59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0e200f6510 0x7f0e20106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:47:13.305 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:13.304+0000 7f0e1e59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0e200f6510 0x7f0e20106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:33858/0 (socket says 172.21.15.134:33858) 2024-01-21T11:47:13.306 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:13.304+0000 7f0e1e59c700 1 -- 172.21.15.134:0/2094603772 learned_addr learned my addr 172.21.15.134:0/2094603772 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:47:13.306 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:13.304+0000 7f0e1e59c700 1 -- 172.21.15.134:0/2094603772 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e10005040 con 0x7f0e200f6510 2024-01-21T11:47:13.306 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:13.304+0000 7f0e1e59c700 1 --2- 172.21.15.134:0/2094603772 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0e200f6510 0x7f0e20106730 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f0e10013510 tx=0x7f0e1000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:47:13.306 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:13.305+0000 7f0e16ffd700 1 -- 172.21.15.134:0/2094603772 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0e10009320 con 0x7f0e200f6510 2024-01-21T11:47:13.306 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:13.305+0000 7f0e249a0700 1 -- 172.21.15.134:0/2094603772 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e20106e20 con 0x7f0e200f6510 2024-01-21T11:47:13.306 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:13.305+0000 7f0e16ffd700 1 -- 172.21.15.134:0/2094603772 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0e10013f00 con 0x7f0e200f6510 2024-01-21T11:47:13.306 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:13.305+0000 7f0e16ffd700 1 -- 172.21.15.134:0/2094603772 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0e10014d90 con 0x7f0e200f6510 2024-01-21T11:47:13.306 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:13.305+0000 7f0e249a0700 1 -- 172.21.15.134:0/2094603772 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e201072b0 con 0x7f0e200f6510 2024-01-21T11:47:13.307 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:13.305+0000 7f0e16ffd700 1 -- 172.21.15.134:0/2094603772 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0e100095f0 con 0x7f0e200f6510 2024-01-21T11:47:13.307 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:13.305+0000 7f0e249a0700 1 -- 172.21.15.134:0/2094603772 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e20045ad0 con 0x7f0e200f6510 2024-01-21T11:47:13.307 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:13.306+0000 7f0e16ffd700 1 --2- 172.21.15.134:0/2094603772 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f0e0c02ff60 0x7f0e0c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:47:13.307 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:13.306+0000 7f0e16ffd700 1 -- 172.21.15.134:0/2094603772 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0e10041000 con 0x7f0e200f6510 2024-01-21T11:47:13.307 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:13.306+0000 7f0e1dd9b700 1 --2- 172.21.15.134:0/2094603772 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f0e0c02ff60 0x7f0e0c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:47:13.307 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:13.306+0000 7f0e1dd9b700 1 --2- 172.21.15.134:0/2094603772 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f0e0c02ff60 0x7f0e0c032420 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f0e080096a0 tx=0x7f0e08006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:47:13.310 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:13.310+0000 7f0e16ffd700 1 -- 172.21.15.134:0/2094603772 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0e1001d7f0 con 0x7f0e200f6510 2024-01-21T11:47:13.518 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:13.518+0000 7f0e249a0700 1 -- 172.21.15.134:0/2094603772 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0e200fbf60 con 0x7f0e200f6510 2024-01-21T11:47:13.519 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:13.519+0000 7f0e16ffd700 1 -- 172.21.15.134:0/2094603772 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0e10010230 con 0x7f0e200f6510 2024-01-21T11:47:13.519 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:47:13.519 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:47:13.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:13.520+0000 7f0e249a0700 1 -- 172.21.15.134:0/2094603772 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f0e0c02ff60 msgr2=0x7f0e0c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:47:13.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:13.520+0000 7f0e249a0700 1 --2- 172.21.15.134:0/2094603772 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f0e0c02ff60 0x7f0e0c032420 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f0e080096a0 tx=0x7f0e08006b40).stop 2024-01-21T11:47:13.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:13.520+0000 7f0e249a0700 1 -- 172.21.15.134:0/2094603772 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0e200f6510 msgr2=0x7f0e20106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:47:13.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:13.520+0000 7f0e249a0700 1 --2- 172.21.15.134:0/2094603772 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0e200f6510 0x7f0e20106730 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f0e10013510 tx=0x7f0e1000b040).stop 2024-01-21T11:47:13.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:13.520+0000 7f0e249a0700 1 -- 172.21.15.134:0/2094603772 shutdown_connections 2024-01-21T11:47:13.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:13.520+0000 7f0e249a0700 1 --2- 172.21.15.134:0/2094603772 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f0e0c02ff60 0x7f0e0c032420 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:47:13.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:13.520+0000 7f0e249a0700 1 --2- 172.21.15.134:0/2094603772 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0e200f6510 0x7f0e20106730 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:47:13.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:13.520+0000 7f0e249a0700 1 -- 172.21.15.134:0/2094603772 >> 172.21.15.134:0/2094603772 conn(0x7f0e200f1c70 msgr2=0x7f0e200f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:47:13.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:13.520+0000 7f0e249a0700 1 -- 172.21.15.134:0/2094603772 shutdown_connections 2024-01-21T11:47:13.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:13.520+0000 7f0e249a0700 1 -- 172.21.15.134:0/2094603772 wait complete. 2024-01-21T11:47:13.522 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:47:13.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:13 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:47:11.851337+0000 mgr.smithi029.blhxxk (mgr.14164) 156 : cluster [DBG] pgmap v149: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:47:13.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:13 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:47:13.519945+0000 mon.smithi029 (mon.0) 229 : audit [DBG] from='client.? 172.21.15.134:0/2094603772' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:47:14.899 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:47:14.899 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:47:15.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:15 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:47:13.851575+0000 mgr.smithi029.blhxxk (mgr.14164) 157 : cluster [DBG] pgmap v150: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:47:16.013 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:16.012+0000 7f27c84c1700 1 -- 172.21.15.134:0/2924732032 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f27c00f5410 msgr2=0x7f27c00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:47:16.013 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:16.012+0000 7f27c84c1700 1 --2- 172.21.15.134:0/2924732032 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f27c00f5410 0x7f27c00f5830 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7f27b0009fb0 tx=0x7f27b0009b30).stop 2024-01-21T11:47:16.013 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:16.012+0000 7f27c84c1700 1 -- 172.21.15.134:0/2924732032 shutdown_connections 2024-01-21T11:47:16.013 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:16.012+0000 7f27c84c1700 1 --2- 172.21.15.134:0/2924732032 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f27c00f5410 0x7f27c00f5830 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:47:16.013 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:16.012+0000 7f27c84c1700 1 -- 172.21.15.134:0/2924732032 >> 172.21.15.134:0/2924732032 conn(0x7f27c00f0b70 msgr2=0x7f27c00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:47:16.014 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:16.012+0000 7f27c84c1700 1 -- 172.21.15.134:0/2924732032 shutdown_connections 2024-01-21T11:47:16.014 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:16.012+0000 7f27c84c1700 1 -- 172.21.15.134:0/2924732032 wait complete. 2024-01-21T11:47:16.014 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:16.013+0000 7f27c84c1700 1 Processor -- start 2024-01-21T11:47:16.014 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:16.013+0000 7f27c84c1700 1 -- start start 2024-01-21T11:47:16.014 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:16.013+0000 7f27c84c1700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f27c00f5410 0x7f27c0106710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:47:16.014 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:16.013+0000 7f27c84c1700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27c0106c00 con 0x7f27c00f5410 2024-01-21T11:47:16.014 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:16.013+0000 7f27c625d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f27c00f5410 0x7f27c0106710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:47:16.014 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:16.013+0000 7f27c625d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f27c00f5410 0x7f27c0106710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:53178/0 (socket says 172.21.15.134:53178) 2024-01-21T11:47:16.014 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:16.013+0000 7f27c625d700 1 -- 172.21.15.134:0/84641785 learned_addr learned my addr 172.21.15.134:0/84641785 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:47:16.015 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:16.013+0000 7f27c625d700 1 -- 172.21.15.134:0/84641785 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27b0005040 con 0x7f27c00f5410 2024-01-21T11:47:16.015 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:16.014+0000 7f27c625d700 1 --2- 172.21.15.134:0/84641785 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f27c00f5410 0x7f27c0106710 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f27b0013510 tx=0x7f27b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:47:16.016 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:16.014+0000 7f27b6ffd700 1 -- 172.21.15.134:0/84641785 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f27b0009320 con 0x7f27c00f5410 2024-01-21T11:47:16.016 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:16.014+0000 7f27c84c1700 1 -- 172.21.15.134:0/84641785 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27c0106e00 con 0x7f27c00f5410 2024-01-21T11:47:16.016 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:16.014+0000 7f27b6ffd700 1 -- 172.21.15.134:0/84641785 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f27b0013f00 con 0x7f27c00f5410 2024-01-21T11:47:16.016 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:16.014+0000 7f27b6ffd700 1 -- 172.21.15.134:0/84641785 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f27b0014d90 con 0x7f27c00f5410 2024-01-21T11:47:16.016 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:16.014+0000 7f27c84c1700 1 -- 172.21.15.134:0/84641785 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27c0107290 con 0x7f27c00f5410 2024-01-21T11:47:16.016 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:16.015+0000 7f27c84c1700 1 -- 172.21.15.134:0/84641785 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27c0045ad0 con 0x7f27c00f5410 2024-01-21T11:47:16.016 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:16.015+0000 7f27b6ffd700 1 -- 172.21.15.134:0/84641785 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f27b00095f0 con 0x7f27c00f5410 2024-01-21T11:47:16.016 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:16.015+0000 7f27b6ffd700 1 --2- 172.21.15.134:0/84641785 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f27ac02ff60 0x7f27ac032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:47:16.016 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:16.015+0000 7f27b6ffd700 1 -- 172.21.15.134:0/84641785 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f27b00410c0 con 0x7f27c00f5410 2024-01-21T11:47:16.016 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:16.015+0000 7f27c5a5c700 1 --2- 172.21.15.134:0/84641785 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f27ac02ff60 0x7f27ac032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:47:16.016 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:16.015+0000 7f27c5a5c700 1 --2- 172.21.15.134:0/84641785 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f27ac02ff60 0x7f27ac032420 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f27bc0096a0 tx=0x7f27bc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:47:16.019 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:16.019+0000 7f27b6ffd700 1 -- 172.21.15.134:0/84641785 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f27b001d7f0 con 0x7f27c00f5410 2024-01-21T11:47:16.227 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:16.227+0000 7f27c84c1700 1 -- 172.21.15.134:0/84641785 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f27c00fae60 con 0x7f27c00f5410 2024-01-21T11:47:16.228 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:16.227+0000 7f27b6ffd700 1 -- 172.21.15.134:0/84641785 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f27b0010230 con 0x7f27c00f5410 2024-01-21T11:47:16.228 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:47:16.228 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:47:16.229 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:16.229+0000 7f27c84c1700 1 -- 172.21.15.134:0/84641785 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f27ac02ff60 msgr2=0x7f27ac032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:47:16.229 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:16.229+0000 7f27c84c1700 1 --2- 172.21.15.134:0/84641785 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f27ac02ff60 0x7f27ac032420 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f27bc0096a0 tx=0x7f27bc006b40).stop 2024-01-21T11:47:16.229 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:16.229+0000 7f27c84c1700 1 -- 172.21.15.134:0/84641785 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f27c00f5410 msgr2=0x7f27c0106710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:47:16.229 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:16.229+0000 7f27c84c1700 1 --2- 172.21.15.134:0/84641785 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f27c00f5410 0x7f27c0106710 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f27b0013510 tx=0x7f27b000b040).stop 2024-01-21T11:47:16.229 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:16.229+0000 7f27c84c1700 1 -- 172.21.15.134:0/84641785 shutdown_connections 2024-01-21T11:47:16.230 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:16.229+0000 7f27c84c1700 1 --2- 172.21.15.134:0/84641785 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f27ac02ff60 0x7f27ac032420 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:47:16.230 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:16.229+0000 7f27c84c1700 1 --2- 172.21.15.134:0/84641785 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f27c00f5410 0x7f27c0106710 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:47:16.230 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:16.229+0000 7f27c84c1700 1 -- 172.21.15.134:0/84641785 >> 172.21.15.134:0/84641785 conn(0x7f27c00f0b70 msgr2=0x7f27c005fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:47:16.230 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:16.229+0000 7f27c84c1700 1 -- 172.21.15.134:0/84641785 shutdown_connections 2024-01-21T11:47:16.230 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:16.229+0000 7f27c84c1700 1 -- 172.21.15.134:0/84641785 wait complete. 2024-01-21T11:47:16.230 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:47:16.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:16 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:47:16.228594+0000 mon.smithi029 (mon.0) 230 : audit [DBG] from='client.? 172.21.15.134:0/84641785' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:47:17.617 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:47:17.617 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:47:17.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:17 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:47:15.851933+0000 mgr.smithi029.blhxxk (mgr.14164) 158 : cluster [DBG] pgmap v151: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:47:18.725 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:18.723+0000 7f459c76b700 1 -- 172.21.15.134:0/1059180926 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45940f6510 msgr2=0x7f45940f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:47:18.725 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:18.723+0000 7f459c76b700 1 --2- 172.21.15.134:0/1059180926 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45940f6510 0x7f45940f6930 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7f4584009fb0 tx=0x7f4584009b30).stop 2024-01-21T11:47:18.725 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:18.723+0000 7f459c76b700 1 -- 172.21.15.134:0/1059180926 shutdown_connections 2024-01-21T11:47:18.725 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:18.723+0000 7f459c76b700 1 --2- 172.21.15.134:0/1059180926 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45940f6510 0x7f45940f6930 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:47:18.725 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:18.723+0000 7f459c76b700 1 -- 172.21.15.134:0/1059180926 >> 172.21.15.134:0/1059180926 conn(0x7f45940f1c70 msgr2=0x7f45940f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:47:18.725 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:18.723+0000 7f459c76b700 1 -- 172.21.15.134:0/1059180926 shutdown_connections 2024-01-21T11:47:18.725 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:18.723+0000 7f459c76b700 1 -- 172.21.15.134:0/1059180926 wait complete. 2024-01-21T11:47:18.725 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:18.724+0000 7f459c76b700 1 Processor -- start 2024-01-21T11:47:18.726 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:18.724+0000 7f459c76b700 1 -- start start 2024-01-21T11:47:18.726 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:18.724+0000 7f459c76b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45940f6510 0x7f4594106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:47:18.726 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:18.724+0000 7f459c76b700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4594106c20 con 0x7f45940f6510 2024-01-21T11:47:18.726 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:18.724+0000 7f459a507700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45940f6510 0x7f4594106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:47:18.726 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:18.724+0000 7f459a507700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45940f6510 0x7f4594106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:53192/0 (socket says 172.21.15.134:53192) 2024-01-21T11:47:18.726 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:18.724+0000 7f459a507700 1 -- 172.21.15.134:0/1896456422 learned_addr learned my addr 172.21.15.134:0/1896456422 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:47:18.726 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:18.724+0000 7f459a507700 1 -- 172.21.15.134:0/1896456422 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4584005040 con 0x7f45940f6510 2024-01-21T11:47:18.726 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:18.725+0000 7f459a507700 1 --2- 172.21.15.134:0/1896456422 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45940f6510 0x7f4594106730 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f4584013510 tx=0x7f458400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:47:18.726 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:18.725+0000 7f458affd700 1 -- 172.21.15.134:0/1896456422 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4584009320 con 0x7f45940f6510 2024-01-21T11:47:18.726 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:18.725+0000 7f459c76b700 1 -- 172.21.15.134:0/1896456422 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4594106e20 con 0x7f45940f6510 2024-01-21T11:47:18.726 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:18.725+0000 7f458affd700 1 -- 172.21.15.134:0/1896456422 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4584013f00 con 0x7f45940f6510 2024-01-21T11:47:18.726 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:18.725+0000 7f458affd700 1 -- 172.21.15.134:0/1896456422 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4584014d90 con 0x7f45940f6510 2024-01-21T11:47:18.726 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:18.725+0000 7f459c76b700 1 -- 172.21.15.134:0/1896456422 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45941072b0 con 0x7f45940f6510 2024-01-21T11:47:18.727 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:18.726+0000 7f458affd700 1 -- 172.21.15.134:0/1896456422 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f45840095f0 con 0x7f45940f6510 2024-01-21T11:47:18.727 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:18.726+0000 7f459c76b700 1 -- 172.21.15.134:0/1896456422 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4594045ad0 con 0x7f45940f6510 2024-01-21T11:47:18.727 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:18.726+0000 7f458affd700 1 --2- 172.21.15.134:0/1896456422 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f458002ff60 0x7f4580032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:47:18.727 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:18.726+0000 7f458affd700 1 -- 172.21.15.134:0/1896456422 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f45840410c0 con 0x7f45940f6510 2024-01-21T11:47:18.727 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:18.726+0000 7f4599d06700 1 --2- 172.21.15.134:0/1896456422 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f458002ff60 0x7f4580032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:47:18.727 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:18.726+0000 7f4599d06700 1 --2- 172.21.15.134:0/1896456422 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f458002ff60 0x7f4580032420 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f45900096a0 tx=0x7f4590006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:47:18.731 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:18.731+0000 7f458affd700 1 -- 172.21.15.134:0/1896456422 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4584014420 con 0x7f45940f6510 2024-01-21T11:47:18.939 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:18.938+0000 7f459c76b700 1 -- 172.21.15.134:0/1896456422 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f45940fbf60 con 0x7f45940f6510 2024-01-21T11:47:18.939 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:18.939+0000 7f458affd700 1 -- 172.21.15.134:0/1896456422 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4584010230 con 0x7f45940f6510 2024-01-21T11:47:18.939 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:47:18.939 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:47:18.941 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:18.941+0000 7f459c76b700 1 -- 172.21.15.134:0/1896456422 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f458002ff60 msgr2=0x7f4580032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:47:18.941 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:18.941+0000 7f459c76b700 1 --2- 172.21.15.134:0/1896456422 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f458002ff60 0x7f4580032420 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f45900096a0 tx=0x7f4590006b40).stop 2024-01-21T11:47:18.941 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:18.941+0000 7f459c76b700 1 -- 172.21.15.134:0/1896456422 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45940f6510 msgr2=0x7f4594106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:47:18.941 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:18.941+0000 7f459c76b700 1 --2- 172.21.15.134:0/1896456422 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45940f6510 0x7f4594106730 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f4584013510 tx=0x7f458400b040).stop 2024-01-21T11:47:18.941 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:18.941+0000 7f459c76b700 1 -- 172.21.15.134:0/1896456422 shutdown_connections 2024-01-21T11:47:18.941 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:18.941+0000 7f459c76b700 1 --2- 172.21.15.134:0/1896456422 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f458002ff60 0x7f4580032420 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:47:18.941 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:18.941+0000 7f459c76b700 1 --2- 172.21.15.134:0/1896456422 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45940f6510 0x7f4594106730 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:47:18.941 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:18.941+0000 7f459c76b700 1 -- 172.21.15.134:0/1896456422 >> 172.21.15.134:0/1896456422 conn(0x7f45940f1c70 msgr2=0x7f45940f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:47:18.941 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:18.941+0000 7f459c76b700 1 -- 172.21.15.134:0/1896456422 shutdown_connections 2024-01-21T11:47:18.942 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:18.941+0000 7f459c76b700 1 -- 172.21.15.134:0/1896456422 wait complete. 2024-01-21T11:47:18.942 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:47:19.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:19 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:47:17.852295+0000 mgr.smithi029.blhxxk (mgr.14164) 159 : cluster [DBG] pgmap v152: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:47:19.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:19 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:47:18.940327+0000 mon.smithi029 (mon.0) 231 : audit [DBG] from='client.? 172.21.15.134:0/1896456422' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:47:20.323 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:47:20.323 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:47:21.429 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:21.428+0000 7fc560852700 1 -- 172.21.15.134:0/460504807 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc55c0f46a0 msgr2=0x7fc55c0f4ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:47:21.429 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:21.428+0000 7fc560852700 1 --2- 172.21.15.134:0/460504807 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc55c0f46a0 0x7fc55c0f4ac0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7fc550009fb0 tx=0x7fc550009b30).stop 2024-01-21T11:47:21.429 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:21.428+0000 7fc560852700 1 -- 172.21.15.134:0/460504807 shutdown_connections 2024-01-21T11:47:21.429 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:21.428+0000 7fc560852700 1 --2- 172.21.15.134:0/460504807 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc55c0f46a0 0x7fc55c0f4ac0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:47:21.430 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:21.428+0000 7fc560852700 1 -- 172.21.15.134:0/460504807 >> 172.21.15.134:0/460504807 conn(0x7fc55c0f0120 msgr2=0x7fc55c0f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:47:21.430 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:21.428+0000 7fc560852700 1 -- 172.21.15.134:0/460504807 shutdown_connections 2024-01-21T11:47:21.430 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:21.428+0000 7fc560852700 1 -- 172.21.15.134:0/460504807 wait complete. 2024-01-21T11:47:21.430 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:21.429+0000 7fc560852700 1 Processor -- start 2024-01-21T11:47:21.430 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:21.429+0000 7fc560852700 1 -- start start 2024-01-21T11:47:21.430 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:21.429+0000 7fc560852700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc55c0f46a0 0x7fc55c1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:47:21.430 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:21.429+0000 7fc560852700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc55c106bd0 con 0x7fc55c0f46a0 2024-01-21T11:47:21.432 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:21.429+0000 7fc55b7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc55c0f46a0 0x7fc55c1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:47:21.432 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:21.429+0000 7fc55b7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc55c0f46a0 0x7fc55c1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:53216/0 (socket says 172.21.15.134:53216) 2024-01-21T11:47:21.432 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:21.429+0000 7fc55b7fe700 1 -- 172.21.15.134:0/3241719706 learned_addr learned my addr 172.21.15.134:0/3241719706 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:47:21.432 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:21.430+0000 7fc55b7fe700 1 -- 172.21.15.134:0/3241719706 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc550005040 con 0x7fc55c0f46a0 2024-01-21T11:47:21.432 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:21.430+0000 7fc55b7fe700 1 --2- 172.21.15.134:0/3241719706 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc55c0f46a0 0x7fc55c1066e0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7fc550009830 tx=0x7fc55000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:47:21.432 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:21.430+0000 7fc543fff700 1 -- 172.21.15.134:0/3241719706 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc550009320 con 0x7fc55c0f46a0 2024-01-21T11:47:21.432 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:21.430+0000 7fc560852700 1 -- 172.21.15.134:0/3241719706 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc55c106dd0 con 0x7fc55c0f46a0 2024-01-21T11:47:21.432 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:21.430+0000 7fc543fff700 1 -- 172.21.15.134:0/3241719706 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc5500136c0 con 0x7fc55c0f46a0 2024-01-21T11:47:21.432 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:21.430+0000 7fc543fff700 1 -- 172.21.15.134:0/3241719706 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc55001ee90 con 0x7fc55c0f46a0 2024-01-21T11:47:21.432 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:21.430+0000 7fc560852700 1 -- 172.21.15.134:0/3241719706 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc55c107260 con 0x7fc55c0f46a0 2024-01-21T11:47:21.432 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:21.431+0000 7fc543fff700 1 -- 172.21.15.134:0/3241719706 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc550014440 con 0x7fc55c0f46a0 2024-01-21T11:47:21.432 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:21.431+0000 7fc560852700 1 -- 172.21.15.134:0/3241719706 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc55c045ad0 con 0x7fc55c0f46a0 2024-01-21T11:47:21.433 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:21.431+0000 7fc543fff700 1 --2- 172.21.15.134:0/3241719706 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fc54402ff10 0x7fc5440323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:47:21.433 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:21.431+0000 7fc543fff700 1 -- 172.21.15.134:0/3241719706 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc550041020 con 0x7fc55c0f46a0 2024-01-21T11:47:21.433 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:21.431+0000 7fc55affd700 1 --2- 172.21.15.134:0/3241719706 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fc54402ff10 0x7fc5440323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:47:21.433 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:21.432+0000 7fc55affd700 1 --2- 172.21.15.134:0/3241719706 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fc54402ff10 0x7fc5440323d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7fc54c0096a0 tx=0x7fc54c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:47:21.436 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:21.436+0000 7fc543fff700 1 -- 172.21.15.134:0/3241719706 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc55001c970 con 0x7fc55c0f46a0 2024-01-21T11:47:21.641 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:21.641+0000 7fc560852700 1 -- 172.21.15.134:0/3241719706 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc55c05eb50 con 0x7fc55c0f46a0 2024-01-21T11:47:21.642 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:21.642+0000 7fc543fff700 1 -- 172.21.15.134:0/3241719706 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc55002c2d0 con 0x7fc55c0f46a0 2024-01-21T11:47:21.642 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:47:21.642 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:47:21.644 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:21.644+0000 7fc560852700 1 -- 172.21.15.134:0/3241719706 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fc54402ff10 msgr2=0x7fc5440323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:47:21.644 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:21.644+0000 7fc560852700 1 --2- 172.21.15.134:0/3241719706 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fc54402ff10 0x7fc5440323d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7fc54c0096a0 tx=0x7fc54c006b40).stop 2024-01-21T11:47:21.644 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:21.644+0000 7fc560852700 1 -- 172.21.15.134:0/3241719706 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc55c0f46a0 msgr2=0x7fc55c1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:47:21.644 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:21.644+0000 7fc560852700 1 --2- 172.21.15.134:0/3241719706 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc55c0f46a0 0x7fc55c1066e0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7fc550009830 tx=0x7fc55000b040).stop 2024-01-21T11:47:21.644 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:21.644+0000 7fc560852700 1 -- 172.21.15.134:0/3241719706 shutdown_connections 2024-01-21T11:47:21.644 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:21.644+0000 7fc560852700 1 --2- 172.21.15.134:0/3241719706 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fc54402ff10 0x7fc5440323d0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:47:21.644 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:21.644+0000 7fc560852700 1 --2- 172.21.15.134:0/3241719706 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc55c0f46a0 0x7fc55c1066e0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:47:21.644 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:21.644+0000 7fc560852700 1 -- 172.21.15.134:0/3241719706 >> 172.21.15.134:0/3241719706 conn(0x7fc55c0f0120 msgr2=0x7fc55c0f0c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:47:21.645 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:21.644+0000 7fc560852700 1 -- 172.21.15.134:0/3241719706 shutdown_connections 2024-01-21T11:47:21.645 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:21.644+0000 7fc560852700 1 -- 172.21.15.134:0/3241719706 wait complete. 2024-01-21T11:47:21.645 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:47:21.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:21 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:47:19.852649+0000 mgr.smithi029.blhxxk (mgr.14164) 160 : cluster [DBG] pgmap v153: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:47:22.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:22 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:47:21.643184+0000 mon.smithi029 (mon.0) 232 : audit [DBG] from='client.? 172.21.15.134:0/3241719706' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:47:23.027 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:47:23.028 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:47:23.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:23 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:47:21.852980+0000 mgr.smithi029.blhxxk (mgr.14164) 161 : cluster [DBG] pgmap v154: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:47:24.172 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:24.171+0000 7f1e5c323700 1 -- 172.21.15.134:0/2666322018 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e540f6510 msgr2=0x7f1e540f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:47:24.172 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:24.171+0000 7f1e5c323700 1 --2- 172.21.15.134:0/2666322018 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e540f6510 0x7f1e540f6930 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7f1e50009fb0 tx=0x7f1e50009b30).stop 2024-01-21T11:47:24.172 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:24.171+0000 7f1e5c323700 1 -- 172.21.15.134:0/2666322018 shutdown_connections 2024-01-21T11:47:24.172 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:24.171+0000 7f1e5c323700 1 --2- 172.21.15.134:0/2666322018 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e540f6510 0x7f1e540f6930 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:47:24.173 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:24.171+0000 7f1e5c323700 1 -- 172.21.15.134:0/2666322018 >> 172.21.15.134:0/2666322018 conn(0x7f1e540f1c70 msgr2=0x7f1e540f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:47:24.173 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:24.171+0000 7f1e5c323700 1 -- 172.21.15.134:0/2666322018 shutdown_connections 2024-01-21T11:47:24.173 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:24.171+0000 7f1e5c323700 1 -- 172.21.15.134:0/2666322018 wait complete. 2024-01-21T11:47:24.173 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:24.171+0000 7f1e5c323700 1 Processor -- start 2024-01-21T11:47:24.173 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:24.171+0000 7f1e5c323700 1 -- start start 2024-01-21T11:47:24.173 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:24.171+0000 7f1e5c323700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e540f6510 0x7f1e5406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:47:24.173 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:24.172+0000 7f1e5c323700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e54069e20 con 0x7f1e540f6510 2024-01-21T11:47:24.173 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:24.172+0000 7f1e5a0bf700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e540f6510 0x7f1e5406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:47:24.173 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:24.172+0000 7f1e5a0bf700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e540f6510 0x7f1e5406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:53238/0 (socket says 172.21.15.134:53238) 2024-01-21T11:47:24.173 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:24.172+0000 7f1e5a0bf700 1 -- 172.21.15.134:0/1107168837 learned_addr learned my addr 172.21.15.134:0/1107168837 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:47:24.173 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:24.172+0000 7f1e5a0bf700 1 -- 172.21.15.134:0/1107168837 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e50005040 con 0x7f1e540f6510 2024-01-21T11:47:24.173 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:24.172+0000 7f1e5a0bf700 1 --2- 172.21.15.134:0/1107168837 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e540f6510 0x7f1e5406b9a0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f1e50013510 tx=0x7f1e5000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:47:24.174 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:24.172+0000 7f1e4affd700 1 -- 172.21.15.134:0/1107168837 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1e50009320 con 0x7f1e540f6510 2024-01-21T11:47:24.175 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:24.173+0000 7f1e4affd700 1 -- 172.21.15.134:0/1107168837 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1e50013f00 con 0x7f1e540f6510 2024-01-21T11:47:24.175 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:24.173+0000 7f1e5c323700 1 -- 172.21.15.134:0/1107168837 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e54069fc0 con 0x7f1e540f6510 2024-01-21T11:47:24.175 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:24.173+0000 7f1e4affd700 1 -- 172.21.15.134:0/1107168837 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1e50014d90 con 0x7f1e540f6510 2024-01-21T11:47:24.175 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:24.173+0000 7f1e5c323700 1 -- 172.21.15.134:0/1107168837 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e5406a450 con 0x7f1e540f6510 2024-01-21T11:47:24.175 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:24.174+0000 7f1e4affd700 1 -- 172.21.15.134:0/1107168837 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1e500095f0 con 0x7f1e540f6510 2024-01-21T11:47:24.175 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:24.174+0000 7f1e5c323700 1 -- 172.21.15.134:0/1107168837 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e54059100 con 0x7f1e540f6510 2024-01-21T11:47:24.175 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:24.174+0000 7f1e4affd700 1 --2- 172.21.15.134:0/1107168837 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f1e4002ffb0 0x7f1e40032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:47:24.175 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:24.174+0000 7f1e4affd700 1 -- 172.21.15.134:0/1107168837 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1e50041050 con 0x7f1e540f6510 2024-01-21T11:47:24.175 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:24.174+0000 7f1e598be700 1 --2- 172.21.15.134:0/1107168837 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f1e4002ffb0 0x7f1e40032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:47:24.175 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:24.174+0000 7f1e598be700 1 --2- 172.21.15.134:0/1107168837 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f1e4002ffb0 0x7f1e40032470 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f1e440096a0 tx=0x7f1e44006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:47:24.178 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:24.179+0000 7f1e4affd700 1 -- 172.21.15.134:0/1107168837 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1e50014420 con 0x7f1e540f6510 2024-01-21T11:47:24.384 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:24.384+0000 7f1e5c323700 1 -- 172.21.15.134:0/1107168837 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1e540fbf60 con 0x7f1e540f6510 2024-01-21T11:47:24.385 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:24.385+0000 7f1e4affd700 1 -- 172.21.15.134:0/1107168837 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1e5001d7f0 con 0x7f1e540f6510 2024-01-21T11:47:24.385 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:47:24.386 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:47:24.387 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:24.387+0000 7f1e5c323700 1 -- 172.21.15.134:0/1107168837 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f1e4002ffb0 msgr2=0x7f1e40032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:47:24.387 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:24.387+0000 7f1e5c323700 1 --2- 172.21.15.134:0/1107168837 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f1e4002ffb0 0x7f1e40032470 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f1e440096a0 tx=0x7f1e44006b40).stop 2024-01-21T11:47:24.387 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:24.387+0000 7f1e5c323700 1 -- 172.21.15.134:0/1107168837 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e540f6510 msgr2=0x7f1e5406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:47:24.387 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:24.387+0000 7f1e5c323700 1 --2- 172.21.15.134:0/1107168837 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e540f6510 0x7f1e5406b9a0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f1e50013510 tx=0x7f1e5000b040).stop 2024-01-21T11:47:24.387 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:24.387+0000 7f1e5c323700 1 -- 172.21.15.134:0/1107168837 shutdown_connections 2024-01-21T11:47:24.387 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:24.387+0000 7f1e5c323700 1 --2- 172.21.15.134:0/1107168837 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f1e4002ffb0 0x7f1e40032470 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:47:24.387 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:24.387+0000 7f1e5c323700 1 --2- 172.21.15.134:0/1107168837 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e540f6510 0x7f1e5406b9a0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:47:24.387 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:24.387+0000 7f1e5c323700 1 -- 172.21.15.134:0/1107168837 >> 172.21.15.134:0/1107168837 conn(0x7f1e540f1c70 msgr2=0x7f1e540f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:47:24.387 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:24.387+0000 7f1e5c323700 1 -- 172.21.15.134:0/1107168837 shutdown_connections 2024-01-21T11:47:24.387 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:24.387+0000 7f1e5c323700 1 -- 172.21.15.134:0/1107168837 wait complete. 2024-01-21T11:47:24.388 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:47:24.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:24 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:47:24.386099+0000 mon.smithi029 (mon.0) 233 : audit [DBG] from='client.? 172.21.15.134:0/1107168837' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:47:25.760 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:47:25.760 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:47:25.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:25 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:47:23.853309+0000 mgr.smithi029.blhxxk (mgr.14164) 162 : cluster [DBG] pgmap v155: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:47:26.881 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:26.879+0000 7fd9fe9bf700 1 -- 172.21.15.134:0/354251768 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9f8060280 msgr2=0x7fd9f80f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:47:26.882 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:26.879+0000 7fd9fe9bf700 1 --2- 172.21.15.134:0/354251768 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9f8060280 0x7fd9f80f90f0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7fd9e8009fb0 tx=0x7fd9e8009b30).stop 2024-01-21T11:47:26.882 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:26.879+0000 7fd9fe9bf700 1 -- 172.21.15.134:0/354251768 shutdown_connections 2024-01-21T11:47:26.882 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:26.879+0000 7fd9fe9bf700 1 --2- 172.21.15.134:0/354251768 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9f8060280 0x7fd9f80f90f0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:47:26.882 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:26.879+0000 7fd9fe9bf700 1 -- 172.21.15.134:0/354251768 >> 172.21.15.134:0/354251768 conn(0x7fd9f80f0b70 msgr2=0x7fd9f80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:47:26.882 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:26.879+0000 7fd9fe9bf700 1 -- 172.21.15.134:0/354251768 shutdown_connections 2024-01-21T11:47:26.882 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:26.880+0000 7fd9fe9bf700 1 -- 172.21.15.134:0/354251768 wait complete. 2024-01-21T11:47:26.882 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:26.880+0000 7fd9fe9bf700 1 Processor -- start 2024-01-21T11:47:26.882 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:26.880+0000 7fd9fe9bf700 1 -- start start 2024-01-21T11:47:26.882 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:26.880+0000 7fd9fe9bf700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9f8060280 0x7fd9f81066c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:47:26.882 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:26.880+0000 7fd9fe9bf700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9f8106bb0 con 0x7fd9f8060280 2024-01-21T11:47:26.882 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:26.881+0000 7fd9f7fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9f8060280 0x7fd9f81066c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:47:26.882 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:26.881+0000 7fd9f7fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9f8060280 0x7fd9f81066c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:39290/0 (socket says 172.21.15.134:39290) 2024-01-21T11:47:26.882 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:26.881+0000 7fd9f7fff700 1 -- 172.21.15.134:0/702217214 learned_addr learned my addr 172.21.15.134:0/702217214 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:47:26.882 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:26.881+0000 7fd9f7fff700 1 -- 172.21.15.134:0/702217214 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9e8005040 con 0x7fd9f8060280 2024-01-21T11:47:26.882 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:26.881+0000 7fd9f7fff700 1 --2- 172.21.15.134:0/702217214 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9f8060280 0x7fd9f81066c0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7fd9e8013510 tx=0x7fd9e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:47:26.883 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:26.881+0000 7fd9f4ff9700 1 -- 172.21.15.134:0/702217214 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd9e8009320 con 0x7fd9f8060280 2024-01-21T11:47:26.883 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:26.881+0000 7fd9fe9bf700 1 -- 172.21.15.134:0/702217214 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9f8106db0 con 0x7fd9f8060280 2024-01-21T11:47:26.883 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:26.881+0000 7fd9f4ff9700 1 -- 172.21.15.134:0/702217214 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd9e8013f00 con 0x7fd9f8060280 2024-01-21T11:47:26.883 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:26.881+0000 7fd9f4ff9700 1 -- 172.21.15.134:0/702217214 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd9e8014dc0 con 0x7fd9f8060280 2024-01-21T11:47:26.883 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:26.881+0000 7fd9fe9bf700 1 -- 172.21.15.134:0/702217214 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9f8107240 con 0x7fd9f8060280 2024-01-21T11:47:26.884 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:26.882+0000 7fd9f4ff9700 1 -- 172.21.15.134:0/702217214 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd9e8014420 con 0x7fd9f8060280 2024-01-21T11:47:26.884 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:26.882+0000 7fd9fe9bf700 1 -- 172.21.15.134:0/702217214 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9f8045ad0 con 0x7fd9f8060280 2024-01-21T11:47:26.884 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:26.882+0000 7fd9f4ff9700 1 --2- 172.21.15.134:0/702217214 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fd9e002ff60 0x7fd9e0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:47:26.884 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:26.882+0000 7fd9f4ff9700 1 -- 172.21.15.134:0/702217214 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd9e8041030 con 0x7fd9f8060280 2024-01-21T11:47:26.884 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:26.883+0000 7fd9f77fe700 1 --2- 172.21.15.134:0/702217214 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fd9e002ff60 0x7fd9e0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:47:26.884 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:26.883+0000 7fd9f77fe700 1 --2- 172.21.15.134:0/702217214 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fd9e002ff60 0x7fd9e0032420 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fd9ec0096a0 tx=0x7fd9ec006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:47:26.887 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:26.888+0000 7fd9f4ff9700 1 -- 172.21.15.134:0/702217214 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd9e80095f0 con 0x7fd9f8060280 2024-01-21T11:47:27.095 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:27.094+0000 7fd9fe9bf700 1 -- 172.21.15.134:0/702217214 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd9f81080f0 con 0x7fd9f8060280 2024-01-21T11:47:27.095 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:27.095+0000 7fd9f4ff9700 1 -- 172.21.15.134:0/702217214 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd9e801cb10 con 0x7fd9f8060280 2024-01-21T11:47:27.096 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:47:27.096 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:47:27.097 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:27.097+0000 7fd9fe9bf700 1 -- 172.21.15.134:0/702217214 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fd9e002ff60 msgr2=0x7fd9e0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:47:27.097 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:27.097+0000 7fd9fe9bf700 1 --2- 172.21.15.134:0/702217214 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fd9e002ff60 0x7fd9e0032420 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fd9ec0096a0 tx=0x7fd9ec006b40).stop 2024-01-21T11:47:27.097 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:27.097+0000 7fd9fe9bf700 1 -- 172.21.15.134:0/702217214 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9f8060280 msgr2=0x7fd9f81066c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:47:27.097 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:27.097+0000 7fd9fe9bf700 1 --2- 172.21.15.134:0/702217214 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9f8060280 0x7fd9f81066c0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7fd9e8013510 tx=0x7fd9e800b040).stop 2024-01-21T11:47:27.097 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:27.097+0000 7fd9fe9bf700 1 -- 172.21.15.134:0/702217214 shutdown_connections 2024-01-21T11:47:27.097 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:27.097+0000 7fd9fe9bf700 1 --2- 172.21.15.134:0/702217214 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fd9e002ff60 0x7fd9e0032420 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:47:27.097 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:27.097+0000 7fd9fe9bf700 1 --2- 172.21.15.134:0/702217214 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9f8060280 0x7fd9f81066c0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:47:27.098 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:27.097+0000 7fd9fe9bf700 1 -- 172.21.15.134:0/702217214 >> 172.21.15.134:0/702217214 conn(0x7fd9f80f0b70 msgr2=0x7fd9f80f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:47:27.098 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:27.097+0000 7fd9fe9bf700 1 -- 172.21.15.134:0/702217214 shutdown_connections 2024-01-21T11:47:27.098 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:27.097+0000 7fd9fe9bf700 1 -- 172.21.15.134:0/702217214 wait complete. 2024-01-21T11:47:27.098 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:47:27.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:27 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:47:25.853683+0000 mgr.smithi029.blhxxk (mgr.14164) 163 : cluster [DBG] pgmap v156: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:47:27.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:27 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:47:27.096346+0000 mon.smithi029 (mon.0) 234 : audit [DBG] from='client.? 172.21.15.134:0/702217214' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:47:28.496 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:47:28.497 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:47:29.600 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:29.599+0000 7ff2ec89e700 1 -- 172.21.15.134:0/1031333068 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2e80f5410 msgr2=0x7ff2e80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:47:29.600 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:29.599+0000 7ff2ec89e700 1 --2- 172.21.15.134:0/1031333068 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2e80f5410 0x7ff2e80f5830 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7ff2dc009fb0 tx=0x7ff2dc009b30).stop 2024-01-21T11:47:29.600 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:29.600+0000 7ff2ec89e700 1 -- 172.21.15.134:0/1031333068 shutdown_connections 2024-01-21T11:47:29.600 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:29.600+0000 7ff2ec89e700 1 --2- 172.21.15.134:0/1031333068 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2e80f5410 0x7ff2e80f5830 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:47:29.600 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:29.600+0000 7ff2ec89e700 1 -- 172.21.15.134:0/1031333068 >> 172.21.15.134:0/1031333068 conn(0x7ff2e80f0b70 msgr2=0x7ff2e80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:47:29.600 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:29.600+0000 7ff2ec89e700 1 -- 172.21.15.134:0/1031333068 shutdown_connections 2024-01-21T11:47:29.601 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:29.600+0000 7ff2ec89e700 1 -- 172.21.15.134:0/1031333068 wait complete. 2024-01-21T11:47:29.601 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:29.600+0000 7ff2ec89e700 1 Processor -- start 2024-01-21T11:47:29.601 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:29.600+0000 7ff2ec89e700 1 -- start start 2024-01-21T11:47:29.601 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:29.600+0000 7ff2ec89e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2e80f5410 0x7ff2e8109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:47:29.601 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:29.600+0000 7ff2ec89e700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2e8060220 con 0x7ff2e80f5410 2024-01-21T11:47:29.601 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:29.601+0000 7ff2e659c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2e80f5410 0x7ff2e8109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:47:29.601 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:29.601+0000 7ff2e659c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2e80f5410 0x7ff2e8109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:39306/0 (socket says 172.21.15.134:39306) 2024-01-21T11:47:29.601 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:29.601+0000 7ff2e659c700 1 -- 172.21.15.134:0/769884703 learned_addr learned my addr 172.21.15.134:0/769884703 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:47:29.601 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:29.601+0000 7ff2e659c700 1 -- 172.21.15.134:0/769884703 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2dc005040 con 0x7ff2e80f5410 2024-01-21T11:47:29.601 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:29.601+0000 7ff2e659c700 1 --2- 172.21.15.134:0/769884703 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2e80f5410 0x7ff2e8109450 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7ff2dc013510 tx=0x7ff2dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:47:29.602 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:29.601+0000 7ff2d6ffd700 1 -- 172.21.15.134:0/769884703 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2dc009320 con 0x7ff2e80f5410 2024-01-21T11:47:29.602 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:29.601+0000 7ff2ec89e700 1 -- 172.21.15.134:0/769884703 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2e8060420 con 0x7ff2e80f5410 2024-01-21T11:47:29.602 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:29.601+0000 7ff2ec89e700 1 -- 172.21.15.134:0/769884703 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2e8109b00 con 0x7ff2e80f5410 2024-01-21T11:47:29.602 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:29.602+0000 7ff2d6ffd700 1 -- 172.21.15.134:0/769884703 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff2dc013f00 con 0x7ff2e80f5410 2024-01-21T11:47:29.602 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:29.602+0000 7ff2d6ffd700 1 -- 172.21.15.134:0/769884703 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2dc014cc0 con 0x7ff2e80f5410 2024-01-21T11:47:29.603 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:29.602+0000 7ff2ec89e700 1 -- 172.21.15.134:0/769884703 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2cc004fa0 con 0x7ff2e80f5410 2024-01-21T11:47:29.603 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:29.602+0000 7ff2d6ffd700 1 -- 172.21.15.134:0/769884703 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff2dc014420 con 0x7ff2e80f5410 2024-01-21T11:47:29.603 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:29.602+0000 7ff2d6ffd700 1 --2- 172.21.15.134:0/769884703 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7ff2d002ff60 0x7ff2d0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:47:29.603 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:29.602+0000 7ff2d6ffd700 1 -- 172.21.15.134:0/769884703 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff2dc040fa0 con 0x7ff2e80f5410 2024-01-21T11:47:29.603 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:29.603+0000 7ff2e5d9b700 1 --2- 172.21.15.134:0/769884703 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7ff2d002ff60 0x7ff2d0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:47:29.603 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:29.603+0000 7ff2e5d9b700 1 --2- 172.21.15.134:0/769884703 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7ff2d002ff60 0x7ff2d0032420 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7ff2d80096a0 tx=0x7ff2d8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:47:29.606 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:29.607+0000 7ff2d6ffd700 1 -- 172.21.15.134:0/769884703 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff2dc0095f0 con 0x7ff2e80f5410 2024-01-21T11:47:29.813 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:29.813+0000 7ff2ec89e700 1 -- 172.21.15.134:0/769884703 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff2cc005670 con 0x7ff2e80f5410 2024-01-21T11:47:29.813 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:29.814+0000 7ff2d6ffd700 1 -- 172.21.15.134:0/769884703 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff2dc010e60 con 0x7ff2e80f5410 2024-01-21T11:47:29.814 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:47:29.814 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:47:29.816 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:29.815+0000 7ff2ec89e700 1 -- 172.21.15.134:0/769884703 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7ff2d002ff60 msgr2=0x7ff2d0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:47:29.816 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:29.815+0000 7ff2ec89e700 1 --2- 172.21.15.134:0/769884703 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7ff2d002ff60 0x7ff2d0032420 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7ff2d80096a0 tx=0x7ff2d8006b40).stop 2024-01-21T11:47:29.816 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:29.815+0000 7ff2ec89e700 1 -- 172.21.15.134:0/769884703 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2e80f5410 msgr2=0x7ff2e8109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:47:29.816 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:29.815+0000 7ff2ec89e700 1 --2- 172.21.15.134:0/769884703 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2e80f5410 0x7ff2e8109450 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7ff2dc013510 tx=0x7ff2dc00b040).stop 2024-01-21T11:47:29.816 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:29.815+0000 7ff2ec89e700 1 -- 172.21.15.134:0/769884703 shutdown_connections 2024-01-21T11:47:29.816 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:29.815+0000 7ff2ec89e700 1 --2- 172.21.15.134:0/769884703 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7ff2d002ff60 0x7ff2d0032420 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:47:29.816 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:29.815+0000 7ff2ec89e700 1 --2- 172.21.15.134:0/769884703 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2e80f5410 0x7ff2e8109450 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:47:29.816 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:29.815+0000 7ff2ec89e700 1 -- 172.21.15.134:0/769884703 >> 172.21.15.134:0/769884703 conn(0x7ff2e80f0b70 msgr2=0x7ff2e80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:47:29.816 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:29.815+0000 7ff2ec89e700 1 -- 172.21.15.134:0/769884703 shutdown_connections 2024-01-21T11:47:29.816 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:29.815+0000 7ff2ec89e700 1 -- 172.21.15.134:0/769884703 wait complete. 2024-01-21T11:47:29.816 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:47:29.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:29 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:47:27.854044+0000 mgr.smithi029.blhxxk (mgr.14164) 164 : cluster [DBG] pgmap v157: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:47:30.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:30 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:47:29.814701+0000 mon.smithi029 (mon.0) 235 : audit [DBG] from='client.? 172.21.15.134:0/769884703' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:47:31.198 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:47:31.198 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:47:31.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:31 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:47:29.854235+0000 mgr.smithi029.blhxxk (mgr.14164) 165 : cluster [DBG] pgmap v158: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:47:32.335 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:32.334+0000 7f49e0093700 1 -- 172.21.15.134:0/177247022 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49d80f6510 msgr2=0x7f49d80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:47:32.335 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:32.334+0000 7f49e0093700 1 --2- 172.21.15.134:0/177247022 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49d80f6510 0x7f49d80f6930 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7f49c8009fb0 tx=0x7f49c8009b30).stop 2024-01-21T11:47:32.336 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:32.334+0000 7f49e0093700 1 -- 172.21.15.134:0/177247022 shutdown_connections 2024-01-21T11:47:32.336 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:32.334+0000 7f49e0093700 1 --2- 172.21.15.134:0/177247022 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49d80f6510 0x7f49d80f6930 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:47:32.336 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:32.334+0000 7f49e0093700 1 -- 172.21.15.134:0/177247022 >> 172.21.15.134:0/177247022 conn(0x7f49d80f1c70 msgr2=0x7f49d80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:47:32.336 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:32.334+0000 7f49e0093700 1 -- 172.21.15.134:0/177247022 shutdown_connections 2024-01-21T11:47:32.336 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:32.334+0000 7f49e0093700 1 -- 172.21.15.134:0/177247022 wait complete. 2024-01-21T11:47:32.336 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:32.334+0000 7f49e0093700 1 Processor -- start 2024-01-21T11:47:32.336 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:32.334+0000 7f49e0093700 1 -- start start 2024-01-21T11:47:32.336 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:32.334+0000 7f49e0093700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49d80f6510 0x7f49d8106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:47:32.336 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:32.334+0000 7f49e0093700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49d8106c20 con 0x7f49d80f6510 2024-01-21T11:47:32.336 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:32.335+0000 7f49dde2f700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49d80f6510 0x7f49d8106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:47:32.336 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:32.335+0000 7f49dde2f700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49d80f6510 0x7f49d8106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:39322/0 (socket says 172.21.15.134:39322) 2024-01-21T11:47:32.336 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:32.335+0000 7f49dde2f700 1 -- 172.21.15.134:0/2981552408 learned_addr learned my addr 172.21.15.134:0/2981552408 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:47:32.336 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:32.335+0000 7f49dde2f700 1 -- 172.21.15.134:0/2981552408 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49c8005040 con 0x7f49d80f6510 2024-01-21T11:47:32.336 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:32.335+0000 7f49dde2f700 1 --2- 172.21.15.134:0/2981552408 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49d80f6510 0x7f49d8106730 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f49c8013510 tx=0x7f49c800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:47:32.337 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:32.335+0000 7f49ce7fc700 1 -- 172.21.15.134:0/2981552408 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f49c8009320 con 0x7f49d80f6510 2024-01-21T11:47:32.337 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:32.335+0000 7f49ce7fc700 1 -- 172.21.15.134:0/2981552408 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f49c8013f00 con 0x7f49d80f6510 2024-01-21T11:47:32.337 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:32.335+0000 7f49e0093700 1 -- 172.21.15.134:0/2981552408 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49d8106e20 con 0x7f49d80f6510 2024-01-21T11:47:32.337 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:32.335+0000 7f49ce7fc700 1 -- 172.21.15.134:0/2981552408 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f49c8014d90 con 0x7f49d80f6510 2024-01-21T11:47:32.337 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:32.335+0000 7f49e0093700 1 -- 172.21.15.134:0/2981552408 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49d81072b0 con 0x7f49d80f6510 2024-01-21T11:47:32.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:32.336+0000 7f49ce7fc700 1 -- 172.21.15.134:0/2981552408 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f49c80095f0 con 0x7f49d80f6510 2024-01-21T11:47:32.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:32.336+0000 7f49e0093700 1 -- 172.21.15.134:0/2981552408 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49d8045ad0 con 0x7f49d80f6510 2024-01-21T11:47:32.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:32.336+0000 7f49ce7fc700 1 --2- 172.21.15.134:0/2981552408 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f49c402ffb0 0x7f49c4032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:47:32.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:32.336+0000 7f49ce7fc700 1 -- 172.21.15.134:0/2981552408 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f49c80410c0 con 0x7f49d80f6510 2024-01-21T11:47:32.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:32.337+0000 7f49dd62e700 1 --2- 172.21.15.134:0/2981552408 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f49c402ffb0 0x7f49c4032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:47:32.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:32.337+0000 7f49dd62e700 1 --2- 172.21.15.134:0/2981552408 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f49c402ffb0 0x7f49c4032470 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f49d40096a0 tx=0x7f49d4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:47:32.341 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:32.341+0000 7f49ce7fc700 1 -- 172.21.15.134:0/2981552408 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f49c801d7f0 con 0x7f49d80f6510 2024-01-21T11:47:32.550 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:32.550+0000 7f49e0093700 1 -- 172.21.15.134:0/2981552408 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f49d80fbf60 con 0x7f49d80f6510 2024-01-21T11:47:32.551 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:32.551+0000 7f49ce7fc700 1 -- 172.21.15.134:0/2981552408 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f49c8010230 con 0x7f49d80f6510 2024-01-21T11:47:32.551 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:47:32.551 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:47:32.552 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:32.552+0000 7f49e0093700 1 -- 172.21.15.134:0/2981552408 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f49c402ffb0 msgr2=0x7f49c4032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:47:32.552 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:32.552+0000 7f49e0093700 1 --2- 172.21.15.134:0/2981552408 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f49c402ffb0 0x7f49c4032470 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f49d40096a0 tx=0x7f49d4006b40).stop 2024-01-21T11:47:32.553 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:32.552+0000 7f49e0093700 1 -- 172.21.15.134:0/2981552408 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49d80f6510 msgr2=0x7f49d8106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:47:32.553 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:32.552+0000 7f49e0093700 1 --2- 172.21.15.134:0/2981552408 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49d80f6510 0x7f49d8106730 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f49c8013510 tx=0x7f49c800b040).stop 2024-01-21T11:47:32.553 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:32.552+0000 7f49e0093700 1 -- 172.21.15.134:0/2981552408 shutdown_connections 2024-01-21T11:47:32.553 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:32.552+0000 7f49e0093700 1 --2- 172.21.15.134:0/2981552408 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f49c402ffb0 0x7f49c4032470 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:47:32.553 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:32.552+0000 7f49e0093700 1 --2- 172.21.15.134:0/2981552408 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49d80f6510 0x7f49d8106730 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:47:32.553 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:32.552+0000 7f49e0093700 1 -- 172.21.15.134:0/2981552408 >> 172.21.15.134:0/2981552408 conn(0x7f49d80f1c70 msgr2=0x7f49d80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:47:32.553 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:32.552+0000 7f49e0093700 1 -- 172.21.15.134:0/2981552408 shutdown_connections 2024-01-21T11:47:32.553 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:32.552+0000 7f49e0093700 1 -- 172.21.15.134:0/2981552408 wait complete. 2024-01-21T11:47:32.553 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:47:32.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:32 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:47:32.551930+0000 mon.smithi029 (mon.0) 236 : audit [DBG] from='client.? 172.21.15.134:0/2981552408' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:47:33.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:33 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:47:31.854526+0000 mgr.smithi029.blhxxk (mgr.14164) 166 : cluster [DBG] pgmap v159: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:47:33.923 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:47:33.923 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:47:35.118 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:35.118+0000 7f0e6ffff700 1 -- 172.21.15.134:0/3380672599 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0e70060140 msgr2=0x7f0e700f8fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:47:35.119 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:35.118+0000 7f0e6ffff700 1 --2- 172.21.15.134:0/3380672599 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0e70060140 0x7f0e700f8fc0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7f0e60009fb0 tx=0x7f0e60009b30).stop 2024-01-21T11:47:35.119 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:35.118+0000 7f0e6ffff700 1 -- 172.21.15.134:0/3380672599 shutdown_connections 2024-01-21T11:47:35.119 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:35.118+0000 7f0e6ffff700 1 --2- 172.21.15.134:0/3380672599 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0e70060140 0x7f0e700f8fc0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:47:35.119 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:35.118+0000 7f0e6ffff700 1 -- 172.21.15.134:0/3380672599 >> 172.21.15.134:0/3380672599 conn(0x7f0e700f09a0 msgr2=0x7f0e700f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:47:35.119 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:35.118+0000 7f0e6ffff700 1 -- 172.21.15.134:0/3380672599 shutdown_connections 2024-01-21T11:47:35.119 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:35.118+0000 7f0e6ffff700 1 -- 172.21.15.134:0/3380672599 wait complete. 2024-01-21T11:47:35.119 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:35.118+0000 7f0e6ffff700 1 Processor -- start 2024-01-21T11:47:35.119 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:35.118+0000 7f0e6ffff700 1 -- start start 2024-01-21T11:47:35.121 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:35.119+0000 7f0e6ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0e70060140 0x7f0e70106520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:47:35.121 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:35.119+0000 7f0e6ffff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e70106a10 con 0x7f0e70060140 2024-01-21T11:47:35.121 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:35.119+0000 7f0e6effd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0e70060140 0x7f0e70106520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:47:35.121 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:35.119+0000 7f0e6effd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0e70060140 0x7f0e70106520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:32830/0 (socket says 172.21.15.134:32830) 2024-01-21T11:47:35.121 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:35.119+0000 7f0e6effd700 1 -- 172.21.15.134:0/3521813828 learned_addr learned my addr 172.21.15.134:0/3521813828 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:47:35.121 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:35.119+0000 7f0e6effd700 1 -- 172.21.15.134:0/3521813828 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e60005040 con 0x7f0e70060140 2024-01-21T11:47:35.121 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:35.120+0000 7f0e6effd700 1 --2- 172.21.15.134:0/3521813828 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0e70060140 0x7f0e70106520 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7f0e60013510 tx=0x7f0e6000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:47:35.121 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:35.120+0000 7f0e577fe700 1 -- 172.21.15.134:0/3521813828 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0e60009320 con 0x7f0e70060140 2024-01-21T11:47:35.122 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:35.120+0000 7f0e577fe700 1 -- 172.21.15.134:0/3521813828 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0e60013f00 con 0x7f0e70060140 2024-01-21T11:47:35.122 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:35.120+0000 7f0e6ffff700 1 -- 172.21.15.134:0/3521813828 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e70106c10 con 0x7f0e70060140 2024-01-21T11:47:35.122 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:35.120+0000 7f0e577fe700 1 -- 172.21.15.134:0/3521813828 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0e60014d90 con 0x7f0e70060140 2024-01-21T11:47:35.122 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:35.120+0000 7f0e6ffff700 1 -- 172.21.15.134:0/3521813828 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e701070a0 con 0x7f0e70060140 2024-01-21T11:47:35.122 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:35.121+0000 7f0e577fe700 1 -- 172.21.15.134:0/3521813828 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0e60014420 con 0x7f0e70060140 2024-01-21T11:47:35.122 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:35.121+0000 7f0e577fe700 1 --2- 172.21.15.134:0/3521813828 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f0e5802ff60 0x7f0e58032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:47:35.122 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:35.121+0000 7f0e577fe700 1 -- 172.21.15.134:0/3521813828 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0e60041000 con 0x7f0e70060140 2024-01-21T11:47:35.122 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:35.121+0000 7f0e6ffff700 1 -- 172.21.15.134:0/3521813828 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e70045ad0 con 0x7f0e70060140 2024-01-21T11:47:35.122 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:35.121+0000 7f0e6e7fc700 1 --2- 172.21.15.134:0/3521813828 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f0e5802ff60 0x7f0e58032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:47:35.122 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:35.121+0000 7f0e6e7fc700 1 --2- 172.21.15.134:0/3521813828 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f0e5802ff60 0x7f0e58032420 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f0e640096a0 tx=0x7f0e64006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:47:35.125 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:35.126+0000 7f0e577fe700 1 -- 172.21.15.134:0/3521813828 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0e600095f0 con 0x7f0e70060140 2024-01-21T11:47:35.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:35.331+0000 7f0e6ffff700 1 -- 172.21.15.134:0/3521813828 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0e70107d20 con 0x7f0e70060140 2024-01-21T11:47:35.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:35.332+0000 7f0e577fe700 1 -- 172.21.15.134:0/3521813828 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0e6001db70 con 0x7f0e70060140 2024-01-21T11:47:35.332 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:47:35.332 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:47:35.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:35.333+0000 7f0e6ffff700 1 -- 172.21.15.134:0/3521813828 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f0e5802ff60 msgr2=0x7f0e58032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:47:35.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:35.333+0000 7f0e6ffff700 1 --2- 172.21.15.134:0/3521813828 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f0e5802ff60 0x7f0e58032420 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f0e640096a0 tx=0x7f0e64006b40).stop 2024-01-21T11:47:35.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:35.333+0000 7f0e6ffff700 1 -- 172.21.15.134:0/3521813828 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0e70060140 msgr2=0x7f0e70106520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:47:35.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:35.333+0000 7f0e6ffff700 1 --2- 172.21.15.134:0/3521813828 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0e70060140 0x7f0e70106520 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7f0e60013510 tx=0x7f0e6000b040).stop 2024-01-21T11:47:35.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:35.333+0000 7f0e6ffff700 1 -- 172.21.15.134:0/3521813828 shutdown_connections 2024-01-21T11:47:35.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:35.333+0000 7f0e6ffff700 1 --2- 172.21.15.134:0/3521813828 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f0e5802ff60 0x7f0e58032420 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:47:35.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:35.333+0000 7f0e6ffff700 1 --2- 172.21.15.134:0/3521813828 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0e70060140 0x7f0e70106520 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:47:35.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:35.333+0000 7f0e6ffff700 1 -- 172.21.15.134:0/3521813828 >> 172.21.15.134:0/3521813828 conn(0x7f0e700f09a0 msgr2=0x7f0e700f14d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:47:35.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:35.333+0000 7f0e6ffff700 1 -- 172.21.15.134:0/3521813828 shutdown_connections 2024-01-21T11:47:35.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:35.333+0000 7f0e6ffff700 1 -- 172.21.15.134:0/3521813828 wait complete. 2024-01-21T11:47:35.334 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:47:35.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:35 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:47:33.854852+0000 mgr.smithi029.blhxxk (mgr.14164) 167 : cluster [DBG] pgmap v160: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:47:35.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:35 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:47:35.332984+0000 mon.smithi029 (mon.0) 237 : audit [DBG] from='client.? 172.21.15.134:0/3521813828' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:47:36.743 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:47:36.743 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:47:37.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:37 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:47:35.855177+0000 mgr.smithi029.blhxxk (mgr.14164) 168 : cluster [DBG] pgmap v161: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:47:37.972 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:37.971+0000 7f4d964de700 1 -- 172.21.15.134:0/2880873395 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d900f5410 msgr2=0x7f4d900f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:47:37.972 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:37.971+0000 7f4d964de700 1 --2- 172.21.15.134:0/2880873395 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d900f5410 0x7f4d900f5830 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7f4d80009fb0 tx=0x7f4d80009b30).stop 2024-01-21T11:47:37.972 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:37.972+0000 7f4d964de700 1 -- 172.21.15.134:0/2880873395 shutdown_connections 2024-01-21T11:47:37.972 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:37.972+0000 7f4d964de700 1 --2- 172.21.15.134:0/2880873395 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d900f5410 0x7f4d900f5830 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:47:37.972 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:37.972+0000 7f4d964de700 1 -- 172.21.15.134:0/2880873395 >> 172.21.15.134:0/2880873395 conn(0x7f4d900f0b70 msgr2=0x7f4d900f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:47:37.973 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:37.972+0000 7f4d964de700 1 -- 172.21.15.134:0/2880873395 shutdown_connections 2024-01-21T11:47:37.973 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:37.972+0000 7f4d964de700 1 -- 172.21.15.134:0/2880873395 wait complete. 2024-01-21T11:47:37.973 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:37.972+0000 7f4d964de700 1 Processor -- start 2024-01-21T11:47:37.973 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:37.972+0000 7f4d964de700 1 -- start start 2024-01-21T11:47:37.973 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:37.972+0000 7f4d964de700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d900f5410 0x7f4d900fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:47:37.973 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:37.972+0000 7f4d964de700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d900fb820 con 0x7f4d900f5410 2024-01-21T11:47:37.974 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:37.973+0000 7f4d8ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d900f5410 0x7f4d900fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:47:37.974 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:37.973+0000 7f4d8ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d900f5410 0x7f4d900fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:32856/0 (socket says 172.21.15.134:32856) 2024-01-21T11:47:37.974 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:37.973+0000 7f4d8ffff700 1 -- 172.21.15.134:0/3265067826 learned_addr learned my addr 172.21.15.134:0/3265067826 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:47:37.974 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:37.973+0000 7f4d8ffff700 1 -- 172.21.15.134:0/3265067826 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d80005040 con 0x7f4d900f5410 2024-01-21T11:47:37.975 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:37.973+0000 7f4d8ffff700 1 --2- 172.21.15.134:0/3265067826 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d900f5410 0x7f4d900fd3a0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f4d80013510 tx=0x7f4d8000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:47:37.975 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:37.973+0000 7f4d8cff9700 1 -- 172.21.15.134:0/3265067826 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d80009320 con 0x7f4d900f5410 2024-01-21T11:47:37.975 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:37.973+0000 7f4d8cff9700 1 -- 172.21.15.134:0/3265067826 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4d80013f00 con 0x7f4d900f5410 2024-01-21T11:47:37.975 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:37.973+0000 7f4d964de700 1 -- 172.21.15.134:0/3265067826 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d900fb9c0 con 0x7f4d900f5410 2024-01-21T11:47:37.975 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:37.973+0000 7f4d8cff9700 1 -- 172.21.15.134:0/3265067826 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d80014d90 con 0x7f4d900f5410 2024-01-21T11:47:37.975 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:37.973+0000 7f4d964de700 1 -- 172.21.15.134:0/3265067826 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d900fbe50 con 0x7f4d900f5410 2024-01-21T11:47:37.975 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:37.974+0000 7f4d8cff9700 1 -- 172.21.15.134:0/3265067826 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4d800095f0 con 0x7f4d900f5410 2024-01-21T11:47:37.975 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:37.974+0000 7f4d964de700 1 -- 172.21.15.134:0/3265067826 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d90059100 con 0x7f4d900f5410 2024-01-21T11:47:37.975 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:37.974+0000 7f4d8cff9700 1 --2- 172.21.15.134:0/3265067826 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f4d7802ff60 0x7f4d78032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:47:37.975 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:37.974+0000 7f4d8cff9700 1 -- 172.21.15.134:0/3265067826 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4d80040fe0 con 0x7f4d900f5410 2024-01-21T11:47:37.975 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:37.974+0000 7f4d8f7fe700 1 --2- 172.21.15.134:0/3265067826 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f4d7802ff60 0x7f4d78032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:47:37.976 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:37.975+0000 7f4d8f7fe700 1 --2- 172.21.15.134:0/3265067826 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f4d7802ff60 0x7f4d78032420 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f4d840096a0 tx=0x7f4d84006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:47:37.979 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:37.979+0000 7f4d8cff9700 1 -- 172.21.15.134:0/3265067826 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4d800268c0 con 0x7f4d900f5410 2024-01-21T11:47:38.184 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:38.184+0000 7f4d964de700 1 -- 172.21.15.134:0/3265067826 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4d900fae60 con 0x7f4d900f5410 2024-01-21T11:47:38.184 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:38.184+0000 7f4d8cff9700 1 -- 172.21.15.134:0/3265067826 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4d80010230 con 0x7f4d900f5410 2024-01-21T11:47:38.185 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:47:38.185 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:47:38.186 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:38.186+0000 7f4d964de700 1 -- 172.21.15.134:0/3265067826 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f4d7802ff60 msgr2=0x7f4d78032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:47:38.186 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:38.186+0000 7f4d964de700 1 --2- 172.21.15.134:0/3265067826 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f4d7802ff60 0x7f4d78032420 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f4d840096a0 tx=0x7f4d84006b40).stop 2024-01-21T11:47:38.186 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:38.186+0000 7f4d964de700 1 -- 172.21.15.134:0/3265067826 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d900f5410 msgr2=0x7f4d900fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:47:38.186 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:38.186+0000 7f4d964de700 1 --2- 172.21.15.134:0/3265067826 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d900f5410 0x7f4d900fd3a0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f4d80013510 tx=0x7f4d8000b040).stop 2024-01-21T11:47:38.187 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:38.186+0000 7f4d964de700 1 -- 172.21.15.134:0/3265067826 shutdown_connections 2024-01-21T11:47:38.187 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:38.186+0000 7f4d964de700 1 --2- 172.21.15.134:0/3265067826 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f4d7802ff60 0x7f4d78032420 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:47:38.187 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:38.186+0000 7f4d964de700 1 --2- 172.21.15.134:0/3265067826 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d900f5410 0x7f4d900fd3a0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:47:38.187 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:38.186+0000 7f4d964de700 1 -- 172.21.15.134:0/3265067826 >> 172.21.15.134:0/3265067826 conn(0x7f4d900f0b70 msgr2=0x7f4d9005fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:47:38.187 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:38.186+0000 7f4d964de700 1 -- 172.21.15.134:0/3265067826 shutdown_connections 2024-01-21T11:47:38.187 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:38.186+0000 7f4d964de700 1 -- 172.21.15.134:0/3265067826 wait complete. 2024-01-21T11:47:38.187 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:47:38.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:38 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:47:38.185555+0000 mon.smithi029 (mon.0) 238 : audit [DBG] from='client.? 172.21.15.134:0/3265067826' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:47:39.573 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:47:39.573 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:47:39.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:39 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:47:37.855505+0000 mgr.smithi029.blhxxk (mgr.14164) 169 : cluster [DBG] pgmap v162: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:47:40.754 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:40.753+0000 7ff0250c0700 1 -- 172.21.15.134:0/3296389993 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0200f5410 msgr2=0x7ff0200f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:47:40.754 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:40.753+0000 7ff0250c0700 1 --2- 172.21.15.134:0/3296389993 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0200f5410 0x7ff0200f5830 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7ff010009fb0 tx=0x7ff010009b30).stop 2024-01-21T11:47:40.754 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:40.753+0000 7ff0250c0700 1 -- 172.21.15.134:0/3296389993 shutdown_connections 2024-01-21T11:47:40.754 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:40.753+0000 7ff0250c0700 1 --2- 172.21.15.134:0/3296389993 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0200f5410 0x7ff0200f5830 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:47:40.754 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:40.753+0000 7ff0250c0700 1 -- 172.21.15.134:0/3296389993 >> 172.21.15.134:0/3296389993 conn(0x7ff0200f0b70 msgr2=0x7ff0200f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:47:40.754 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:40.753+0000 7ff0250c0700 1 -- 172.21.15.134:0/3296389993 shutdown_connections 2024-01-21T11:47:40.755 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:40.753+0000 7ff0250c0700 1 -- 172.21.15.134:0/3296389993 wait complete. 2024-01-21T11:47:40.755 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:40.753+0000 7ff0250c0700 1 Processor -- start 2024-01-21T11:47:40.755 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:40.753+0000 7ff0250c0700 1 -- start start 2024-01-21T11:47:40.755 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:40.754+0000 7ff0250c0700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0200f5410 0x7ff020105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:47:40.755 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:40.754+0000 7ff0250c0700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff020105b20 con 0x7ff0200f5410 2024-01-21T11:47:40.755 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:40.754+0000 7ff01ed9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0200f5410 0x7ff020105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:47:40.755 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:40.754+0000 7ff01ed9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0200f5410 0x7ff020105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:32874/0 (socket says 172.21.15.134:32874) 2024-01-21T11:47:40.755 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:40.754+0000 7ff01ed9d700 1 -- 172.21.15.134:0/874387692 learned_addr learned my addr 172.21.15.134:0/874387692 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:47:40.757 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:40.754+0000 7ff01ed9d700 1 -- 172.21.15.134:0/874387692 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff010005040 con 0x7ff0200f5410 2024-01-21T11:47:40.757 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:40.754+0000 7ff01ed9d700 1 --2- 172.21.15.134:0/874387692 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0200f5410 0x7ff020105630 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7ff010014000 tx=0x7ff01000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:47:40.757 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:40.754+0000 7ff0077fe700 1 -- 172.21.15.134:0/874387692 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff0100095f0 con 0x7ff0200f5410 2024-01-21T11:47:40.757 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:40.754+0000 7ff0250c0700 1 -- 172.21.15.134:0/874387692 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff020105d20 con 0x7ff0200f5410 2024-01-21T11:47:40.757 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:40.754+0000 7ff0077fe700 1 -- 172.21.15.134:0/874387692 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff010007e70 con 0x7ff0200f5410 2024-01-21T11:47:40.757 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:40.754+0000 7ff0077fe700 1 -- 172.21.15.134:0/874387692 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff010014ea0 con 0x7ff0200f5410 2024-01-21T11:47:40.757 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:40.755+0000 7ff0250c0700 1 -- 172.21.15.134:0/874387692 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0201061b0 con 0x7ff0200f5410 2024-01-21T11:47:40.757 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:40.755+0000 7ff0250c0700 1 -- 172.21.15.134:0/874387692 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff020045ad0 con 0x7ff0200f5410 2024-01-21T11:47:40.757 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:40.755+0000 7ff0077fe700 1 -- 172.21.15.134:0/874387692 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff010025070 con 0x7ff0200f5410 2024-01-21T11:47:40.757 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:40.756+0000 7ff0077fe700 1 --2- 172.21.15.134:0/874387692 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7ff00802ff60 0x7ff008032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:47:40.757 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:40.756+0000 7ff0077fe700 1 -- 172.21.15.134:0/874387692 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff010040930 con 0x7ff0200f5410 2024-01-21T11:47:40.758 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:40.756+0000 7ff01e59c700 1 --2- 172.21.15.134:0/874387692 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7ff00802ff60 0x7ff008032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:47:40.758 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:40.756+0000 7ff01e59c700 1 --2- 172.21.15.134:0/874387692 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7ff00802ff60 0x7ff008032420 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7ff0140096a0 tx=0x7ff014006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:47:40.760 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:40.760+0000 7ff0077fe700 1 -- 172.21.15.134:0/874387692 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff01000f4e0 con 0x7ff0200f5410 2024-01-21T11:47:40.967 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:40.966+0000 7ff0250c0700 1 -- 172.21.15.134:0/874387692 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff0200fae60 con 0x7ff0200f5410 2024-01-21T11:47:40.967 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:40.967+0000 7ff0077fe700 1 -- 172.21.15.134:0/874387692 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff01000f300 con 0x7ff0200f5410 2024-01-21T11:47:40.968 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:47:40.968 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:47:40.969 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:40.969+0000 7ff0250c0700 1 -- 172.21.15.134:0/874387692 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7ff00802ff60 msgr2=0x7ff008032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:47:40.969 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:40.969+0000 7ff0250c0700 1 --2- 172.21.15.134:0/874387692 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7ff00802ff60 0x7ff008032420 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7ff0140096a0 tx=0x7ff014006b40).stop 2024-01-21T11:47:40.969 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:40.969+0000 7ff0250c0700 1 -- 172.21.15.134:0/874387692 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0200f5410 msgr2=0x7ff020105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:47:40.969 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:40.969+0000 7ff0250c0700 1 --2- 172.21.15.134:0/874387692 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0200f5410 0x7ff020105630 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7ff010014000 tx=0x7ff01000b040).stop 2024-01-21T11:47:40.969 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:40.969+0000 7ff0250c0700 1 -- 172.21.15.134:0/874387692 shutdown_connections 2024-01-21T11:47:40.970 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:40.969+0000 7ff0250c0700 1 --2- 172.21.15.134:0/874387692 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7ff00802ff60 0x7ff008032420 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:47:40.970 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:40.969+0000 7ff0250c0700 1 --2- 172.21.15.134:0/874387692 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0200f5410 0x7ff020105630 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:47:40.970 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:40.969+0000 7ff0250c0700 1 -- 172.21.15.134:0/874387692 >> 172.21.15.134:0/874387692 conn(0x7ff0200f0b70 msgr2=0x7ff0200f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:47:40.970 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:40.969+0000 7ff0250c0700 1 -- 172.21.15.134:0/874387692 shutdown_connections 2024-01-21T11:47:40.970 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:40.969+0000 7ff0250c0700 1 -- 172.21.15.134:0/874387692 wait complete. 2024-01-21T11:47:40.970 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:47:41.912 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:41 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:47:39.855801+0000 mgr.smithi029.blhxxk (mgr.14164) 170 : cluster [DBG] pgmap v163: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:47:41.913 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:41 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:47:40.968513+0000 mon.smithi029 (mon.0) 239 : audit [DBG] from='client.? 172.21.15.134:0/874387692' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:47:42.336 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:47:42.336 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:47:42.905 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:42 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:47:41.914945+0000 mon.smithi029 (mon.0) 240 : audit [INF] from='mgr.14164 172.21.15.29:0/3252518221' entity='mgr.smithi029.blhxxk' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi029", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-01-21T11:47:42.905 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:42 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:47:41.917408+0000 mon.smithi029 (mon.0) 241 : audit [INF] from='mgr.14164 172.21.15.29:0/3252518221' entity='mgr.smithi029.blhxxk' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi029", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-01-21T11:47:42.905 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:42 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:47:41.918182+0000 mon.smithi029 (mon.0) 242 : audit [DBG] from='mgr.14164 172.21.15.29:0/3252518221' entity='mgr.smithi029.blhxxk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-21T11:47:43.553 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:43.550+0000 7f1d3f971700 1 -- 172.21.15.134:0/3434854443 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d38060220 msgr2=0x7f1d380f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:47:43.553 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:43.550+0000 7f1d3f971700 1 --2- 172.21.15.134:0/3434854443 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d38060220 0x7f1d380f9150 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7f1d28009fb0 tx=0x7f1d28009b30).stop 2024-01-21T11:47:43.553 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:43.550+0000 7f1d3f971700 1 -- 172.21.15.134:0/3434854443 shutdown_connections 2024-01-21T11:47:43.553 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:43.550+0000 7f1d3f971700 1 --2- 172.21.15.134:0/3434854443 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d38060220 0x7f1d380f9150 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:47:43.553 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:43.550+0000 7f1d3f971700 1 -- 172.21.15.134:0/3434854443 >> 172.21.15.134:0/3434854443 conn(0x7f1d380f0b70 msgr2=0x7f1d380f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:47:43.553 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:43.550+0000 7f1d3f971700 1 -- 172.21.15.134:0/3434854443 shutdown_connections 2024-01-21T11:47:43.553 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:43.550+0000 7f1d3f971700 1 -- 172.21.15.134:0/3434854443 wait complete. 2024-01-21T11:47:43.553 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:43.551+0000 7f1d3f971700 1 Processor -- start 2024-01-21T11:47:43.553 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:43.551+0000 7f1d3f971700 1 -- start start 2024-01-21T11:47:43.553 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:43.551+0000 7f1d3f971700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d38060220 0x7f1d380f6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:47:43.553 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:43.551+0000 7f1d3f971700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d380f5050 con 0x7f1d38060220 2024-01-21T11:47:43.553 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:43.551+0000 7f1d3d70d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d38060220 0x7f1d380f6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:47:43.553 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:43.551+0000 7f1d3d70d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d38060220 0x7f1d380f6bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:32882/0 (socket says 172.21.15.134:32882) 2024-01-21T11:47:43.554 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:43.551+0000 7f1d3d70d700 1 -- 172.21.15.134:0/391599872 learned_addr learned my addr 172.21.15.134:0/391599872 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:47:43.554 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:43.551+0000 7f1d3d70d700 1 -- 172.21.15.134:0/391599872 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d28005040 con 0x7f1d38060220 2024-01-21T11:47:43.554 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:43.552+0000 7f1d3d70d700 1 --2- 172.21.15.134:0/391599872 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d38060220 0x7f1d380f6bd0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7f1d28009830 tx=0x7f1d2800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:47:43.554 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:43.552+0000 7f1d2dffb700 1 -- 172.21.15.134:0/391599872 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1d28009320 con 0x7f1d38060220 2024-01-21T11:47:43.554 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:43.552+0000 7f1d3f971700 1 -- 172.21.15.134:0/391599872 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d380f51f0 con 0x7f1d38060220 2024-01-21T11:47:43.554 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:43.552+0000 7f1d2dffb700 1 -- 172.21.15.134:0/391599872 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1d280136c0 con 0x7f1d38060220 2024-01-21T11:47:43.554 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:43.552+0000 7f1d2dffb700 1 -- 172.21.15.134:0/391599872 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1d2801ee90 con 0x7f1d38060220 2024-01-21T11:47:43.554 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:43.552+0000 7f1d3f971700 1 -- 172.21.15.134:0/391599872 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d380f5680 con 0x7f1d38060220 2024-01-21T11:47:43.555 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:43.553+0000 7f1d2dffb700 1 -- 172.21.15.134:0/391599872 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1d2801e420 con 0x7f1d38060220 2024-01-21T11:47:43.555 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:43.553+0000 7f1d3f971700 1 -- 172.21.15.134:0/391599872 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d38059100 con 0x7f1d38060220 2024-01-21T11:47:43.555 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:43.553+0000 7f1d2dffb700 1 --2- 172.21.15.134:0/391599872 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f1d2402ff10 0x7f1d240323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:47:43.555 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:43.553+0000 7f1d2dffb700 1 -- 172.21.15.134:0/391599872 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1d280410b0 con 0x7f1d38060220 2024-01-21T11:47:43.555 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:43.553+0000 7f1d3cf0c700 1 --2- 172.21.15.134:0/391599872 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f1d2402ff10 0x7f1d240323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:47:43.555 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:43.554+0000 7f1d3cf0c700 1 --2- 172.21.15.134:0/391599872 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f1d2402ff10 0x7f1d240323d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f1d340096a0 tx=0x7f1d34006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:47:43.558 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:43.558+0000 7f1d2dffb700 1 -- 172.21.15.134:0/391599872 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1d2806d2b0 con 0x7f1d38060220 2024-01-21T11:47:43.597 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:43 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:47:41.856071+0000 mgr.smithi029.blhxxk (mgr.14164) 171 : cluster [DBG] pgmap v164: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:47:43.598 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:43 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cephadm 2024-01-21T11:47:41.913673+0000 mgr.smithi029.blhxxk (mgr.14164) 172 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon alertmanager.smithi029 ... 2024-01-21T11:47:43.598 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:43 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2024-01-21T11:47:43.598 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:43 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-01-21T11:47:43.598 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:43 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: stat: stderr time="2024-01-21T11:43:20Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: copying system image from manifest list: determining manifest MIME type for docker://prom/alertmanager:v0.20.0: reading manifest sha256:b9323917a2eda265bec69e59a457f001c529facbbc8166df277f4850cdac61a0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T11:47:43.598 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:43 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: stat: stderr time="2024-01-21T11:43:58Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T11:47:43.598 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:43 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: stat: stderr time="2024-01-21T11:44:35Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T11:47:43.598 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:43 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-01-21T11:47:43.598 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:43 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2024-01-21T11:47:43.598 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:43 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-01-21T11:47:43.598 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:43 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: stat: stderr time="2024-01-21T11:45:49Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T11:47:43.598 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:43 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: stat: stderr time="2024-01-21T11:46:26Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T11:47:43.598 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:43 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: stat: stderr time="2024-01-21T11:47:04Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T11:47:43.598 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:43 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-01-21T11:47:43.598 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:43 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: Traceback (most recent call last): 2024-01-21T11:47:43.599 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:43 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: File "/var/lib/ceph/e4084074-b851-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-21T11:47:43.599 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:43 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: main() 2024-01-21T11:47:43.599 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:43 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: File "/var/lib/ceph/e4084074-b851-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-21T11:47:43.599 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:43 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: r = ctx.func(ctx) 2024-01-21T11:47:43.599 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:43 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: File "/var/lib/ceph/e4084074-b851-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-21T11:47:43.599 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:43 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: return func(ctx) 2024-01-21T11:47:43.599 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:43 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: File "/var/lib/ceph/e4084074-b851-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-01-21T11:47:43.599 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:43 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-01-21T11:47:43.599 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:43 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: File "/var/lib/ceph/e4084074-b851-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4191, in extract_uid_gid_monitoring 2024-01-21T11:47:43.599 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:43 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2024-01-21T11:47:43.599 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:43 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: File "/var/lib/ceph/e4084074-b851-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-01-21T11:47:43.599 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:43 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: raise RuntimeError('uid/gid not found') 2024-01-21T11:47:43.599 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:43 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: RuntimeError: uid/gid not found 2024-01-21T11:47:43.599 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:43 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: Traceback (most recent call last): 2024-01-21T11:47:43.599 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:43 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2024-01-21T11:47:43.600 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:43 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: yield (conn, connr) 2024-01-21T11:47:43.600 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:43 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2024-01-21T11:47:43.600 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:43 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: code, '\n'.join(err))) 2024-01-21T11:47:43.600 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:43 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon alertmanager.smithi029 ... 2024-01-21T11:47:43.600 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:43 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2024-01-21T11:47:43.600 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:43 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-01-21T11:47:43.600 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:43 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: stat: stderr time="2024-01-21T11:43:20Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: copying system image from manifest list: determining manifest MIME type for docker://prom/alertmanager:v0.20.0: reading manifest sha256:b9323917a2eda265bec69e59a457f001c529facbbc8166df277f4850cdac61a0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T11:47:43.600 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:43 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: stat: stderr time="2024-01-21T11:43:58Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T11:47:43.600 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:43 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: stat: stderr time="2024-01-21T11:44:35Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T11:47:43.600 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:43 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-01-21T11:47:43.600 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:43 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2024-01-21T11:47:43.600 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:43 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-01-21T11:47:43.600 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:43 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: stat: stderr time="2024-01-21T11:45:49Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T11:47:43.600 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:43 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: stat: stderr time="2024-01-21T11:46:26Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T11:47:43.601 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:43 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: stat: stderr time="2024-01-21T11:47:04Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T11:47:43.601 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:43 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-01-21T11:47:43.601 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:43 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: Traceback (most recent call last): 2024-01-21T11:47:43.601 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:43 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: File "/var/lib/ceph/e4084074-b851-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-21T11:47:43.765 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:43.764+0000 7f1d3f971700 1 -- 172.21.15.134:0/391599872 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1d380fbf50 con 0x7f1d38060220 2024-01-21T11:47:43.765 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:43.765+0000 7f1d2dffb700 1 -- 172.21.15.134:0/391599872 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1d2801cb10 con 0x7f1d38060220 2024-01-21T11:47:43.766 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:47:43.766 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:47:43.767 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:43.766+0000 7f1d3f971700 1 -- 172.21.15.134:0/391599872 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f1d2402ff10 msgr2=0x7f1d240323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:47:43.767 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:43.767+0000 7f1d3f971700 1 --2- 172.21.15.134:0/391599872 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f1d2402ff10 0x7f1d240323d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f1d340096a0 tx=0x7f1d34006b40).stop 2024-01-21T11:47:43.767 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:43.767+0000 7f1d3f971700 1 -- 172.21.15.134:0/391599872 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d38060220 msgr2=0x7f1d380f6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:47:43.767 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:43.767+0000 7f1d3f971700 1 --2- 172.21.15.134:0/391599872 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d38060220 0x7f1d380f6bd0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7f1d28009830 tx=0x7f1d2800b040).stop 2024-01-21T11:47:43.767 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:43.767+0000 7f1d3f971700 1 -- 172.21.15.134:0/391599872 shutdown_connections 2024-01-21T11:47:43.767 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:43.767+0000 7f1d3f971700 1 --2- 172.21.15.134:0/391599872 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f1d2402ff10 0x7f1d240323d0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:47:43.767 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:43.767+0000 7f1d3f971700 1 --2- 172.21.15.134:0/391599872 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d38060220 0x7f1d380f6bd0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:47:43.767 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:43.767+0000 7f1d3f971700 1 -- 172.21.15.134:0/391599872 >> 172.21.15.134:0/391599872 conn(0x7f1d380f0b70 msgr2=0x7f1d380f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:47:43.767 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:43.767+0000 7f1d3f971700 1 -- 172.21.15.134:0/391599872 shutdown_connections 2024-01-21T11:47:43.767 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:43.767+0000 7f1d3f971700 1 -- 172.21.15.134:0/391599872 wait complete. 2024-01-21T11:47:43.768 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:47:43.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:43 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: main() 2024-01-21T11:47:43.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:43 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: File "/var/lib/ceph/e4084074-b851-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-21T11:47:43.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:43 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: r = ctx.func(ctx) 2024-01-21T11:47:43.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:43 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: File "/var/lib/ceph/e4084074-b851-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-21T11:47:43.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:43 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: return func(ctx) 2024-01-21T11:47:43.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:43 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: File "/var/lib/ceph/e4084074-b851-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-01-21T11:47:43.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:43 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-01-21T11:47:43.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:43 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: File "/var/lib/ceph/e4084074-b851-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4191, in extract_uid_gid_monitoring 2024-01-21T11:47:43.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:43 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2024-01-21T11:47:43.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:43 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: File "/var/lib/ceph/e4084074-b851-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-01-21T11:47:43.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:43 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: raise RuntimeError('uid/gid not found') 2024-01-21T11:47:43.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:43 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: RuntimeError: uid/gid not found 2024-01-21T11:47:43.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:43 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cephadm 2024-01-21T11:47:41.918794+0000 mgr.smithi029.blhxxk (mgr.14164) 173 : cephadm [INF] Deploying daemon crash.smithi029 on smithi029 2024-01-21T11:47:43.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:43 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:47:43.392335+0000 mon.smithi029 (mon.0) 243 : audit [INF] from='mgr.14164 172.21.15.29:0/3252518221' entity='mgr.smithi029.blhxxk' 2024-01-21T11:47:45.157 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:47:45.157 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:47:45.168 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:44 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:47:43.710810+0000 mon.smithi029 (mon.0) 244 : audit [INF] from='mgr.14164 172.21.15.29:0/3252518221' entity='mgr.smithi029.blhxxk' 2024-01-21T11:47:45.169 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:44 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:47:43.712605+0000 mon.smithi029 (mon.0) 245 : audit [INF] from='mgr.14164 172.21.15.29:0/3252518221' entity='mgr.smithi029.blhxxk' 2024-01-21T11:47:45.169 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:44 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:47:43.713000+0000 mon.smithi029 (mon.0) 246 : audit [INF] from='mgr.14164 172.21.15.29:0/3252518221' entity='mgr.smithi029.blhxxk' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-01-21T11:47:45.169 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:44 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:47:43.713401+0000 mgr.smithi029.blhxxk (mgr.14164) 174 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-01-21T11:47:45.169 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:44 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:47:43.715668+0000 mon.smithi029 (mon.0) 247 : audit [INF] from='mgr.14164 172.21.15.29:0/3252518221' entity='mgr.smithi029.blhxxk' 2024-01-21T11:47:45.169 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:44 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cephadm 2024-01-21T11:47:43.718554+0000 mgr.smithi029.blhxxk (mgr.14164) 175 : cephadm [INF] Deploying daemon grafana.smithi029 on smithi029 2024-01-21T11:47:45.169 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:44 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:47:43.766302+0000 mon.smithi029 (mon.0) 248 : audit [DBG] from='client.? 172.21.15.134:0/391599872' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:47:45.169 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:44 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:47:43.856339+0000 mgr.smithi029.blhxxk (mgr.14164) 176 : cluster [DBG] pgmap v165: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:47:46.361 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:46.360+0000 7fad597b5700 1 -- 172.21.15.134:0/2332437540 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fad540f62c0 msgr2=0x7fad540f66e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:47:46.361 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:46.360+0000 7fad597b5700 1 --2- 172.21.15.134:0/2332437540 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fad540f62c0 0x7fad540f66e0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7fad44009fb0 tx=0x7fad44009b30).stop 2024-01-21T11:47:46.361 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:46.360+0000 7fad597b5700 1 -- 172.21.15.134:0/2332437540 shutdown_connections 2024-01-21T11:47:46.361 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:46.360+0000 7fad597b5700 1 --2- 172.21.15.134:0/2332437540 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fad540f62c0 0x7fad540f66e0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:47:46.361 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:46.360+0000 7fad597b5700 1 -- 172.21.15.134:0/2332437540 >> 172.21.15.134:0/2332437540 conn(0x7fad540f1a20 msgr2=0x7fad540f3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:47:46.361 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:46.360+0000 7fad597b5700 1 -- 172.21.15.134:0/2332437540 shutdown_connections 2024-01-21T11:47:46.361 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:46.361+0000 7fad597b5700 1 -- 172.21.15.134:0/2332437540 wait complete. 2024-01-21T11:47:46.363 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:46.361+0000 7fad597b5700 1 Processor -- start 2024-01-21T11:47:46.363 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:46.361+0000 7fad597b5700 1 -- start start 2024-01-21T11:47:46.363 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:46.361+0000 7fad597b5700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fad540f62c0 0x7fad5406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:47:46.363 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:46.361+0000 7fad597b5700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad54069e20 con 0x7fad540f62c0 2024-01-21T11:47:46.363 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:46.361+0000 7fad52ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fad540f62c0 0x7fad5406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:47:46.363 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:46.362+0000 7fad52ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fad540f62c0 0x7fad5406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:51204/0 (socket says 172.21.15.134:51204) 2024-01-21T11:47:46.363 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:46.362+0000 7fad52ffd700 1 -- 172.21.15.134:0/3531912112 learned_addr learned my addr 172.21.15.134:0/3531912112 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:47:46.363 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:46.362+0000 7fad52ffd700 1 -- 172.21.15.134:0/3531912112 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad44005040 con 0x7fad540f62c0 2024-01-21T11:47:46.363 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:46.362+0000 7fad52ffd700 1 --2- 172.21.15.134:0/3531912112 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fad540f62c0 0x7fad5406b9a0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7fad44013510 tx=0x7fad4400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:47:46.363 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:46.362+0000 7fad3b7fe700 1 -- 172.21.15.134:0/3531912112 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fad44009320 con 0x7fad540f62c0 2024-01-21T11:47:46.363 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:46.362+0000 7fad597b5700 1 -- 172.21.15.134:0/3531912112 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad54069fc0 con 0x7fad540f62c0 2024-01-21T11:47:46.363 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:46.362+0000 7fad3b7fe700 1 -- 172.21.15.134:0/3531912112 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fad44013f00 con 0x7fad540f62c0 2024-01-21T11:47:46.363 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:46.362+0000 7fad3b7fe700 1 -- 172.21.15.134:0/3531912112 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fad44014d90 con 0x7fad540f62c0 2024-01-21T11:47:46.364 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:46.362+0000 7fad597b5700 1 -- 172.21.15.134:0/3531912112 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad5406a450 con 0x7fad540f62c0 2024-01-21T11:47:46.364 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:46.363+0000 7fad3b7fe700 1 -- 172.21.15.134:0/3531912112 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fad440095f0 con 0x7fad540f62c0 2024-01-21T11:47:46.399 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:46.363+0000 7fad597b5700 1 -- 172.21.15.134:0/3531912112 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad54059100 con 0x7fad540f62c0 2024-01-21T11:47:46.399 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:46.363+0000 7fad3b7fe700 1 --2- 172.21.15.134:0/3531912112 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fad3c02ff60 0x7fad3c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:47:46.399 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:46.363+0000 7fad3b7fe700 1 -- 172.21.15.134:0/3531912112 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fad44041000 con 0x7fad540f62c0 2024-01-21T11:47:46.399 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:46.363+0000 7fad527fc700 1 --2- 172.21.15.134:0/3531912112 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fad3c02ff60 0x7fad3c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:47:46.399 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:46.364+0000 7fad527fc700 1 --2- 172.21.15.134:0/3531912112 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fad3c02ff60 0x7fad3c032420 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7fad480096a0 tx=0x7fad48006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:47:46.400 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:46.368+0000 7fad3b7fe700 1 -- 172.21.15.134:0/3531912112 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fad44014420 con 0x7fad540f62c0 2024-01-21T11:47:46.572 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:46.571+0000 7fad597b5700 1 -- 172.21.15.134:0/3531912112 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fad54045ad0 con 0x7fad540f62c0 2024-01-21T11:47:46.572 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:46.572+0000 7fad3b7fe700 1 -- 172.21.15.134:0/3531912112 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fad4401d7f0 con 0x7fad540f62c0 2024-01-21T11:47:46.572 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:47:46.572 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:47:46.574 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:46.574+0000 7fad597b5700 1 -- 172.21.15.134:0/3531912112 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fad3c02ff60 msgr2=0x7fad3c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:47:46.574 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:46.574+0000 7fad597b5700 1 --2- 172.21.15.134:0/3531912112 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fad3c02ff60 0x7fad3c032420 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7fad480096a0 tx=0x7fad48006b40).stop 2024-01-21T11:47:46.574 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:46.574+0000 7fad597b5700 1 -- 172.21.15.134:0/3531912112 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fad540f62c0 msgr2=0x7fad5406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:47:46.574 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:46.574+0000 7fad597b5700 1 --2- 172.21.15.134:0/3531912112 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fad540f62c0 0x7fad5406b9a0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7fad44013510 tx=0x7fad4400b040).stop 2024-01-21T11:47:46.574 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:46.574+0000 7fad597b5700 1 -- 172.21.15.134:0/3531912112 shutdown_connections 2024-01-21T11:47:46.574 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:46.574+0000 7fad597b5700 1 --2- 172.21.15.134:0/3531912112 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fad3c02ff60 0x7fad3c032420 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:47:46.574 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:46.574+0000 7fad597b5700 1 --2- 172.21.15.134:0/3531912112 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fad540f62c0 0x7fad5406b9a0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:47:46.574 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:46.574+0000 7fad597b5700 1 -- 172.21.15.134:0/3531912112 >> 172.21.15.134:0/3531912112 conn(0x7fad540f1a20 msgr2=0x7fad540f2580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:47:46.574 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:46.574+0000 7fad597b5700 1 -- 172.21.15.134:0/3531912112 shutdown_connections 2024-01-21T11:47:46.574 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:46.574+0000 7fad597b5700 1 -- 172.21.15.134:0/3531912112 wait complete. 2024-01-21T11:47:46.575 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:47:47.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:47 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:47:45.856632+0000 mgr.smithi029.blhxxk (mgr.14164) 177 : cluster [DBG] pgmap v166: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:47:47.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:47 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:47:46.573130+0000 mon.smithi029 (mon.0) 249 : audit [DBG] from='client.? 172.21.15.134:0/3531912112' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:47:47.949 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:47:47.949 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:47:49.129 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:49.127+0000 7feec6823700 1 -- 172.21.15.134:0/1075773849 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feec00f5410 msgr2=0x7feec00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:47:49.129 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:49.127+0000 7feec6823700 1 --2- 172.21.15.134:0/1075773849 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feec00f5410 0x7feec00f5830 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7feeb0009fb0 tx=0x7feeb0009b30).stop 2024-01-21T11:47:49.129 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:49.127+0000 7feec6823700 1 -- 172.21.15.134:0/1075773849 shutdown_connections 2024-01-21T11:47:49.129 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:49.127+0000 7feec6823700 1 --2- 172.21.15.134:0/1075773849 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feec00f5410 0x7feec00f5830 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:47:49.129 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:49.127+0000 7feec6823700 1 -- 172.21.15.134:0/1075773849 >> 172.21.15.134:0/1075773849 conn(0x7feec00f0b70 msgr2=0x7feec00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:47:49.129 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:49.127+0000 7feec6823700 1 -- 172.21.15.134:0/1075773849 shutdown_connections 2024-01-21T11:47:49.129 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:49.128+0000 7feec6823700 1 -- 172.21.15.134:0/1075773849 wait complete. 2024-01-21T11:47:49.129 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:49.128+0000 7feec6823700 1 Processor -- start 2024-01-21T11:47:49.129 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:49.128+0000 7feec6823700 1 -- start start 2024-01-21T11:47:49.129 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:49.128+0000 7feec6823700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feec00f5410 0x7feec0105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:47:49.130 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:49.128+0000 7feec6823700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feec0105b20 con 0x7feec00f5410 2024-01-21T11:47:49.130 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:49.128+0000 7feebffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feec00f5410 0x7feec0105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:47:49.130 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:49.128+0000 7feebffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feec00f5410 0x7feec0105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:51232/0 (socket says 172.21.15.134:51232) 2024-01-21T11:47:49.130 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:49.128+0000 7feebffff700 1 -- 172.21.15.134:0/1011416442 learned_addr learned my addr 172.21.15.134:0/1011416442 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:47:49.130 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:49.129+0000 7feebffff700 1 -- 172.21.15.134:0/1011416442 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feeb0005040 con 0x7feec00f5410 2024-01-21T11:47:49.130 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:49.129+0000 7feebffff700 1 --2- 172.21.15.134:0/1011416442 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feec00f5410 0x7feec0105630 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7feeb0013510 tx=0x7feeb000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:47:49.131 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:49.129+0000 7feebcff9700 1 -- 172.21.15.134:0/1011416442 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feeb0009320 con 0x7feec00f5410 2024-01-21T11:47:49.131 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:49.129+0000 7feebcff9700 1 -- 172.21.15.134:0/1011416442 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7feeb0013f00 con 0x7feec00f5410 2024-01-21T11:47:49.131 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:49.129+0000 7feec6823700 1 -- 172.21.15.134:0/1011416442 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feec0105d20 con 0x7feec00f5410 2024-01-21T11:47:49.131 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:49.129+0000 7feebcff9700 1 -- 172.21.15.134:0/1011416442 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feeb0014d90 con 0x7feec00f5410 2024-01-21T11:47:49.131 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:49.129+0000 7feec6823700 1 -- 172.21.15.134:0/1011416442 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feec01061b0 con 0x7feec00f5410 2024-01-21T11:47:49.132 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:49.130+0000 7feebcff9700 1 -- 172.21.15.134:0/1011416442 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7feeb0014420 con 0x7feec00f5410 2024-01-21T11:47:49.132 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:49.130+0000 7feec6823700 1 -- 172.21.15.134:0/1011416442 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feec0045ad0 con 0x7feec00f5410 2024-01-21T11:47:49.132 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:49.130+0000 7feebcff9700 1 --2- 172.21.15.134:0/1011416442 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7feea802ff60 0x7feea8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:47:49.132 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:49.130+0000 7feebcff9700 1 -- 172.21.15.134:0/1011416442 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7feeb0041000 con 0x7feec00f5410 2024-01-21T11:47:49.132 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:49.130+0000 7feebf7fe700 1 --2- 172.21.15.134:0/1011416442 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7feea802ff60 0x7feea8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:47:49.132 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:49.131+0000 7feebf7fe700 1 --2- 172.21.15.134:0/1011416442 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7feea802ff60 0x7feea8032420 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7feeb40096a0 tx=0x7feeb4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:47:49.135 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:49.135+0000 7feebcff9700 1 -- 172.21.15.134:0/1011416442 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7feeb00095f0 con 0x7feec00f5410 2024-01-21T11:47:49.340 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:49.340+0000 7feec6823700 1 -- 172.21.15.134:0/1011416442 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7feec00fae60 con 0x7feec00f5410 2024-01-21T11:47:49.341 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:49.341+0000 7feebcff9700 1 -- 172.21.15.134:0/1011416442 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7feeb001db70 con 0x7feec00f5410 2024-01-21T11:47:49.341 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:47:49.341 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:47:49.343 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:49.342+0000 7feec6823700 1 -- 172.21.15.134:0/1011416442 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7feea802ff60 msgr2=0x7feea8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:47:49.343 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:49.342+0000 7feec6823700 1 --2- 172.21.15.134:0/1011416442 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7feea802ff60 0x7feea8032420 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7feeb40096a0 tx=0x7feeb4006b40).stop 2024-01-21T11:47:49.343 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:49.342+0000 7feec6823700 1 -- 172.21.15.134:0/1011416442 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feec00f5410 msgr2=0x7feec0105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:47:49.343 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:49.342+0000 7feec6823700 1 --2- 172.21.15.134:0/1011416442 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feec00f5410 0x7feec0105630 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7feeb0013510 tx=0x7feeb000b040).stop 2024-01-21T11:47:49.343 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:49.342+0000 7feec6823700 1 -- 172.21.15.134:0/1011416442 shutdown_connections 2024-01-21T11:47:49.343 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:49.342+0000 7feec6823700 1 --2- 172.21.15.134:0/1011416442 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7feea802ff60 0x7feea8032420 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:47:49.343 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:49.342+0000 7feec6823700 1 --2- 172.21.15.134:0/1011416442 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feec00f5410 0x7feec0105630 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:47:49.343 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:49.342+0000 7feec6823700 1 -- 172.21.15.134:0/1011416442 >> 172.21.15.134:0/1011416442 conn(0x7feec00f0b70 msgr2=0x7feec00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:47:49.343 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:49.342+0000 7feec6823700 1 -- 172.21.15.134:0/1011416442 shutdown_connections 2024-01-21T11:47:49.343 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:49.342+0000 7feec6823700 1 -- 172.21.15.134:0/1011416442 wait complete. 2024-01-21T11:47:49.344 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:47:49.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:49 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:47:47.856926+0000 mgr.smithi029.blhxxk (mgr.14164) 178 : cluster [DBG] pgmap v167: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:47:49.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:49 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:47:49.341986+0000 mon.smithi029 (mon.0) 250 : audit [DBG] from='client.? 172.21.15.134:0/1011416442' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:47:50.712 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:47:50.712 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:47:51.848 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:51.847+0000 7f782d54b700 1 -- 172.21.15.134:0/2163772710 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78280f6510 msgr2=0x7f78280f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:47:51.848 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:51.847+0000 7f782d54b700 1 --2- 172.21.15.134:0/2163772710 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78280f6510 0x7f78280f6930 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7f781c009fb0 tx=0x7f781c009b30).stop 2024-01-21T11:47:51.848 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:51.847+0000 7f782d54b700 1 -- 172.21.15.134:0/2163772710 shutdown_connections 2024-01-21T11:47:51.849 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:51.847+0000 7f782d54b700 1 --2- 172.21.15.134:0/2163772710 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78280f6510 0x7f78280f6930 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:47:51.849 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:51.847+0000 7f782d54b700 1 -- 172.21.15.134:0/2163772710 >> 172.21.15.134:0/2163772710 conn(0x7f78280f1c70 msgr2=0x7f78280f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:47:51.849 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:51.847+0000 7f782d54b700 1 -- 172.21.15.134:0/2163772710 shutdown_connections 2024-01-21T11:47:51.849 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:51.847+0000 7f782d54b700 1 -- 172.21.15.134:0/2163772710 wait complete. 2024-01-21T11:47:51.849 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:51.847+0000 7f782d54b700 1 Processor -- start 2024-01-21T11:47:51.849 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:51.847+0000 7f782d54b700 1 -- start start 2024-01-21T11:47:51.849 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:51.848+0000 7f782d54b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78280f6510 0x7f7828106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:47:51.849 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:51.848+0000 7f782d54b700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7828106c20 con 0x7f78280f6510 2024-01-21T11:47:51.849 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:51.848+0000 7f7826ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78280f6510 0x7f7828106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:47:51.849 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:51.848+0000 7f7826ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78280f6510 0x7f7828106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:51244/0 (socket says 172.21.15.134:51244) 2024-01-21T11:47:51.849 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:51.848+0000 7f7826ffd700 1 -- 172.21.15.134:0/253179068 learned_addr learned my addr 172.21.15.134:0/253179068 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:47:51.849 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:51.848+0000 7f7826ffd700 1 -- 172.21.15.134:0/253179068 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f781c005040 con 0x7f78280f6510 2024-01-21T11:47:51.849 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:51.848+0000 7f7826ffd700 1 --2- 172.21.15.134:0/253179068 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78280f6510 0x7f7828106730 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f781c013510 tx=0x7f781c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:47:51.851 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:51.848+0000 7f780f7fe700 1 -- 172.21.15.134:0/253179068 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f781c009320 con 0x7f78280f6510 2024-01-21T11:47:51.851 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:51.848+0000 7f782d54b700 1 -- 172.21.15.134:0/253179068 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7828106e20 con 0x7f78280f6510 2024-01-21T11:47:51.851 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:51.849+0000 7f782d54b700 1 -- 172.21.15.134:0/253179068 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78281072b0 con 0x7f78280f6510 2024-01-21T11:47:51.851 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:51.849+0000 7f780f7fe700 1 -- 172.21.15.134:0/253179068 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f781c013f00 con 0x7f78280f6510 2024-01-21T11:47:51.851 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:51.849+0000 7f780f7fe700 1 -- 172.21.15.134:0/253179068 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f781c014d60 con 0x7f78280f6510 2024-01-21T11:47:51.851 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:51.849+0000 7f782d54b700 1 -- 172.21.15.134:0/253179068 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7828045ad0 con 0x7f78280f6510 2024-01-21T11:47:51.851 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:51.849+0000 7f780f7fe700 1 -- 172.21.15.134:0/253179068 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f781c0095f0 con 0x7f78280f6510 2024-01-21T11:47:51.851 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:51.850+0000 7f780f7fe700 1 --2- 172.21.15.134:0/253179068 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f781002ff60 0x7f7810032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:47:51.851 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:51.850+0000 7f780f7fe700 1 -- 172.21.15.134:0/253179068 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f781c0410d0 con 0x7f78280f6510 2024-01-21T11:47:51.851 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:51.850+0000 7f78267fc700 1 --2- 172.21.15.134:0/253179068 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f781002ff60 0x7f7810032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:47:51.851 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:51.850+0000 7f78267fc700 1 --2- 172.21.15.134:0/253179068 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f781002ff60 0x7f7810032420 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f78180096a0 tx=0x7f7818006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:47:51.854 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:51.854+0000 7f780f7fe700 1 -- 172.21.15.134:0/253179068 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f781c014420 con 0x7f78280f6510 2024-01-21T11:47:51.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:51 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:47:49.857205+0000 mgr.smithi029.blhxxk (mgr.14164) 179 : cluster [DBG] pgmap v168: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:47:52.063 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:52.063+0000 7f782d54b700 1 -- 172.21.15.134:0/253179068 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f78280fbf60 con 0x7f78280f6510 2024-01-21T11:47:52.064 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:52.064+0000 7f780f7fe700 1 -- 172.21.15.134:0/253179068 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f781c012980 con 0x7f78280f6510 2024-01-21T11:47:52.064 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:47:52.064 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:47:52.065 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:52.065+0000 7f782d54b700 1 -- 172.21.15.134:0/253179068 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f781002ff60 msgr2=0x7f7810032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:47:52.066 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:52.065+0000 7f782d54b700 1 --2- 172.21.15.134:0/253179068 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f781002ff60 0x7f7810032420 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f78180096a0 tx=0x7f7818006b40).stop 2024-01-21T11:47:52.066 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:52.065+0000 7f782d54b700 1 -- 172.21.15.134:0/253179068 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78280f6510 msgr2=0x7f7828106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:47:52.066 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:52.065+0000 7f782d54b700 1 --2- 172.21.15.134:0/253179068 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78280f6510 0x7f7828106730 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f781c013510 tx=0x7f781c00b040).stop 2024-01-21T11:47:52.066 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:52.065+0000 7f782d54b700 1 -- 172.21.15.134:0/253179068 shutdown_connections 2024-01-21T11:47:52.066 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:52.065+0000 7f782d54b700 1 --2- 172.21.15.134:0/253179068 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f781002ff60 0x7f7810032420 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:47:52.066 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:52.065+0000 7f782d54b700 1 --2- 172.21.15.134:0/253179068 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78280f6510 0x7f7828106730 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:47:52.066 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:52.065+0000 7f782d54b700 1 -- 172.21.15.134:0/253179068 >> 172.21.15.134:0/253179068 conn(0x7f78280f1c70 msgr2=0x7f78280f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:47:52.066 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:52.065+0000 7f782d54b700 1 -- 172.21.15.134:0/253179068 shutdown_connections 2024-01-21T11:47:52.066 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:52.065+0000 7f782d54b700 1 -- 172.21.15.134:0/253179068 wait complete. 2024-01-21T11:47:52.067 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:47:52.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:52 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:47:52.064983+0000 mon.smithi029 (mon.0) 251 : audit [DBG] from='client.? 172.21.15.134:0/253179068' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:47:53.440 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:47:53.440 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:47:53.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:53 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:47:51.857604+0000 mgr.smithi029.blhxxk (mgr.14164) 180 : cluster [DBG] pgmap v169: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:47:54.599 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:54.598+0000 7fa6d6771700 1 -- 172.21.15.134:0/2292772050 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6d00f5410 msgr2=0x7fa6d00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:47:54.600 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:54.598+0000 7fa6d6771700 1 --2- 172.21.15.134:0/2292772050 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6d00f5410 0x7fa6d00f5830 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7fa6c0009fb0 tx=0x7fa6c0009b30).stop 2024-01-21T11:47:54.600 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:54.598+0000 7fa6d6771700 1 -- 172.21.15.134:0/2292772050 shutdown_connections 2024-01-21T11:47:54.600 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:54.598+0000 7fa6d6771700 1 --2- 172.21.15.134:0/2292772050 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6d00f5410 0x7fa6d00f5830 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:47:54.600 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:54.598+0000 7fa6d6771700 1 -- 172.21.15.134:0/2292772050 >> 172.21.15.134:0/2292772050 conn(0x7fa6d00f0b70 msgr2=0x7fa6d00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:47:54.600 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:54.598+0000 7fa6d6771700 1 -- 172.21.15.134:0/2292772050 shutdown_connections 2024-01-21T11:47:54.600 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:54.598+0000 7fa6d6771700 1 -- 172.21.15.134:0/2292772050 wait complete. 2024-01-21T11:47:54.600 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:54.598+0000 7fa6d6771700 1 Processor -- start 2024-01-21T11:47:54.600 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:54.599+0000 7fa6d6771700 1 -- start start 2024-01-21T11:47:54.600 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:54.599+0000 7fa6d6771700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6d00f5410 0x7fa6d00fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:47:54.600 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:54.599+0000 7fa6d6771700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6d00fb820 con 0x7fa6d00f5410 2024-01-21T11:47:54.600 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:54.599+0000 7fa6cffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6d00f5410 0x7fa6d00fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:47:54.601 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:54.599+0000 7fa6cffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6d00f5410 0x7fa6d00fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:47700/0 (socket says 172.21.15.134:47700) 2024-01-21T11:47:54.601 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:54.599+0000 7fa6cffff700 1 -- 172.21.15.134:0/61850636 learned_addr learned my addr 172.21.15.134:0/61850636 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:47:54.601 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:54.599+0000 7fa6cffff700 1 -- 172.21.15.134:0/61850636 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6c0005040 con 0x7fa6d00f5410 2024-01-21T11:47:54.601 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:54.599+0000 7fa6cffff700 1 --2- 172.21.15.134:0/61850636 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6d00f5410 0x7fa6d00fd3a0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7fa6c0013510 tx=0x7fa6c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:47:54.601 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:54.599+0000 7fa6ccff9700 1 -- 172.21.15.134:0/61850636 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa6c0009320 con 0x7fa6d00f5410 2024-01-21T11:47:54.601 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:54.599+0000 7fa6d6771700 1 -- 172.21.15.134:0/61850636 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6d00fb9c0 con 0x7fa6d00f5410 2024-01-21T11:47:54.601 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:54.599+0000 7fa6ccff9700 1 -- 172.21.15.134:0/61850636 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa6c0013f00 con 0x7fa6d00f5410 2024-01-21T11:47:54.601 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:54.599+0000 7fa6ccff9700 1 -- 172.21.15.134:0/61850636 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa6c0014d90 con 0x7fa6d00f5410 2024-01-21T11:47:54.601 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:54.599+0000 7fa6d6771700 1 -- 172.21.15.134:0/61850636 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6d00fbe50 con 0x7fa6d00f5410 2024-01-21T11:47:54.602 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:54.600+0000 7fa6ccff9700 1 -- 172.21.15.134:0/61850636 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa6c0014420 con 0x7fa6d00f5410 2024-01-21T11:47:54.602 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:54.600+0000 7fa6ccff9700 1 --2- 172.21.15.134:0/61850636 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fa6b802ff60 0x7fa6b8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:47:54.602 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:54.600+0000 7fa6ccff9700 1 -- 172.21.15.134:0/61850636 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa6c00410c0 con 0x7fa6d00f5410 2024-01-21T11:47:54.602 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:54.600+0000 7fa6d6771700 1 -- 172.21.15.134:0/61850636 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6d0059100 con 0x7fa6d00f5410 2024-01-21T11:47:54.602 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:54.600+0000 7fa6cf7fe700 1 --2- 172.21.15.134:0/61850636 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fa6b802ff60 0x7fa6b8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:47:54.602 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:54.601+0000 7fa6cf7fe700 1 --2- 172.21.15.134:0/61850636 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fa6b802ff60 0x7fa6b8032420 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fa6c40096a0 tx=0x7fa6c4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:47:54.605 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:54.605+0000 7fa6ccff9700 1 -- 172.21.15.134:0/61850636 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa6c00095f0 con 0x7fa6d00f5410 2024-01-21T11:47:54.810 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:54.810+0000 7fa6d6771700 1 -- 172.21.15.134:0/61850636 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa6d00fc550 con 0x7fa6d00f5410 2024-01-21T11:47:54.810 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:54.810+0000 7fa6ccff9700 1 -- 172.21.15.134:0/61850636 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa6c001db70 con 0x7fa6d00f5410 2024-01-21T11:47:54.811 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:47:54.811 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:47:54.812 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:54.812+0000 7fa6d6771700 1 -- 172.21.15.134:0/61850636 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fa6b802ff60 msgr2=0x7fa6b8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:47:54.812 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:54.812+0000 7fa6d6771700 1 --2- 172.21.15.134:0/61850636 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fa6b802ff60 0x7fa6b8032420 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fa6c40096a0 tx=0x7fa6c4006b40).stop 2024-01-21T11:47:54.813 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:54.812+0000 7fa6d6771700 1 -- 172.21.15.134:0/61850636 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6d00f5410 msgr2=0x7fa6d00fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:47:54.813 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:54.812+0000 7fa6d6771700 1 --2- 172.21.15.134:0/61850636 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6d00f5410 0x7fa6d00fd3a0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7fa6c0013510 tx=0x7fa6c000b040).stop 2024-01-21T11:47:54.813 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:54.812+0000 7fa6d6771700 1 -- 172.21.15.134:0/61850636 shutdown_connections 2024-01-21T11:47:54.813 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:54.812+0000 7fa6d6771700 1 --2- 172.21.15.134:0/61850636 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fa6b802ff60 0x7fa6b8032420 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:47:54.813 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:54.812+0000 7fa6d6771700 1 --2- 172.21.15.134:0/61850636 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6d00f5410 0x7fa6d00fd3a0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:47:54.813 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:54.812+0000 7fa6d6771700 1 -- 172.21.15.134:0/61850636 >> 172.21.15.134:0/61850636 conn(0x7fa6d00f0b70 msgr2=0x7fa6d005fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:47:54.813 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:54.812+0000 7fa6d6771700 1 -- 172.21.15.134:0/61850636 shutdown_connections 2024-01-21T11:47:54.813 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:54.812+0000 7fa6d6771700 1 -- 172.21.15.134:0/61850636 wait complete. 2024-01-21T11:47:54.813 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:47:55.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:55 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:47:53.857882+0000 mgr.smithi029.blhxxk (mgr.14164) 181 : cluster [DBG] pgmap v170: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:47:55.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:55 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:47:54.811757+0000 mon.smithi029 (mon.0) 252 : audit [DBG] from='client.? 172.21.15.134:0/61850636' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:47:56.165 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:47:56.165 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:47:57.344 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:57.342+0000 7f9425d23700 1 -- 172.21.15.134:0/2395760218 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94200f6510 msgr2=0x7f94200f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:47:57.344 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:57.342+0000 7f9425d23700 1 --2- 172.21.15.134:0/2395760218 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94200f6510 0x7f94200f6930 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7f9410009fb0 tx=0x7f9410009b30).stop 2024-01-21T11:47:57.345 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:57.342+0000 7f9425d23700 1 -- 172.21.15.134:0/2395760218 shutdown_connections 2024-01-21T11:47:57.345 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:57.342+0000 7f9425d23700 1 --2- 172.21.15.134:0/2395760218 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94200f6510 0x7f94200f6930 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:47:57.345 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:57.342+0000 7f9425d23700 1 -- 172.21.15.134:0/2395760218 >> 172.21.15.134:0/2395760218 conn(0x7f94200f1c70 msgr2=0x7f94200f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:47:57.345 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:57.342+0000 7f9425d23700 1 -- 172.21.15.134:0/2395760218 shutdown_connections 2024-01-21T11:47:57.345 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:57.342+0000 7f9425d23700 1 -- 172.21.15.134:0/2395760218 wait complete. 2024-01-21T11:47:57.345 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:57.343+0000 7f9425d23700 1 Processor -- start 2024-01-21T11:47:57.345 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:57.343+0000 7f9425d23700 1 -- start start 2024-01-21T11:47:57.345 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:57.343+0000 7f9425d23700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94200f6510 0x7f94201066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:47:57.345 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:57.343+0000 7f9425d23700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9420106bc0 con 0x7f94200f6510 2024-01-21T11:47:57.345 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:57.343+0000 7f941f7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94200f6510 0x7f94201066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:47:57.346 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:57.343+0000 7f941f7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94200f6510 0x7f94201066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:47714/0 (socket says 172.21.15.134:47714) 2024-01-21T11:47:57.346 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:57.343+0000 7f941f7fe700 1 -- 172.21.15.134:0/4263606898 learned_addr learned my addr 172.21.15.134:0/4263606898 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:47:57.346 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:57.343+0000 7f941f7fe700 1 -- 172.21.15.134:0/4263606898 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9410005040 con 0x7f94200f6510 2024-01-21T11:47:57.346 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:57.344+0000 7f941f7fe700 1 --2- 172.21.15.134:0/4263606898 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94200f6510 0x7f94201066d0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f9410013510 tx=0x7f941000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:47:57.346 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:57.344+0000 7f9407fff700 1 -- 172.21.15.134:0/4263606898 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9410009320 con 0x7f94200f6510 2024-01-21T11:47:57.346 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:57.344+0000 7f9425d23700 1 -- 172.21.15.134:0/4263606898 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9420106dc0 con 0x7f94200f6510 2024-01-21T11:47:57.346 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:57.344+0000 7f9407fff700 1 -- 172.21.15.134:0/4263606898 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9410013f00 con 0x7f94200f6510 2024-01-21T11:47:57.346 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:57.344+0000 7f9407fff700 1 -- 172.21.15.134:0/4263606898 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9410014d90 con 0x7f94200f6510 2024-01-21T11:47:57.347 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:57.344+0000 7f9425d23700 1 -- 172.21.15.134:0/4263606898 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9420107250 con 0x7f94200f6510 2024-01-21T11:47:57.347 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:57.345+0000 7f9407fff700 1 -- 172.21.15.134:0/4263606898 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f94100263a0 con 0x7f94200f6510 2024-01-21T11:47:57.347 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:57.345+0000 7f9407fff700 1 --2- 172.21.15.134:0/4263606898 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f940802ff60 0x7f9408032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:47:57.348 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:57.345+0000 7f9407fff700 1 -- 172.21.15.134:0/4263606898 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f94100410c0 con 0x7f94200f6510 2024-01-21T11:47:57.348 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:57.345+0000 7f9425d23700 1 -- 172.21.15.134:0/4263606898 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9420045ad0 con 0x7f94200f6510 2024-01-21T11:47:57.348 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:57.345+0000 7f941effd700 1 --2- 172.21.15.134:0/4263606898 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f940802ff60 0x7f9408032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:47:57.348 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:57.345+0000 7f941effd700 1 --2- 172.21.15.134:0/4263606898 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f940802ff60 0x7f9408032420 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f94140096a0 tx=0x7f9414006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:47:57.353 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:57.353+0000 7f9407fff700 1 -- 172.21.15.134:0/4263606898 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9410012920 con 0x7f94200f6510 2024-01-21T11:47:57.559 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:57.559+0000 7f9425d23700 1 -- 172.21.15.134:0/4263606898 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f94200fc050 con 0x7f94200f6510 2024-01-21T11:47:57.560 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:57.560+0000 7f9407fff700 1 -- 172.21.15.134:0/4263606898 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f941002b020 con 0x7f94200f6510 2024-01-21T11:47:57.560 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:47:57.560 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:47:57.562 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:57.561+0000 7f9425d23700 1 -- 172.21.15.134:0/4263606898 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f940802ff60 msgr2=0x7f9408032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:47:57.562 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:57.561+0000 7f9425d23700 1 --2- 172.21.15.134:0/4263606898 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f940802ff60 0x7f9408032420 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f94140096a0 tx=0x7f9414006b40).stop 2024-01-21T11:47:57.562 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:57.561+0000 7f9425d23700 1 -- 172.21.15.134:0/4263606898 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94200f6510 msgr2=0x7f94201066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:47:57.562 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:57.561+0000 7f9425d23700 1 --2- 172.21.15.134:0/4263606898 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94200f6510 0x7f94201066d0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f9410013510 tx=0x7f941000b040).stop 2024-01-21T11:47:57.562 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:57.561+0000 7f9425d23700 1 -- 172.21.15.134:0/4263606898 shutdown_connections 2024-01-21T11:47:57.562 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:57.561+0000 7f9425d23700 1 --2- 172.21.15.134:0/4263606898 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f940802ff60 0x7f9408032420 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:47:57.562 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:57.561+0000 7f9425d23700 1 --2- 172.21.15.134:0/4263606898 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94200f6510 0x7f94201066d0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:47:57.562 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:57.561+0000 7f9425d23700 1 -- 172.21.15.134:0/4263606898 >> 172.21.15.134:0/4263606898 conn(0x7f94200f1c70 msgr2=0x7f94200f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:47:57.562 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:57.561+0000 7f9425d23700 1 -- 172.21.15.134:0/4263606898 shutdown_connections 2024-01-21T11:47:57.562 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:47:57.561+0000 7f9425d23700 1 -- 172.21.15.134:0/4263606898 wait complete. 2024-01-21T11:47:57.563 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:47:57.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:57 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:47:55.858094+0000 mgr.smithi029.blhxxk (mgr.14164) 182 : cluster [DBG] pgmap v171: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:47:58.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:58 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:47:57.560802+0000 mon.smithi029 (mon.0) 253 : audit [DBG] from='client.? 172.21.15.134:0/4263606898' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:47:58.941 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:47:58.942 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:47:59.878 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:47:59 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:47:57.858316+0000 mgr.smithi029.blhxxk (mgr.14164) 183 : cluster [DBG] pgmap v172: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:48:00.076 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:00.075+0000 7fa8b52ff700 1 -- 172.21.15.134:0/811520730 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8b00601e0 msgr2=0x7fa8b00f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:48:00.077 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:00.075+0000 7fa8b52ff700 1 --2- 172.21.15.134:0/811520730 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8b00601e0 0x7fa8b00f8f20 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7fa8a4009fb0 tx=0x7fa8a4009b30).stop 2024-01-21T11:48:00.077 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:00.076+0000 7fa8b52ff700 1 -- 172.21.15.134:0/811520730 shutdown_connections 2024-01-21T11:48:00.077 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:00.076+0000 7fa8b52ff700 1 --2- 172.21.15.134:0/811520730 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8b00601e0 0x7fa8b00f8f20 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:48:00.077 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:00.076+0000 7fa8b52ff700 1 -- 172.21.15.134:0/811520730 >> 172.21.15.134:0/811520730 conn(0x7fa8b00f09a0 msgr2=0x7fa8b00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:48:00.077 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:00.076+0000 7fa8b52ff700 1 -- 172.21.15.134:0/811520730 shutdown_connections 2024-01-21T11:48:00.077 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:00.076+0000 7fa8b52ff700 1 -- 172.21.15.134:0/811520730 wait complete. 2024-01-21T11:48:00.077 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:00.076+0000 7fa8b52ff700 1 Processor -- start 2024-01-21T11:48:00.077 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:00.077+0000 7fa8b52ff700 1 -- start start 2024-01-21T11:48:00.078 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:00.077+0000 7fa8b52ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8b00601e0 0x7fa8b0106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:48:00.078 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:00.077+0000 7fa8b52ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b01069f0 con 0x7fa8b00601e0 2024-01-21T11:48:00.078 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:00.077+0000 7fa8affff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8b00601e0 0x7fa8b0106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:48:00.078 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:00.077+0000 7fa8affff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8b00601e0 0x7fa8b0106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:47722/0 (socket says 172.21.15.134:47722) 2024-01-21T11:48:00.078 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:00.077+0000 7fa8affff700 1 -- 172.21.15.134:0/1893161361 learned_addr learned my addr 172.21.15.134:0/1893161361 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:48:00.079 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:00.077+0000 7fa8affff700 1 -- 172.21.15.134:0/1893161361 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8a4005040 con 0x7fa8b00601e0 2024-01-21T11:48:00.079 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:00.077+0000 7fa8affff700 1 --2- 172.21.15.134:0/1893161361 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8b00601e0 0x7fa8b0106500 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7fa8a4013510 tx=0x7fa8a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:48:00.079 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:00.078+0000 7fa8acff9700 1 -- 172.21.15.134:0/1893161361 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa8a4009320 con 0x7fa8b00601e0 2024-01-21T11:48:00.079 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:00.078+0000 7fa8b52ff700 1 -- 172.21.15.134:0/1893161361 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8b0106bf0 con 0x7fa8b00601e0 2024-01-21T11:48:00.079 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:00.078+0000 7fa8b52ff700 1 -- 172.21.15.134:0/1893161361 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8b0107080 con 0x7fa8b00601e0 2024-01-21T11:48:00.079 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:00.078+0000 7fa8acff9700 1 -- 172.21.15.134:0/1893161361 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa8a4013f00 con 0x7fa8b00601e0 2024-01-21T11:48:00.079 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:00.078+0000 7fa8acff9700 1 -- 172.21.15.134:0/1893161361 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa8a4014d60 con 0x7fa8b00601e0 2024-01-21T11:48:00.080 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:00.079+0000 7fa8acff9700 1 -- 172.21.15.134:0/1893161361 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa8a4014420 con 0x7fa8b00601e0 2024-01-21T11:48:00.080 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:00.079+0000 7fa8b52ff700 1 -- 172.21.15.134:0/1893161361 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8b0045ad0 con 0x7fa8b00601e0 2024-01-21T11:48:00.080 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:00.079+0000 7fa8acff9700 1 --2- 172.21.15.134:0/1893161361 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fa89802ff60 0x7fa898032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:48:00.080 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:00.079+0000 7fa8acff9700 1 -- 172.21.15.134:0/1893161361 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa8a40412d0 con 0x7fa8b00601e0 2024-01-21T11:48:00.080 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:00.079+0000 7fa8af7fe700 1 --2- 172.21.15.134:0/1893161361 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fa89802ff60 0x7fa898032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:48:00.080 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:00.079+0000 7fa8af7fe700 1 --2- 172.21.15.134:0/1893161361 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fa89802ff60 0x7fa898032420 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7fa8a00096a0 tx=0x7fa8a0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:48:00.083 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:00.083+0000 7fa8acff9700 1 -- 172.21.15.134:0/1893161361 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa8a4012b20 con 0x7fa8b00601e0 2024-01-21T11:48:00.292 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:00.292+0000 7fa8b52ff700 1 -- 172.21.15.134:0/1893161361 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa8b0107f30 con 0x7fa8b00601e0 2024-01-21T11:48:00.293 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:00.293+0000 7fa8acff9700 1 -- 172.21.15.134:0/1893161361 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa8a4012b20 con 0x7fa8b00601e0 2024-01-21T11:48:00.293 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:48:00.293 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:48:00.295 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:00.294+0000 7fa8b52ff700 1 -- 172.21.15.134:0/1893161361 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fa89802ff60 msgr2=0x7fa898032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:48:00.295 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:00.294+0000 7fa8b52ff700 1 --2- 172.21.15.134:0/1893161361 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fa89802ff60 0x7fa898032420 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7fa8a00096a0 tx=0x7fa8a0006b40).stop 2024-01-21T11:48:00.295 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:00.294+0000 7fa8b52ff700 1 -- 172.21.15.134:0/1893161361 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8b00601e0 msgr2=0x7fa8b0106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:48:00.295 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:00.294+0000 7fa8b52ff700 1 --2- 172.21.15.134:0/1893161361 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8b00601e0 0x7fa8b0106500 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7fa8a4013510 tx=0x7fa8a400b040).stop 2024-01-21T11:48:00.295 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:00.294+0000 7fa8b52ff700 1 -- 172.21.15.134:0/1893161361 shutdown_connections 2024-01-21T11:48:00.295 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:00.294+0000 7fa8b52ff700 1 --2- 172.21.15.134:0/1893161361 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fa89802ff60 0x7fa898032420 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:48:00.295 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:00.294+0000 7fa8b52ff700 1 --2- 172.21.15.134:0/1893161361 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8b00601e0 0x7fa8b0106500 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:48:00.295 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:00.294+0000 7fa8b52ff700 1 -- 172.21.15.134:0/1893161361 >> 172.21.15.134:0/1893161361 conn(0x7fa8b00f09a0 msgr2=0x7fa8b00f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:48:00.295 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:00.295+0000 7fa8b52ff700 1 -- 172.21.15.134:0/1893161361 shutdown_connections 2024-01-21T11:48:00.295 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:00.295+0000 7fa8b52ff700 1 -- 172.21.15.134:0/1893161361 wait complete. 2024-01-21T11:48:00.296 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:48:00.844 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:48:00 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:48:00.293712+0000 mon.smithi029 (mon.0) 254 : audit [DBG] from='client.? 172.21.15.134:0/1893161361' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:48:01.667 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:48:01.667 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:48:01.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:48:01 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:47:59.858662+0000 mgr.smithi029.blhxxk (mgr.14164) 184 : cluster [DBG] pgmap v173: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:48:01.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:48:01 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:48:01.032955+0000 mon.smithi029 (mon.0) 255 : audit [INF] from='mgr.14164 172.21.15.29:0/3252518221' entity='mgr.smithi029.blhxxk' 2024-01-21T11:48:02.798 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:02.796+0000 7ffaccbea700 1 -- 172.21.15.134:0/2473012427 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffac80f6510 msgr2=0x7ffac80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:48:02.799 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:02.796+0000 7ffaccbea700 1 --2- 172.21.15.134:0/2473012427 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffac80f6510 0x7ffac80f6930 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7ffab8009fb0 tx=0x7ffab8009b30).stop 2024-01-21T11:48:02.799 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:02.797+0000 7ffaccbea700 1 -- 172.21.15.134:0/2473012427 shutdown_connections 2024-01-21T11:48:02.799 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:02.797+0000 7ffaccbea700 1 --2- 172.21.15.134:0/2473012427 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffac80f6510 0x7ffac80f6930 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:48:02.799 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:02.797+0000 7ffaccbea700 1 -- 172.21.15.134:0/2473012427 >> 172.21.15.134:0/2473012427 conn(0x7ffac80f1c70 msgr2=0x7ffac80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:48:02.799 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:02.797+0000 7ffaccbea700 1 -- 172.21.15.134:0/2473012427 shutdown_connections 2024-01-21T11:48:02.799 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:02.797+0000 7ffaccbea700 1 -- 172.21.15.134:0/2473012427 wait complete. 2024-01-21T11:48:02.799 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:02.797+0000 7ffaccbea700 1 Processor -- start 2024-01-21T11:48:02.799 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:02.797+0000 7ffaccbea700 1 -- start start 2024-01-21T11:48:02.799 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:02.797+0000 7ffaccbea700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffac80f6510 0x7ffac8106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:48:02.799 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:02.797+0000 7ffaccbea700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffac8106c20 con 0x7ffac80f6510 2024-01-21T11:48:02.799 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:02.798+0000 7ffac659c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffac80f6510 0x7ffac8106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:48:02.799 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:02.798+0000 7ffac659c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffac80f6510 0x7ffac8106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:47738/0 (socket says 172.21.15.134:47738) 2024-01-21T11:48:02.799 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:02.798+0000 7ffac659c700 1 -- 172.21.15.134:0/1776495644 learned_addr learned my addr 172.21.15.134:0/1776495644 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:48:02.800 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:02.798+0000 7ffac659c700 1 -- 172.21.15.134:0/1776495644 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffab8005040 con 0x7ffac80f6510 2024-01-21T11:48:02.800 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:02.798+0000 7ffac659c700 1 --2- 172.21.15.134:0/1776495644 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffac80f6510 0x7ffac8106730 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7ffab8013510 tx=0x7ffab800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:48:02.800 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:02.798+0000 7ffabeffd700 1 -- 172.21.15.134:0/1776495644 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffab8009320 con 0x7ffac80f6510 2024-01-21T11:48:02.800 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:02.798+0000 7ffaccbea700 1 -- 172.21.15.134:0/1776495644 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffac8106e20 con 0x7ffac80f6510 2024-01-21T11:48:02.800 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:02.798+0000 7ffabeffd700 1 -- 172.21.15.134:0/1776495644 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffab8013f00 con 0x7ffac80f6510 2024-01-21T11:48:02.800 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:02.798+0000 7ffaccbea700 1 -- 172.21.15.134:0/1776495644 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffac81072b0 con 0x7ffac80f6510 2024-01-21T11:48:02.800 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:02.798+0000 7ffabeffd700 1 -- 172.21.15.134:0/1776495644 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffab8014db0 con 0x7ffac80f6510 2024-01-21T11:48:02.801 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:02.799+0000 7ffabeffd700 1 -- 172.21.15.134:0/1776495644 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ffab80095f0 con 0x7ffac80f6510 2024-01-21T11:48:02.801 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:02.799+0000 7ffaccbea700 1 -- 172.21.15.134:0/1776495644 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffac8045ad0 con 0x7ffac80f6510 2024-01-21T11:48:02.801 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:02.799+0000 7ffabeffd700 1 --2- 172.21.15.134:0/1776495644 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7ffab402ff60 0x7ffab4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:48:02.801 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:02.799+0000 7ffabeffd700 1 -- 172.21.15.134:0/1776495644 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ffab8040f80 con 0x7ffac80f6510 2024-01-21T11:48:02.801 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:02.799+0000 7ffac5d9b700 1 --2- 172.21.15.134:0/1776495644 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7ffab402ff60 0x7ffab4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:48:02.801 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:02.800+0000 7ffac5d9b700 1 --2- 172.21.15.134:0/1776495644 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7ffab402ff60 0x7ffab4032420 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7ffab00096a0 tx=0x7ffab0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:48:02.804 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:02.804+0000 7ffabeffd700 1 -- 172.21.15.134:0/1776495644 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ffab801c790 con 0x7ffac80f6510 2024-01-21T11:48:02.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:48:02 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cephadm 2024-01-21T11:48:01.034312+0000 mgr.smithi029.blhxxk (mgr.14164) 185 : cephadm [INF] Deploying daemon node-exporter.smithi029 on smithi029 2024-01-21T11:48:02.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:48:02 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:48:01.942915+0000 mon.smithi029 (mon.0) 256 : audit [INF] from='mgr.14164 172.21.15.29:0/3252518221' entity='mgr.smithi029.blhxxk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi029.blhxxk/mirror_snapshot_schedule"}]: dispatch 2024-01-21T11:48:02.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:48:02 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:48:01.944764+0000 mon.smithi029 (mon.0) 257 : audit [INF] from='mgr.14164 172.21.15.29:0/3252518221' entity='mgr.smithi029.blhxxk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi029.blhxxk/trash_purge_schedule"}]: dispatch 2024-01-21T11:48:03.013 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:03.013+0000 7ffaccbea700 1 -- 172.21.15.134:0/1776495644 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ffac80fbf60 con 0x7ffac80f6510 2024-01-21T11:48:03.014 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:03.013+0000 7ffabeffd700 1 -- 172.21.15.134:0/1776495644 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ffab801c790 con 0x7ffac80f6510 2024-01-21T11:48:03.014 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:48:03.014 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:48:03.016 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:03.015+0000 7ffaccbea700 1 -- 172.21.15.134:0/1776495644 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7ffab402ff60 msgr2=0x7ffab4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:48:03.016 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:03.015+0000 7ffaccbea700 1 --2- 172.21.15.134:0/1776495644 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7ffab402ff60 0x7ffab4032420 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7ffab00096a0 tx=0x7ffab0006b40).stop 2024-01-21T11:48:03.016 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:03.015+0000 7ffaccbea700 1 -- 172.21.15.134:0/1776495644 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffac80f6510 msgr2=0x7ffac8106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:48:03.016 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:03.015+0000 7ffaccbea700 1 --2- 172.21.15.134:0/1776495644 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffac80f6510 0x7ffac8106730 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7ffab8013510 tx=0x7ffab800b040).stop 2024-01-21T11:48:03.016 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:03.015+0000 7ffaccbea700 1 -- 172.21.15.134:0/1776495644 shutdown_connections 2024-01-21T11:48:03.016 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:03.015+0000 7ffaccbea700 1 --2- 172.21.15.134:0/1776495644 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7ffab402ff60 0x7ffab4032420 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:48:03.016 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:03.015+0000 7ffaccbea700 1 --2- 172.21.15.134:0/1776495644 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffac80f6510 0x7ffac8106730 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:48:03.016 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:03.015+0000 7ffaccbea700 1 -- 172.21.15.134:0/1776495644 >> 172.21.15.134:0/1776495644 conn(0x7ffac80f1c70 msgr2=0x7ffac80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:48:03.016 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:03.015+0000 7ffaccbea700 1 -- 172.21.15.134:0/1776495644 shutdown_connections 2024-01-21T11:48:03.016 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:03.015+0000 7ffaccbea700 1 -- 172.21.15.134:0/1776495644 wait complete. 2024-01-21T11:48:03.017 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:48:03.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:48:03 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:48:01.859012+0000 mgr.smithi029.blhxxk (mgr.14164) 186 : cluster [DBG] pgmap v174: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:48:03.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:48:03 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:48:03.014670+0000 mon.smithi029 (mon.0) 258 : audit [DBG] from='client.? 172.21.15.134:0/1776495644' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:48:04.413 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:48:04.413 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:48:05.614 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:05.612+0000 7fe367f54700 1 -- 172.21.15.134:0/704841318 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3600f5410 msgr2=0x7fe3600f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:48:05.614 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:05.612+0000 7fe367f54700 1 --2- 172.21.15.134:0/704841318 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3600f5410 0x7fe3600f5830 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7fe350009fb0 tx=0x7fe350009b30).stop 2024-01-21T11:48:05.614 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:05.612+0000 7fe367f54700 1 -- 172.21.15.134:0/704841318 shutdown_connections 2024-01-21T11:48:05.614 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:05.612+0000 7fe367f54700 1 --2- 172.21.15.134:0/704841318 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3600f5410 0x7fe3600f5830 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:48:05.614 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:05.612+0000 7fe367f54700 1 -- 172.21.15.134:0/704841318 >> 172.21.15.134:0/704841318 conn(0x7fe3600f0b70 msgr2=0x7fe3600f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:48:05.614 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:05.612+0000 7fe367f54700 1 -- 172.21.15.134:0/704841318 shutdown_connections 2024-01-21T11:48:05.614 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:05.612+0000 7fe367f54700 1 -- 172.21.15.134:0/704841318 wait complete. 2024-01-21T11:48:05.614 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:05.613+0000 7fe367f54700 1 Processor -- start 2024-01-21T11:48:05.614 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:05.613+0000 7fe367f54700 1 -- start start 2024-01-21T11:48:05.614 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:05.613+0000 7fe367f54700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3600f5410 0x7fe360109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:48:05.614 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:05.613+0000 7fe367f54700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe360060220 con 0x7fe3600f5410 2024-01-21T11:48:05.615 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:05.613+0000 7fe365cf0700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3600f5410 0x7fe360109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:48:05.615 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:05.613+0000 7fe365cf0700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3600f5410 0x7fe360109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:50594/0 (socket says 172.21.15.134:50594) 2024-01-21T11:48:05.615 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:05.613+0000 7fe365cf0700 1 -- 172.21.15.134:0/2168759048 learned_addr learned my addr 172.21.15.134:0/2168759048 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:48:05.615 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:05.613+0000 7fe365cf0700 1 -- 172.21.15.134:0/2168759048 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe350005040 con 0x7fe3600f5410 2024-01-21T11:48:05.615 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:05.613+0000 7fe365cf0700 1 --2- 172.21.15.134:0/2168759048 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3600f5410 0x7fe360109450 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7fe350013510 tx=0x7fe35000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:48:05.615 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:05.614+0000 7fe3567fc700 1 -- 172.21.15.134:0/2168759048 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe350009320 con 0x7fe3600f5410 2024-01-21T11:48:05.616 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:05.614+0000 7fe367f54700 1 -- 172.21.15.134:0/2168759048 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe360060420 con 0x7fe3600f5410 2024-01-21T11:48:05.616 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:05.614+0000 7fe3567fc700 1 -- 172.21.15.134:0/2168759048 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe350013f00 con 0x7fe3600f5410 2024-01-21T11:48:05.616 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:05.614+0000 7fe3567fc700 1 -- 172.21.15.134:0/2168759048 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe350014d90 con 0x7fe3600f5410 2024-01-21T11:48:05.616 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:05.614+0000 7fe367f54700 1 -- 172.21.15.134:0/2168759048 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe360109b00 con 0x7fe3600f5410 2024-01-21T11:48:05.616 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:05.615+0000 7fe3567fc700 1 -- 172.21.15.134:0/2168759048 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe3500095f0 con 0x7fe3600f5410 2024-01-21T11:48:05.616 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:05.615+0000 7fe3567fc700 1 --2- 172.21.15.134:0/2168759048 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fe34c02ff60 0x7fe34c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:48:05.616 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:05.615+0000 7fe3567fc700 1 -- 172.21.15.134:0/2168759048 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe350041000 con 0x7fe3600f5410 2024-01-21T11:48:05.616 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:05.615+0000 7fe3654ef700 1 --2- 172.21.15.134:0/2168759048 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fe34c02ff60 0x7fe34c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:48:05.617 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:05.615+0000 7fe367f54700 1 -- 172.21.15.134:0/2168759048 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe360059100 con 0x7fe3600f5410 2024-01-21T11:48:05.617 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:05.616+0000 7fe3654ef700 1 --2- 172.21.15.134:0/2168759048 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fe34c02ff60 0x7fe34c032420 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7fe35c0096a0 tx=0x7fe35c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:48:05.620 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:05.620+0000 7fe3567fc700 1 -- 172.21.15.134:0/2168759048 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe350014420 con 0x7fe3600f5410 2024-01-21T11:48:05.824 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:05.824+0000 7fe367f54700 1 -- 172.21.15.134:0/2168759048 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe360045ad0 con 0x7fe3600f5410 2024-01-21T11:48:05.825 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:05.825+0000 7fe3567fc700 1 -- 172.21.15.134:0/2168759048 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe35002b020 con 0x7fe3600f5410 2024-01-21T11:48:05.825 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:48:05.825 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:48:05.827 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:05.826+0000 7fe367f54700 1 -- 172.21.15.134:0/2168759048 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fe34c02ff60 msgr2=0x7fe34c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:48:05.827 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:05.826+0000 7fe367f54700 1 --2- 172.21.15.134:0/2168759048 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fe34c02ff60 0x7fe34c032420 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7fe35c0096a0 tx=0x7fe35c006b40).stop 2024-01-21T11:48:05.827 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:05.826+0000 7fe367f54700 1 -- 172.21.15.134:0/2168759048 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3600f5410 msgr2=0x7fe360109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:48:05.827 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:05.826+0000 7fe367f54700 1 --2- 172.21.15.134:0/2168759048 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3600f5410 0x7fe360109450 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7fe350013510 tx=0x7fe35000b040).stop 2024-01-21T11:48:05.827 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:05.827+0000 7fe367f54700 1 -- 172.21.15.134:0/2168759048 shutdown_connections 2024-01-21T11:48:05.827 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:05.827+0000 7fe367f54700 1 --2- 172.21.15.134:0/2168759048 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fe34c02ff60 0x7fe34c032420 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:48:05.828 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:05.827+0000 7fe367f54700 1 --2- 172.21.15.134:0/2168759048 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3600f5410 0x7fe360109450 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:48:05.828 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:05.827+0000 7fe367f54700 1 -- 172.21.15.134:0/2168759048 >> 172.21.15.134:0/2168759048 conn(0x7fe3600f0b70 msgr2=0x7fe3600f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:48:05.828 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:05.827+0000 7fe367f54700 1 -- 172.21.15.134:0/2168759048 shutdown_connections 2024-01-21T11:48:05.828 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:05.827+0000 7fe367f54700 1 -- 172.21.15.134:0/2168759048 wait complete. 2024-01-21T11:48:05.828 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:48:05.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:48:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:48:03.859334+0000 mgr.smithi029.blhxxk (mgr.14164) 187 : cluster [DBG] pgmap v175: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:48:06.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:48:06 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:48:05.826113+0000 mon.smithi029 (mon.0) 259 : audit [DBG] from='client.? 172.21.15.134:0/2168759048' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:48:07.229 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:48:07.229 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:48:07.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:48:07 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:48:05.859675+0000 mgr.smithi029.blhxxk (mgr.14164) 188 : cluster [DBG] pgmap v176: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:48:08.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:08.330+0000 7f344f3a2700 1 -- 172.21.15.134:0/1512787184 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34480f62a0 msgr2=0x7f34480f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:48:08.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:08.330+0000 7f344f3a2700 1 --2- 172.21.15.134:0/1512787184 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34480f62a0 0x7f34480f66c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7f3438009fb0 tx=0x7f3438009b30).stop 2024-01-21T11:48:08.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:08.330+0000 7f344f3a2700 1 -- 172.21.15.134:0/1512787184 shutdown_connections 2024-01-21T11:48:08.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:08.330+0000 7f344f3a2700 1 --2- 172.21.15.134:0/1512787184 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34480f62a0 0x7f34480f66c0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:48:08.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:08.330+0000 7f344f3a2700 1 -- 172.21.15.134:0/1512787184 >> 172.21.15.134:0/1512787184 conn(0x7f34480f1aa0 msgr2=0x7f34480f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:48:08.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:08.330+0000 7f344f3a2700 1 -- 172.21.15.134:0/1512787184 shutdown_connections 2024-01-21T11:48:08.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:08.330+0000 7f344f3a2700 1 -- 172.21.15.134:0/1512787184 wait complete. 2024-01-21T11:48:08.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:08.330+0000 7f344f3a2700 1 Processor -- start 2024-01-21T11:48:08.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:08.330+0000 7f344f3a2700 1 -- start start 2024-01-21T11:48:08.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:08.330+0000 7f344f3a2700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34480f62a0 0x7f34481064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:48:08.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:08.330+0000 7f344f3a2700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34481069a0 con 0x7f34480f62a0 2024-01-21T11:48:08.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:08.331+0000 7f344e3a0700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34480f62a0 0x7f34481064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:48:08.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:08.331+0000 7f344e3a0700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34480f62a0 0x7f34481064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:50622/0 (socket says 172.21.15.134:50622) 2024-01-21T11:48:08.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:08.331+0000 7f344e3a0700 1 -- 172.21.15.134:0/525612059 learned_addr learned my addr 172.21.15.134:0/525612059 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:48:08.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:08.331+0000 7f344e3a0700 1 -- 172.21.15.134:0/525612059 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3438005040 con 0x7f34480f62a0 2024-01-21T11:48:08.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:08.331+0000 7f344e3a0700 1 --2- 172.21.15.134:0/525612059 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34480f62a0 0x7f34481064b0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7f3438013510 tx=0x7f343800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:48:08.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:08.331+0000 7f343effd700 1 -- 172.21.15.134:0/525612059 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3438009320 con 0x7f34480f62a0 2024-01-21T11:48:08.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:08.331+0000 7f344f3a2700 1 -- 172.21.15.134:0/525612059 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3448106ba0 con 0x7f34480f62a0 2024-01-21T11:48:08.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:08.331+0000 7f343effd700 1 -- 172.21.15.134:0/525612059 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3438013f00 con 0x7f34480f62a0 2024-01-21T11:48:08.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:08.331+0000 7f343effd700 1 -- 172.21.15.134:0/525612059 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3438014dc0 con 0x7f34480f62a0 2024-01-21T11:48:08.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:08.331+0000 7f344f3a2700 1 -- 172.21.15.134:0/525612059 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3448107030 con 0x7f34480f62a0 2024-01-21T11:48:08.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:08.332+0000 7f343effd700 1 -- 172.21.15.134:0/525612059 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f34380095f0 con 0x7f34480f62a0 2024-01-21T11:48:08.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:08.332+0000 7f344f3a2700 1 -- 172.21.15.134:0/525612059 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3448045ad0 con 0x7f34480f62a0 2024-01-21T11:48:08.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:08.332+0000 7f343effd700 1 --2- 172.21.15.134:0/525612059 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f343402ff60 0x7f3434032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:48:08.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:08.332+0000 7f343effd700 1 -- 172.21.15.134:0/525612059 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3438041040 con 0x7f34480f62a0 2024-01-21T11:48:08.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:08.332+0000 7f344db9f700 1 --2- 172.21.15.134:0/525612059 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f343402ff60 0x7f3434032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:48:08.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:08.333+0000 7f344db9f700 1 --2- 172.21.15.134:0/525612059 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f343402ff60 0x7f3434032420 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f34440096a0 tx=0x7f3444006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:48:08.337 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:08.338+0000 7f343effd700 1 -- 172.21.15.134:0/525612059 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3438014420 con 0x7f34480f62a0 2024-01-21T11:48:08.543 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:08.542+0000 7f344f3a2700 1 -- 172.21.15.134:0/525612059 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3448107ee0 con 0x7f34480f62a0 2024-01-21T11:48:08.543 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:08.543+0000 7f343effd700 1 -- 172.21.15.134:0/525612059 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3438010230 con 0x7f34480f62a0 2024-01-21T11:48:08.544 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:48:08.544 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:48:08.545 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:08.545+0000 7f344f3a2700 1 -- 172.21.15.134:0/525612059 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f343402ff60 msgr2=0x7f3434032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:48:08.545 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:08.545+0000 7f344f3a2700 1 --2- 172.21.15.134:0/525612059 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f343402ff60 0x7f3434032420 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f34440096a0 tx=0x7f3444006b40).stop 2024-01-21T11:48:08.545 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:08.545+0000 7f344f3a2700 1 -- 172.21.15.134:0/525612059 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34480f62a0 msgr2=0x7f34481064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:48:08.545 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:08.545+0000 7f344f3a2700 1 --2- 172.21.15.134:0/525612059 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34480f62a0 0x7f34481064b0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7f3438013510 tx=0x7f343800b040).stop 2024-01-21T11:48:08.545 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:08.545+0000 7f344f3a2700 1 -- 172.21.15.134:0/525612059 shutdown_connections 2024-01-21T11:48:08.545 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:08.545+0000 7f344f3a2700 1 --2- 172.21.15.134:0/525612059 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f343402ff60 0x7f3434032420 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:48:08.546 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:08.545+0000 7f344f3a2700 1 --2- 172.21.15.134:0/525612059 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34480f62a0 0x7f34481064b0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:48:08.546 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:08.545+0000 7f344f3a2700 1 -- 172.21.15.134:0/525612059 >> 172.21.15.134:0/525612059 conn(0x7f34480f1aa0 msgr2=0x7f34480f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:48:08.546 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:08.545+0000 7f344f3a2700 1 -- 172.21.15.134:0/525612059 shutdown_connections 2024-01-21T11:48:08.546 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:08.545+0000 7f344f3a2700 1 -- 172.21.15.134:0/525612059 wait complete. 2024-01-21T11:48:08.546 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:48:08.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:48:08 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:48:08.544340+0000 mon.smithi029 (mon.0) 260 : audit [DBG] from='client.? 172.21.15.134:0/525612059' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:48:09.913 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:48:09.914 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:48:09.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:48:09 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:48:07.859955+0000 mgr.smithi029.blhxxk (mgr.14164) 189 : cluster [DBG] pgmap v177: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:48:11.070 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:11.069+0000 7fec0af50700 1 -- 172.21.15.134:0/1765696747 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec040f6510 msgr2=0x7fec040f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:48:11.070 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:11.069+0000 7fec0af50700 1 --2- 172.21.15.134:0/1765696747 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec040f6510 0x7fec040f6930 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7febf8009fb0 tx=0x7febf8009b30).stop 2024-01-21T11:48:11.070 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:11.069+0000 7fec0af50700 1 -- 172.21.15.134:0/1765696747 shutdown_connections 2024-01-21T11:48:11.070 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:11.069+0000 7fec0af50700 1 --2- 172.21.15.134:0/1765696747 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec040f6510 0x7fec040f6930 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:48:11.070 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:11.069+0000 7fec0af50700 1 -- 172.21.15.134:0/1765696747 >> 172.21.15.134:0/1765696747 conn(0x7fec040f1c70 msgr2=0x7fec040f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:48:11.070 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:11.069+0000 7fec0af50700 1 -- 172.21.15.134:0/1765696747 shutdown_connections 2024-01-21T11:48:11.070 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:11.069+0000 7fec0af50700 1 -- 172.21.15.134:0/1765696747 wait complete. 2024-01-21T11:48:11.071 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:11.069+0000 7fec0af50700 1 Processor -- start 2024-01-21T11:48:11.071 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:11.070+0000 7fec0af50700 1 -- start start 2024-01-21T11:48:11.071 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:11.070+0000 7fec0af50700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec040f6510 0x7fec04106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:48:11.071 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:11.070+0000 7fec0af50700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec04106c20 con 0x7fec040f6510 2024-01-21T11:48:11.071 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:11.070+0000 7fec08cec700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec040f6510 0x7fec04106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:48:11.071 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:11.070+0000 7fec08cec700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec040f6510 0x7fec04106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:50644/0 (socket says 172.21.15.134:50644) 2024-01-21T11:48:11.071 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:11.070+0000 7fec08cec700 1 -- 172.21.15.134:0/271484492 learned_addr learned my addr 172.21.15.134:0/271484492 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:48:11.071 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:11.070+0000 7fec08cec700 1 -- 172.21.15.134:0/271484492 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febf8005040 con 0x7fec040f6510 2024-01-21T11:48:11.071 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:11.070+0000 7fec08cec700 1 --2- 172.21.15.134:0/271484492 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec040f6510 0x7fec04106730 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7febf8013510 tx=0x7febf800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:48:11.071 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:11.071+0000 7fec017fa700 1 -- 172.21.15.134:0/271484492 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7febf8009320 con 0x7fec040f6510 2024-01-21T11:48:11.071 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:11.071+0000 7fec0af50700 1 -- 172.21.15.134:0/271484492 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec04106e20 con 0x7fec040f6510 2024-01-21T11:48:11.071 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:11.071+0000 7fec017fa700 1 -- 172.21.15.134:0/271484492 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7febf8013f00 con 0x7fec040f6510 2024-01-21T11:48:11.071 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:11.071+0000 7fec017fa700 1 -- 172.21.15.134:0/271484492 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7febf8014d90 con 0x7fec040f6510 2024-01-21T11:48:11.071 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:11.071+0000 7fec0af50700 1 -- 172.21.15.134:0/271484492 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec041072b0 con 0x7fec040f6510 2024-01-21T11:48:11.072 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:11.071+0000 7fec017fa700 1 -- 172.21.15.134:0/271484492 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7febf80263a0 con 0x7fec040f6510 2024-01-21T11:48:11.072 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:11.072+0000 7fec017fa700 1 --2- 172.21.15.134:0/271484492 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7febf402ff60 0x7febf4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:48:11.072 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:11.072+0000 7fec017fa700 1 -- 172.21.15.134:0/271484492 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7febf8041000 con 0x7fec040f6510 2024-01-21T11:48:11.073 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:11.072+0000 7fec0af50700 1 -- 172.21.15.134:0/271484492 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec04045ad0 con 0x7fec040f6510 2024-01-21T11:48:11.073 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:11.072+0000 7fec03fff700 1 --2- 172.21.15.134:0/271484492 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7febf402ff60 0x7febf4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:48:11.073 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:11.072+0000 7fec03fff700 1 --2- 172.21.15.134:0/271484492 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7febf402ff60 0x7febf4032420 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7febf00096a0 tx=0x7febf0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:48:11.076 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:11.077+0000 7fec017fa700 1 -- 172.21.15.134:0/271484492 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7febf8012920 con 0x7fec040f6510 2024-01-21T11:48:11.283 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:11.284+0000 7fec0af50700 1 -- 172.21.15.134:0/271484492 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fec040fc050 con 0x7fec040f6510 2024-01-21T11:48:11.284 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:11.285+0000 7fec017fa700 1 -- 172.21.15.134:0/271484492 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7febf802b020 con 0x7fec040f6510 2024-01-21T11:48:11.284 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:48:11.284 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:48:11.285 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:11.286+0000 7fec0af50700 1 -- 172.21.15.134:0/271484492 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7febf402ff60 msgr2=0x7febf4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:48:11.286 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:11.286+0000 7fec0af50700 1 --2- 172.21.15.134:0/271484492 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7febf402ff60 0x7febf4032420 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7febf00096a0 tx=0x7febf0006b40).stop 2024-01-21T11:48:11.286 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:11.286+0000 7fec0af50700 1 -- 172.21.15.134:0/271484492 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec040f6510 msgr2=0x7fec04106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:48:11.286 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:11.286+0000 7fec0af50700 1 --2- 172.21.15.134:0/271484492 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec040f6510 0x7fec04106730 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7febf8013510 tx=0x7febf800b040).stop 2024-01-21T11:48:11.286 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:11.286+0000 7fec0af50700 1 -- 172.21.15.134:0/271484492 shutdown_connections 2024-01-21T11:48:11.286 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:11.286+0000 7fec0af50700 1 --2- 172.21.15.134:0/271484492 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7febf402ff60 0x7febf4032420 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:48:11.286 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:11.286+0000 7fec0af50700 1 --2- 172.21.15.134:0/271484492 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec040f6510 0x7fec04106730 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:48:11.286 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:11.286+0000 7fec0af50700 1 -- 172.21.15.134:0/271484492 >> 172.21.15.134:0/271484492 conn(0x7fec040f1c70 msgr2=0x7fec040f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:48:11.286 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:11.286+0000 7fec0af50700 1 -- 172.21.15.134:0/271484492 shutdown_connections 2024-01-21T11:48:11.286 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:11.286+0000 7fec0af50700 1 -- 172.21.15.134:0/271484492 wait complete. 2024-01-21T11:48:11.286 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:48:11.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:48:11 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:48:09.860299+0000 mgr.smithi029.blhxxk (mgr.14164) 190 : cluster [DBG] pgmap v178: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:48:11.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:48:11 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:48:11.284950+0000 mon.smithi029 (mon.0) 261 : audit [DBG] from='client.? 172.21.15.134:0/271484492' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:48:12.662 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:48:12.662 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:48:13.757 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:13.755+0000 7f2adf8e9700 1 -- 172.21.15.134:0/1191277877 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ad80601e0 msgr2=0x7f2ad80f9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:48:13.757 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:13.755+0000 7f2adf8e9700 1 --2- 172.21.15.134:0/1191277877 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ad80601e0 0x7f2ad80f9190 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7f2ac8009fb0 tx=0x7f2ac8009b30).stop 2024-01-21T11:48:13.757 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:13.755+0000 7f2adf8e9700 1 -- 172.21.15.134:0/1191277877 shutdown_connections 2024-01-21T11:48:13.757 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:13.755+0000 7f2adf8e9700 1 --2- 172.21.15.134:0/1191277877 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ad80601e0 0x7f2ad80f9190 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:48:13.757 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:13.755+0000 7f2adf8e9700 1 -- 172.21.15.134:0/1191277877 >> 172.21.15.134:0/1191277877 conn(0x7f2ad80f0b70 msgr2=0x7f2ad80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:48:13.757 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:13.755+0000 7f2adf8e9700 1 -- 172.21.15.134:0/1191277877 shutdown_connections 2024-01-21T11:48:13.757 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:13.755+0000 7f2adf8e9700 1 -- 172.21.15.134:0/1191277877 wait complete. 2024-01-21T11:48:13.757 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:13.755+0000 7f2adf8e9700 1 Processor -- start 2024-01-21T11:48:13.757 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:13.755+0000 7f2adf8e9700 1 -- start start 2024-01-21T11:48:13.757 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:13.756+0000 7f2adf8e9700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ad80601e0 0x7f2ad81066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:48:13.757 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:13.756+0000 7f2adf8e9700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ad8106bd0 con 0x7f2ad80601e0 2024-01-21T11:48:13.757 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:13.756+0000 7f2add685700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ad80601e0 0x7f2ad81066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:48:13.758 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:13.756+0000 7f2add685700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ad80601e0 0x7f2ad81066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:50658/0 (socket says 172.21.15.134:50658) 2024-01-21T11:48:13.758 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:13.756+0000 7f2add685700 1 -- 172.21.15.134:0/3881757506 learned_addr learned my addr 172.21.15.134:0/3881757506 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:48:13.758 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:13.756+0000 7f2add685700 1 -- 172.21.15.134:0/3881757506 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ac8005040 con 0x7f2ad80601e0 2024-01-21T11:48:13.758 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:13.756+0000 7f2add685700 1 --2- 172.21.15.134:0/3881757506 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ad80601e0 0x7f2ad81066e0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7f2ac8004dd0 tx=0x7f2ac8009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:48:13.758 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:13.756+0000 7f2acdffb700 1 -- 172.21.15.134:0/3881757506 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2ac8018070 con 0x7f2ad80601e0 2024-01-21T11:48:13.758 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:13.756+0000 7f2adf8e9700 1 -- 172.21.15.134:0/3881757506 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ad8106dd0 con 0x7f2ad80601e0 2024-01-21T11:48:13.758 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:13.756+0000 7f2acdffb700 1 -- 172.21.15.134:0/3881757506 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2ac8013410 con 0x7f2ad80601e0 2024-01-21T11:48:13.758 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:13.756+0000 7f2acdffb700 1 -- 172.21.15.134:0/3881757506 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2ac801c3c0 con 0x7f2ad80601e0 2024-01-21T11:48:13.758 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:13.757+0000 7f2adf8e9700 1 -- 172.21.15.134:0/3881757506 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ad8107260 con 0x7f2ad80601e0 2024-01-21T11:48:13.759 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:13.757+0000 7f2acdffb700 1 -- 172.21.15.134:0/3881757506 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2ac801c630 con 0x7f2ad80601e0 2024-01-21T11:48:13.759 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:13.758+0000 7f2acdffb700 1 --2- 172.21.15.134:0/3881757506 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f2ac402ff60 0x7f2ac4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:48:13.759 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:13.758+0000 7f2adf8e9700 1 -- 172.21.15.134:0/3881757506 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ad8045ad0 con 0x7f2ad80601e0 2024-01-21T11:48:13.759 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:13.758+0000 7f2acdffb700 1 -- 172.21.15.134:0/3881757506 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2ac8041710 con 0x7f2ad80601e0 2024-01-21T11:48:13.759 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:13.758+0000 7f2adce84700 1 --2- 172.21.15.134:0/3881757506 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f2ac402ff60 0x7f2ac4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:48:13.759 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:13.758+0000 7f2adce84700 1 --2- 172.21.15.134:0/3881757506 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f2ac402ff60 0x7f2ac4032420 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f2ad40096a0 tx=0x7f2ad4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:48:13.762 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:13.762+0000 7f2acdffb700 1 -- 172.21.15.134:0/3881757506 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2ac801b550 con 0x7f2ad80601e0 2024-01-21T11:48:13.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:48:13 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:48:11.860695+0000 mgr.smithi029.blhxxk (mgr.14164) 191 : cluster [DBG] pgmap v179: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:48:13.970 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:13.970+0000 7f2adf8e9700 1 -- 172.21.15.134:0/3881757506 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2ad81080b0 con 0x7f2ad80601e0 2024-01-21T11:48:13.971 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:13.971+0000 7f2acdffb700 1 -- 172.21.15.134:0/3881757506 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2ac801b370 con 0x7f2ad80601e0 2024-01-21T11:48:13.971 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:48:13.971 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:48:13.973 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:13.972+0000 7f2adf8e9700 1 -- 172.21.15.134:0/3881757506 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f2ac402ff60 msgr2=0x7f2ac4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:48:13.973 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:13.972+0000 7f2adf8e9700 1 --2- 172.21.15.134:0/3881757506 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f2ac402ff60 0x7f2ac4032420 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f2ad40096a0 tx=0x7f2ad4006b40).stop 2024-01-21T11:48:13.973 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:13.972+0000 7f2adf8e9700 1 -- 172.21.15.134:0/3881757506 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ad80601e0 msgr2=0x7f2ad81066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:48:13.973 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:13.972+0000 7f2adf8e9700 1 --2- 172.21.15.134:0/3881757506 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ad80601e0 0x7f2ad81066e0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7f2ac8004dd0 tx=0x7f2ac8009b30).stop 2024-01-21T11:48:13.973 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:13.972+0000 7f2adf8e9700 1 -- 172.21.15.134:0/3881757506 shutdown_connections 2024-01-21T11:48:13.973 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:13.972+0000 7f2adf8e9700 1 --2- 172.21.15.134:0/3881757506 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f2ac402ff60 0x7f2ac4032420 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:48:13.973 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:13.972+0000 7f2adf8e9700 1 --2- 172.21.15.134:0/3881757506 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ad80601e0 0x7f2ad81066e0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:48:13.973 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:13.972+0000 7f2adf8e9700 1 -- 172.21.15.134:0/3881757506 >> 172.21.15.134:0/3881757506 conn(0x7f2ad80f0b70 msgr2=0x7f2ad80f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:48:13.973 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:13.972+0000 7f2adf8e9700 1 -- 172.21.15.134:0/3881757506 shutdown_connections 2024-01-21T11:48:13.973 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:13.973+0000 7f2adf8e9700 1 -- 172.21.15.134:0/3881757506 wait complete. 2024-01-21T11:48:13.974 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:48:14.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:48:14 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:48:13.972058+0000 mon.smithi029 (mon.0) 262 : audit [DBG] from='client.? 172.21.15.134:0/3881757506' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:48:15.344 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:48:15.345 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:48:15.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:48:15 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:48:13.861043+0000 mgr.smithi029.blhxxk (mgr.14164) 192 : cluster [DBG] pgmap v180: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:48:16.482 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:16.481+0000 7f1d5f0d1700 1 -- 172.21.15.134:0/2583246356 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d580f6510 msgr2=0x7f1d580f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:48:16.482 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:16.481+0000 7f1d5f0d1700 1 --2- 172.21.15.134:0/2583246356 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d580f6510 0x7f1d580f6930 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7f1d4c009fb0 tx=0x7f1d4c009b30).stop 2024-01-21T11:48:16.482 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:16.481+0000 7f1d5f0d1700 1 -- 172.21.15.134:0/2583246356 shutdown_connections 2024-01-21T11:48:16.482 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:16.481+0000 7f1d5f0d1700 1 --2- 172.21.15.134:0/2583246356 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d580f6510 0x7f1d580f6930 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:48:16.482 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:16.481+0000 7f1d5f0d1700 1 -- 172.21.15.134:0/2583246356 >> 172.21.15.134:0/2583246356 conn(0x7f1d580f1c70 msgr2=0x7f1d580f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:48:16.482 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:16.481+0000 7f1d5f0d1700 1 -- 172.21.15.134:0/2583246356 shutdown_connections 2024-01-21T11:48:16.482 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:16.481+0000 7f1d5f0d1700 1 -- 172.21.15.134:0/2583246356 wait complete. 2024-01-21T11:48:16.482 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:16.481+0000 7f1d5f0d1700 1 Processor -- start 2024-01-21T11:48:16.482 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:16.481+0000 7f1d5f0d1700 1 -- start start 2024-01-21T11:48:16.482 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:16.482+0000 7f1d5f0d1700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d580f6510 0x7f1d58106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:48:16.482 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:16.482+0000 7f1d5f0d1700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d58106c20 con 0x7f1d580f6510 2024-01-21T11:48:16.484 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:16.482+0000 7f1d5ce6d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d580f6510 0x7f1d58106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:48:16.484 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:16.482+0000 7f1d5ce6d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d580f6510 0x7f1d58106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:35194/0 (socket says 172.21.15.134:35194) 2024-01-21T11:48:16.484 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:16.482+0000 7f1d5ce6d700 1 -- 172.21.15.134:0/3389743427 learned_addr learned my addr 172.21.15.134:0/3389743427 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:48:16.484 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:16.482+0000 7f1d5ce6d700 1 -- 172.21.15.134:0/3389743427 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d4c005040 con 0x7f1d580f6510 2024-01-21T11:48:16.485 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:16.482+0000 7f1d5ce6d700 1 --2- 172.21.15.134:0/3389743427 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d580f6510 0x7f1d58106730 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7f1d4c013510 tx=0x7f1d4c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:48:16.485 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:16.483+0000 7f1d557fa700 1 -- 172.21.15.134:0/3389743427 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1d4c009320 con 0x7f1d580f6510 2024-01-21T11:48:16.485 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:16.483+0000 7f1d5f0d1700 1 -- 172.21.15.134:0/3389743427 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d58106e20 con 0x7f1d580f6510 2024-01-21T11:48:16.485 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:16.483+0000 7f1d557fa700 1 -- 172.21.15.134:0/3389743427 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1d4c013f00 con 0x7f1d580f6510 2024-01-21T11:48:16.485 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:16.483+0000 7f1d557fa700 1 -- 172.21.15.134:0/3389743427 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1d4c014d90 con 0x7f1d580f6510 2024-01-21T11:48:16.485 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:16.483+0000 7f1d5f0d1700 1 -- 172.21.15.134:0/3389743427 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d581072b0 con 0x7f1d580f6510 2024-01-21T11:48:16.485 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:16.483+0000 7f1d5f0d1700 1 -- 172.21.15.134:0/3389743427 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d58045ad0 con 0x7f1d580f6510 2024-01-21T11:48:16.485 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:16.484+0000 7f1d557fa700 1 -- 172.21.15.134:0/3389743427 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1d4c0095f0 con 0x7f1d580f6510 2024-01-21T11:48:16.485 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:16.484+0000 7f1d557fa700 1 --2- 172.21.15.134:0/3389743427 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f1d4802ff60 0x7f1d48032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:48:16.485 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:16.484+0000 7f1d557fa700 1 -- 172.21.15.134:0/3389743427 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1d4c041000 con 0x7f1d580f6510 2024-01-21T11:48:16.485 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:16.484+0000 7f1d57fff700 1 --2- 172.21.15.134:0/3389743427 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f1d4802ff60 0x7f1d48032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:48:16.485 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:16.484+0000 7f1d57fff700 1 --2- 172.21.15.134:0/3389743427 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f1d4802ff60 0x7f1d48032420 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f1d440096a0 tx=0x7f1d44006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:48:16.488 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:16.489+0000 7f1d557fa700 1 -- 172.21.15.134:0/3389743427 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1d4c014420 con 0x7f1d580f6510 2024-01-21T11:48:16.694 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:16.694+0000 7f1d5f0d1700 1 -- 172.21.15.134:0/3389743427 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1d580fbf60 con 0x7f1d580f6510 2024-01-21T11:48:16.695 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:16.695+0000 7f1d557fa700 1 -- 172.21.15.134:0/3389743427 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1d4c010230 con 0x7f1d580f6510 2024-01-21T11:48:16.695 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:48:16.695 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:48:16.697 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:16.696+0000 7f1d5f0d1700 1 -- 172.21.15.134:0/3389743427 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f1d4802ff60 msgr2=0x7f1d48032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:48:16.697 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:16.696+0000 7f1d5f0d1700 1 --2- 172.21.15.134:0/3389743427 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f1d4802ff60 0x7f1d48032420 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f1d440096a0 tx=0x7f1d44006b40).stop 2024-01-21T11:48:16.697 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:16.696+0000 7f1d5f0d1700 1 -- 172.21.15.134:0/3389743427 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d580f6510 msgr2=0x7f1d58106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:48:16.697 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:16.696+0000 7f1d5f0d1700 1 --2- 172.21.15.134:0/3389743427 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d580f6510 0x7f1d58106730 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7f1d4c013510 tx=0x7f1d4c00b040).stop 2024-01-21T11:48:16.697 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:16.696+0000 7f1d5f0d1700 1 -- 172.21.15.134:0/3389743427 shutdown_connections 2024-01-21T11:48:16.697 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:16.696+0000 7f1d5f0d1700 1 --2- 172.21.15.134:0/3389743427 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f1d4802ff60 0x7f1d48032420 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:48:16.697 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:16.696+0000 7f1d5f0d1700 1 --2- 172.21.15.134:0/3389743427 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d580f6510 0x7f1d58106730 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:48:16.697 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:16.696+0000 7f1d5f0d1700 1 -- 172.21.15.134:0/3389743427 >> 172.21.15.134:0/3389743427 conn(0x7f1d580f1c70 msgr2=0x7f1d580f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:48:16.697 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:16.696+0000 7f1d5f0d1700 1 -- 172.21.15.134:0/3389743427 shutdown_connections 2024-01-21T11:48:16.697 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:16.696+0000 7f1d5f0d1700 1 -- 172.21.15.134:0/3389743427 wait complete. 2024-01-21T11:48:16.698 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:48:17.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:48:17 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:48:15.861392+0000 mgr.smithi029.blhxxk (mgr.14164) 193 : cluster [DBG] pgmap v181: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:48:17.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:48:17 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:48:16.695801+0000 mon.smithi029 (mon.0) 263 : audit [DBG] from='client.? 172.21.15.134:0/3389743427' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:48:18.066 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:48:18.066 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:48:19.229 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:19.227+0000 7f5eb1c95700 1 -- 172.21.15.134:0/3072523736 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5eac0601e0 msgr2=0x7f5eac0f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:48:19.229 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:19.227+0000 7f5eb1c95700 1 --2- 172.21.15.134:0/3072523736 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5eac0601e0 0x7f5eac0f8f20 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7f5ea0009fb0 tx=0x7f5ea0009b30).stop 2024-01-21T11:48:19.229 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:19.227+0000 7f5eb1c95700 1 -- 172.21.15.134:0/3072523736 shutdown_connections 2024-01-21T11:48:19.230 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:19.227+0000 7f5eb1c95700 1 --2- 172.21.15.134:0/3072523736 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5eac0601e0 0x7f5eac0f8f20 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:48:19.230 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:19.228+0000 7f5eb1c95700 1 -- 172.21.15.134:0/3072523736 >> 172.21.15.134:0/3072523736 conn(0x7f5eac0f09a0 msgr2=0x7f5eac0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:48:19.230 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:19.228+0000 7f5eb1c95700 1 -- 172.21.15.134:0/3072523736 shutdown_connections 2024-01-21T11:48:19.230 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:19.228+0000 7f5eb1c95700 1 -- 172.21.15.134:0/3072523736 wait complete. 2024-01-21T11:48:19.230 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:19.228+0000 7f5eb1c95700 1 Processor -- start 2024-01-21T11:48:19.230 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:19.228+0000 7f5eb1c95700 1 -- start start 2024-01-21T11:48:19.230 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:19.228+0000 7f5eb1c95700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5eac0601e0 0x7f5eac106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:48:19.230 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:19.228+0000 7f5eb1c95700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eac1069f0 con 0x7f5eac0601e0 2024-01-21T11:48:19.230 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:19.229+0000 7f5eb0c93700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5eac0601e0 0x7f5eac106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:48:19.230 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:19.229+0000 7f5eb0c93700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5eac0601e0 0x7f5eac106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:35210/0 (socket says 172.21.15.134:35210) 2024-01-21T11:48:19.231 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:19.229+0000 7f5eb0c93700 1 -- 172.21.15.134:0/4254071849 learned_addr learned my addr 172.21.15.134:0/4254071849 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:48:19.231 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:19.229+0000 7f5eb0c93700 1 -- 172.21.15.134:0/4254071849 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ea0005040 con 0x7f5eac0601e0 2024-01-21T11:48:19.231 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:19.229+0000 7f5eb0c93700 1 --2- 172.21.15.134:0/4254071849 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5eac0601e0 0x7f5eac106500 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7f5ea0013510 tx=0x7f5ea000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:48:19.231 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:19.229+0000 7f5ea97fa700 1 -- 172.21.15.134:0/4254071849 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5ea0009320 con 0x7f5eac0601e0 2024-01-21T11:48:19.231 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:19.229+0000 7f5eb1c95700 1 -- 172.21.15.134:0/4254071849 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5eac106bf0 con 0x7f5eac0601e0 2024-01-21T11:48:19.232 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:19.229+0000 7f5eb1c95700 1 -- 172.21.15.134:0/4254071849 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5eac107080 con 0x7f5eac0601e0 2024-01-21T11:48:19.232 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:19.229+0000 7f5ea97fa700 1 -- 172.21.15.134:0/4254071849 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5ea0013f00 con 0x7f5eac0601e0 2024-01-21T11:48:19.232 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:19.229+0000 7f5ea97fa700 1 -- 172.21.15.134:0/4254071849 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5ea0014d60 con 0x7f5eac0601e0 2024-01-21T11:48:19.232 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:19.230+0000 7f5ea97fa700 1 -- 172.21.15.134:0/4254071849 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5ea0014420 con 0x7f5eac0601e0 2024-01-21T11:48:19.232 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:19.230+0000 7f5eb1c95700 1 -- 172.21.15.134:0/4254071849 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5eac045ad0 con 0x7f5eac0601e0 2024-01-21T11:48:19.232 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:19.230+0000 7f5ea97fa700 1 --2- 172.21.15.134:0/4254071849 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f5e9c02ff60 0x7f5e9c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:48:19.232 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:19.230+0000 7f5ea97fa700 1 -- 172.21.15.134:0/4254071849 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5ea00412a0 con 0x7f5eac0601e0 2024-01-21T11:48:19.232 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:19.231+0000 7f5eabfff700 1 --2- 172.21.15.134:0/4254071849 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f5e9c02ff60 0x7f5e9c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:48:19.232 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:19.231+0000 7f5eabfff700 1 --2- 172.21.15.134:0/4254071849 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f5e9c02ff60 0x7f5e9c032420 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f5e980096a0 tx=0x7f5e98006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:48:19.235 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:19.235+0000 7f5ea97fa700 1 -- 172.21.15.134:0/4254071849 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5ea001c320 con 0x7f5eac0601e0 2024-01-21T11:48:19.443 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:19.443+0000 7f5eb1c95700 1 -- 172.21.15.134:0/4254071849 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5eac107f30 con 0x7f5eac0601e0 2024-01-21T11:48:19.444 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:19.444+0000 7f5ea97fa700 1 -- 172.21.15.134:0/4254071849 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5ea0012b20 con 0x7f5eac0601e0 2024-01-21T11:48:19.444 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:48:19.444 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:48:19.445 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:19.445+0000 7f5eb1c95700 1 -- 172.21.15.134:0/4254071849 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f5e9c02ff60 msgr2=0x7f5e9c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:48:19.446 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:19.445+0000 7f5eb1c95700 1 --2- 172.21.15.134:0/4254071849 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f5e9c02ff60 0x7f5e9c032420 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f5e980096a0 tx=0x7f5e98006b40).stop 2024-01-21T11:48:19.446 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:19.445+0000 7f5eb1c95700 1 -- 172.21.15.134:0/4254071849 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5eac0601e0 msgr2=0x7f5eac106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:48:19.446 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:19.445+0000 7f5eb1c95700 1 --2- 172.21.15.134:0/4254071849 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5eac0601e0 0x7f5eac106500 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7f5ea0013510 tx=0x7f5ea000b040).stop 2024-01-21T11:48:19.446 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:19.445+0000 7f5eb1c95700 1 -- 172.21.15.134:0/4254071849 shutdown_connections 2024-01-21T11:48:19.446 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:19.445+0000 7f5eb1c95700 1 --2- 172.21.15.134:0/4254071849 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f5e9c02ff60 0x7f5e9c032420 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:48:19.446 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:19.445+0000 7f5eb1c95700 1 --2- 172.21.15.134:0/4254071849 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5eac0601e0 0x7f5eac106500 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:48:19.446 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:19.445+0000 7f5eb1c95700 1 -- 172.21.15.134:0/4254071849 >> 172.21.15.134:0/4254071849 conn(0x7f5eac0f09a0 msgr2=0x7f5eac0f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:48:19.446 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:19.445+0000 7f5eb1c95700 1 -- 172.21.15.134:0/4254071849 shutdown_connections 2024-01-21T11:48:19.446 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:19.445+0000 7f5eb1c95700 1 -- 172.21.15.134:0/4254071849 wait complete. 2024-01-21T11:48:19.446 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:48:19.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:48:19 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:48:17.861758+0000 mgr.smithi029.blhxxk (mgr.14164) 194 : cluster [DBG] pgmap v182: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:48:19.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:48:19 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:48:19.444928+0000 mon.smithi029 (mon.0) 264 : audit [DBG] from='client.? 172.21.15.134:0/4254071849' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:48:20.817 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:48:20.818 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:48:21.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:48:21 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:48:19.862155+0000 mgr.smithi029.blhxxk (mgr.14164) 195 : cluster [DBG] pgmap v183: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:48:21.990 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:21.988+0000 7f9ea865c700 1 -- 172.21.15.134:0/267989868 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ea00f6480 msgr2=0x7f9ea00f68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:48:21.990 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:21.988+0000 7f9ea865c700 1 --2- 172.21.15.134:0/267989868 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ea00f6480 0x7f9ea00f68a0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7f9e90009fb0 tx=0x7f9e90009b30).stop 2024-01-21T11:48:21.990 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:21.989+0000 7f9ea865c700 1 -- 172.21.15.134:0/267989868 shutdown_connections 2024-01-21T11:48:21.990 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:21.989+0000 7f9ea865c700 1 --2- 172.21.15.134:0/267989868 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ea00f6480 0x7f9ea00f68a0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:48:21.990 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:21.989+0000 7f9ea865c700 1 -- 172.21.15.134:0/267989868 >> 172.21.15.134:0/267989868 conn(0x7f9ea00f1c00 msgr2=0x7f9ea00f4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:48:21.990 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:21.989+0000 7f9ea865c700 1 -- 172.21.15.134:0/267989868 shutdown_connections 2024-01-21T11:48:21.990 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:21.989+0000 7f9ea865c700 1 -- 172.21.15.134:0/267989868 wait complete. 2024-01-21T11:48:21.990 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:21.989+0000 7f9ea865c700 1 Processor -- start 2024-01-21T11:48:21.990 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:21.989+0000 7f9ea865c700 1 -- start start 2024-01-21T11:48:21.990 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:21.989+0000 7f9ea865c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ea00f6480 0x7f9ea006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:48:21.990 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:21.989+0000 7f9ea865c700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea0069e20 con 0x7f9ea00f6480 2024-01-21T11:48:21.990 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:21.989+0000 7f9ea63f8700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ea00f6480 0x7f9ea006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:48:21.991 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:21.989+0000 7f9ea63f8700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ea00f6480 0x7f9ea006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:35230/0 (socket says 172.21.15.134:35230) 2024-01-21T11:48:21.991 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:21.989+0000 7f9ea63f8700 1 -- 172.21.15.134:0/46197727 learned_addr learned my addr 172.21.15.134:0/46197727 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:48:21.992 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:21.990+0000 7f9ea63f8700 1 -- 172.21.15.134:0/46197727 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e90005040 con 0x7f9ea00f6480 2024-01-21T11:48:21.992 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:21.990+0000 7f9ea63f8700 1 --2- 172.21.15.134:0/46197727 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ea00f6480 0x7f9ea006b9a0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f9e90013510 tx=0x7f9e9000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:48:21.992 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:21.990+0000 7f9e96ffd700 1 -- 172.21.15.134:0/46197727 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e90009320 con 0x7f9ea00f6480 2024-01-21T11:48:21.992 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:21.990+0000 7f9e96ffd700 1 -- 172.21.15.134:0/46197727 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9e90013f00 con 0x7f9ea00f6480 2024-01-21T11:48:21.992 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:21.990+0000 7f9ea865c700 1 -- 172.21.15.134:0/46197727 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ea0069fc0 con 0x7f9ea00f6480 2024-01-21T11:48:21.993 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:21.990+0000 7f9e96ffd700 1 -- 172.21.15.134:0/46197727 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e90014d90 con 0x7f9ea00f6480 2024-01-21T11:48:21.993 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:21.990+0000 7f9ea865c700 1 -- 172.21.15.134:0/46197727 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ea006a450 con 0x7f9ea00f6480 2024-01-21T11:48:21.993 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:21.991+0000 7f9e96ffd700 1 -- 172.21.15.134:0/46197727 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9e900095f0 con 0x7f9ea00f6480 2024-01-21T11:48:21.993 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:21.991+0000 7f9ea865c700 1 -- 172.21.15.134:0/46197727 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ea0059100 con 0x7f9ea00f6480 2024-01-21T11:48:21.993 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:21.991+0000 7f9e96ffd700 1 --2- 172.21.15.134:0/46197727 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f9e8c02ff60 0x7f9e8c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:48:21.993 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:21.991+0000 7f9e96ffd700 1 -- 172.21.15.134:0/46197727 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9e900410c0 con 0x7f9ea00f6480 2024-01-21T11:48:21.993 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:21.991+0000 7f9ea5bf7700 1 --2- 172.21.15.134:0/46197727 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f9e8c02ff60 0x7f9e8c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:48:21.993 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:21.992+0000 7f9ea5bf7700 1 --2- 172.21.15.134:0/46197727 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f9e8c02ff60 0x7f9e8c032420 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f9e9c0096a0 tx=0x7f9e9c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:48:21.996 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:21.996+0000 7f9e96ffd700 1 -- 172.21.15.134:0/46197727 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9e9001d7f0 con 0x7f9ea00f6480 2024-01-21T11:48:22.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:22.200+0000 7f9ea865c700 1 -- 172.21.15.134:0/46197727 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9ea00fbf40 con 0x7f9ea00f6480 2024-01-21T11:48:22.201 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:22.200+0000 7f9e96ffd700 1 -- 172.21.15.134:0/46197727 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9e90010230 con 0x7f9ea00f6480 2024-01-21T11:48:22.201 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:48:22.201 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:48:22.202 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:22.202+0000 7f9ea865c700 1 -- 172.21.15.134:0/46197727 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f9e8c02ff60 msgr2=0x7f9e8c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:48:22.203 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:22.202+0000 7f9ea865c700 1 --2- 172.21.15.134:0/46197727 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f9e8c02ff60 0x7f9e8c032420 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f9e9c0096a0 tx=0x7f9e9c006b40).stop 2024-01-21T11:48:22.203 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:22.202+0000 7f9ea865c700 1 -- 172.21.15.134:0/46197727 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ea00f6480 msgr2=0x7f9ea006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:48:22.203 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:22.202+0000 7f9ea865c700 1 --2- 172.21.15.134:0/46197727 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ea00f6480 0x7f9ea006b9a0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f9e90013510 tx=0x7f9e9000b040).stop 2024-01-21T11:48:22.203 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:22.202+0000 7f9ea865c700 1 -- 172.21.15.134:0/46197727 shutdown_connections 2024-01-21T11:48:22.203 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:22.202+0000 7f9ea865c700 1 --2- 172.21.15.134:0/46197727 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f9e8c02ff60 0x7f9e8c032420 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:48:22.203 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:22.202+0000 7f9ea865c700 1 --2- 172.21.15.134:0/46197727 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ea00f6480 0x7f9ea006b9a0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:48:22.203 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:22.202+0000 7f9ea865c700 1 -- 172.21.15.134:0/46197727 >> 172.21.15.134:0/46197727 conn(0x7f9ea00f1c00 msgr2=0x7f9ea00f2720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:48:22.203 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:22.202+0000 7f9ea865c700 1 -- 172.21.15.134:0/46197727 shutdown_connections 2024-01-21T11:48:22.203 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:22.202+0000 7f9ea865c700 1 -- 172.21.15.134:0/46197727 wait complete. 2024-01-21T11:48:22.204 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:48:22.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:48:22 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:48:22.201725+0000 mon.smithi029 (mon.0) 265 : audit [DBG] from='client.? 172.21.15.134:0/46197727' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:48:23.576 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:48:23.577 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:48:23.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:48:23 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:48:21.862522+0000 mgr.smithi029.blhxxk (mgr.14164) 196 : cluster [DBG] pgmap v184: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:48:24.777 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:24.777+0000 7f7cf138a700 1 -- 172.21.15.134:0/2616534959 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7cec0f5410 msgr2=0x7f7cec0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:48:24.778 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:24.777+0000 7f7cf138a700 1 --2- 172.21.15.134:0/2616534959 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7cec0f5410 0x7f7cec0f5830 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7f7ce0009fb0 tx=0x7f7ce0009b30).stop 2024-01-21T11:48:24.778 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:24.777+0000 7f7cf138a700 1 -- 172.21.15.134:0/2616534959 shutdown_connections 2024-01-21T11:48:24.778 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:24.777+0000 7f7cf138a700 1 --2- 172.21.15.134:0/2616534959 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7cec0f5410 0x7f7cec0f5830 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:48:24.778 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:24.777+0000 7f7cf138a700 1 -- 172.21.15.134:0/2616534959 >> 172.21.15.134:0/2616534959 conn(0x7f7cec0f0b70 msgr2=0x7f7cec0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:48:24.778 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:24.777+0000 7f7cf138a700 1 -- 172.21.15.134:0/2616534959 shutdown_connections 2024-01-21T11:48:24.778 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:24.777+0000 7f7cf138a700 1 -- 172.21.15.134:0/2616534959 wait complete. 2024-01-21T11:48:24.779 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:24.778+0000 7f7cf138a700 1 Processor -- start 2024-01-21T11:48:24.780 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:24.778+0000 7f7cf138a700 1 -- start start 2024-01-21T11:48:24.780 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:24.778+0000 7f7cf138a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7cec0f5410 0x7f7cec105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:48:24.780 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:24.778+0000 7f7cf138a700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cec105b20 con 0x7f7cec0f5410 2024-01-21T11:48:24.780 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:24.778+0000 7f7ceaffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7cec0f5410 0x7f7cec105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:48:24.780 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:24.778+0000 7f7ceaffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7cec0f5410 0x7f7cec105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:38190/0 (socket says 172.21.15.134:38190) 2024-01-21T11:48:24.780 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:24.778+0000 7f7ceaffd700 1 -- 172.21.15.134:0/656242260 learned_addr learned my addr 172.21.15.134:0/656242260 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:48:24.780 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:24.779+0000 7f7ceaffd700 1 -- 172.21.15.134:0/656242260 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ce0005040 con 0x7f7cec0f5410 2024-01-21T11:48:24.780 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:24.779+0000 7f7ceaffd700 1 --2- 172.21.15.134:0/656242260 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7cec0f5410 0x7f7cec105630 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f7ce0014000 tx=0x7f7ce000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:48:24.780 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:24.779+0000 7f7cd37fe700 1 -- 172.21.15.134:0/656242260 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7ce00095f0 con 0x7f7cec0f5410 2024-01-21T11:48:24.780 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:24.779+0000 7f7cf138a700 1 -- 172.21.15.134:0/656242260 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cec105d20 con 0x7f7cec0f5410 2024-01-21T11:48:24.780 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:24.779+0000 7f7cd37fe700 1 -- 172.21.15.134:0/656242260 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7ce0007e70 con 0x7f7cec0f5410 2024-01-21T11:48:24.780 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:24.779+0000 7f7cd37fe700 1 -- 172.21.15.134:0/656242260 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7ce0014ea0 con 0x7f7cec0f5410 2024-01-21T11:48:24.781 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:24.779+0000 7f7cf138a700 1 -- 172.21.15.134:0/656242260 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cec1061b0 con 0x7f7cec0f5410 2024-01-21T11:48:24.781 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:24.780+0000 7f7cd37fe700 1 -- 172.21.15.134:0/656242260 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7ce000f530 con 0x7f7cec0f5410 2024-01-21T11:48:24.781 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:24.780+0000 7f7cf138a700 1 -- 172.21.15.134:0/656242260 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cec045ad0 con 0x7f7cec0f5410 2024-01-21T11:48:24.781 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:24.781+0000 7f7cd37fe700 1 --2- 172.21.15.134:0/656242260 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f7cd402ff60 0x7f7cd4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:48:24.782 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:24.781+0000 7f7cd37fe700 1 -- 172.21.15.134:0/656242260 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7ce00425d0 con 0x7f7cec0f5410 2024-01-21T11:48:24.782 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:24.781+0000 7f7cea7fc700 1 --2- 172.21.15.134:0/656242260 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f7cd402ff60 0x7f7cd4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:48:24.782 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:24.781+0000 7f7cea7fc700 1 --2- 172.21.15.134:0/656242260 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f7cd402ff60 0x7f7cd4032420 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f7cdc0096a0 tx=0x7f7cdc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:48:24.785 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:24.785+0000 7f7cd37fe700 1 -- 172.21.15.134:0/656242260 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7ce0027760 con 0x7f7cec0f5410 2024-01-21T11:48:24.987 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:24.987+0000 7f7cf138a700 1 -- 172.21.15.134:0/656242260 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7cec0fae60 con 0x7f7cec0f5410 2024-01-21T11:48:24.988 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:24.988+0000 7f7cd37fe700 1 -- 172.21.15.134:0/656242260 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7ce0011290 con 0x7f7cec0f5410 2024-01-21T11:48:24.988 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:48:24.988 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:48:24.990 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:24.989+0000 7f7cf138a700 1 -- 172.21.15.134:0/656242260 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f7cd402ff60 msgr2=0x7f7cd4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:48:24.990 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:24.989+0000 7f7cf138a700 1 --2- 172.21.15.134:0/656242260 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f7cd402ff60 0x7f7cd4032420 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f7cdc0096a0 tx=0x7f7cdc006b40).stop 2024-01-21T11:48:24.990 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:24.989+0000 7f7cf138a700 1 -- 172.21.15.134:0/656242260 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7cec0f5410 msgr2=0x7f7cec105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:48:24.990 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:24.989+0000 7f7cf138a700 1 --2- 172.21.15.134:0/656242260 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7cec0f5410 0x7f7cec105630 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f7ce0014000 tx=0x7f7ce000b040).stop 2024-01-21T11:48:24.990 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:24.989+0000 7f7cf138a700 1 -- 172.21.15.134:0/656242260 shutdown_connections 2024-01-21T11:48:24.990 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:24.989+0000 7f7cf138a700 1 --2- 172.21.15.134:0/656242260 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f7cd402ff60 0x7f7cd4032420 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:48:24.990 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:24.989+0000 7f7cf138a700 1 --2- 172.21.15.134:0/656242260 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7cec0f5410 0x7f7cec105630 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:48:24.990 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:24.989+0000 7f7cf138a700 1 -- 172.21.15.134:0/656242260 >> 172.21.15.134:0/656242260 conn(0x7f7cec0f0b70 msgr2=0x7f7cec0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:48:24.990 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:24.989+0000 7f7cf138a700 1 -- 172.21.15.134:0/656242260 shutdown_connections 2024-01-21T11:48:24.990 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:24.989+0000 7f7cf138a700 1 -- 172.21.15.134:0/656242260 wait complete. 2024-01-21T11:48:24.991 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:48:25.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:48:25 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:48:23.862829+0000 mgr.smithi029.blhxxk (mgr.14164) 197 : cluster [DBG] pgmap v185: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:48:25.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:48:25 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:48:24.988988+0000 mon.smithi029 (mon.0) 266 : audit [DBG] from='client.? 172.21.15.134:0/656242260' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:48:26.353 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:48:26.353 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:48:27.518 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:27.518+0000 7fded3fff700 1 -- 172.21.15.134:0/1036810286 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fded40f57d0 msgr2=0x7fded40f5bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:48:27.518 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:27.518+0000 7fded3fff700 1 --2- 172.21.15.134:0/1036810286 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fded40f57d0 0x7fded40f5bf0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7fdec4009fb0 tx=0x7fdec4009b30).stop 2024-01-21T11:48:27.519 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:27.518+0000 7fded3fff700 1 -- 172.21.15.134:0/1036810286 shutdown_connections 2024-01-21T11:48:27.519 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:27.518+0000 7fded3fff700 1 --2- 172.21.15.134:0/1036810286 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fded40f57d0 0x7fded40f5bf0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:48:27.519 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:27.518+0000 7fded3fff700 1 -- 172.21.15.134:0/1036810286 >> 172.21.15.134:0/1036810286 conn(0x7fded40f0120 msgr2=0x7fded40f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:48:27.519 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:27.518+0000 7fded3fff700 1 -- 172.21.15.134:0/1036810286 shutdown_connections 2024-01-21T11:48:27.519 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:27.518+0000 7fded3fff700 1 -- 172.21.15.134:0/1036810286 wait complete. 2024-01-21T11:48:27.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:27.519+0000 7fded3fff700 1 Processor -- start 2024-01-21T11:48:27.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:27.519+0000 7fded3fff700 1 -- start start 2024-01-21T11:48:27.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:27.519+0000 7fded3fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fded40f57d0 0x7fded41053a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:48:27.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:27.519+0000 7fded3fff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fded4105890 con 0x7fded40f57d0 2024-01-21T11:48:27.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:27.519+0000 7fded2ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fded40f57d0 0x7fded41053a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:48:27.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:27.519+0000 7fded2ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fded40f57d0 0x7fded41053a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:38208/0 (socket says 172.21.15.134:38208) 2024-01-21T11:48:27.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:27.519+0000 7fded2ffd700 1 -- 172.21.15.134:0/3071176366 learned_addr learned my addr 172.21.15.134:0/3071176366 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:48:27.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:27.520+0000 7fded2ffd700 1 -- 172.21.15.134:0/3071176366 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdec4005040 con 0x7fded40f57d0 2024-01-21T11:48:27.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:27.520+0000 7fded2ffd700 1 --2- 172.21.15.134:0/3071176366 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fded40f57d0 0x7fded41053a0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7fdec4013510 tx=0x7fdec400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:48:27.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:27.520+0000 7fdebb7fe700 1 -- 172.21.15.134:0/3071176366 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdec4009320 con 0x7fded40f57d0 2024-01-21T11:48:27.522 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:27.520+0000 7fded3fff700 1 -- 172.21.15.134:0/3071176366 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fded4105a90 con 0x7fded40f57d0 2024-01-21T11:48:27.522 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:27.520+0000 7fdebb7fe700 1 -- 172.21.15.134:0/3071176366 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdec4013f00 con 0x7fded40f57d0 2024-01-21T11:48:27.522 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:27.520+0000 7fdebb7fe700 1 -- 172.21.15.134:0/3071176366 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdec4014dc0 con 0x7fded40f57d0 2024-01-21T11:48:27.522 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:27.520+0000 7fded3fff700 1 -- 172.21.15.134:0/3071176366 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fded4105f20 con 0x7fded40f57d0 2024-01-21T11:48:27.522 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:27.521+0000 7fdebb7fe700 1 -- 172.21.15.134:0/3071176366 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdec40095f0 con 0x7fded40f57d0 2024-01-21T11:48:27.522 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:27.521+0000 7fded3fff700 1 -- 172.21.15.134:0/3071176366 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fded4045ad0 con 0x7fded40f57d0 2024-01-21T11:48:27.522 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:27.521+0000 7fdebb7fe700 1 --2- 172.21.15.134:0/3071176366 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fdebc02ff60 0x7fdebc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:48:27.522 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:27.521+0000 7fdebb7fe700 1 -- 172.21.15.134:0/3071176366 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdec40410e0 con 0x7fded40f57d0 2024-01-21T11:48:27.522 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:27.521+0000 7fded27fc700 1 --2- 172.21.15.134:0/3071176366 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fdebc02ff60 0x7fdebc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:48:27.523 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:27.522+0000 7fded27fc700 1 --2- 172.21.15.134:0/3071176366 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fdebc02ff60 0x7fdebc032420 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7fdec80096a0 tx=0x7fdec8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:48:27.526 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:27.526+0000 7fdebb7fe700 1 -- 172.21.15.134:0/3071176366 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdec4014420 con 0x7fded40f57d0 2024-01-21T11:48:27.733 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:27.732+0000 7fded3fff700 1 -- 172.21.15.134:0/3071176366 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fded4106d70 con 0x7fded40f57d0 2024-01-21T11:48:27.733 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:27.733+0000 7fdebb7fe700 1 -- 172.21.15.134:0/3071176366 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdec401c920 con 0x7fded40f57d0 2024-01-21T11:48:27.733 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:48:27.734 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:48:27.735 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:27.735+0000 7fded3fff700 1 -- 172.21.15.134:0/3071176366 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fdebc02ff60 msgr2=0x7fdebc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:48:27.735 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:27.735+0000 7fded3fff700 1 --2- 172.21.15.134:0/3071176366 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fdebc02ff60 0x7fdebc032420 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7fdec80096a0 tx=0x7fdec8006b40).stop 2024-01-21T11:48:27.735 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:27.735+0000 7fded3fff700 1 -- 172.21.15.134:0/3071176366 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fded40f57d0 msgr2=0x7fded41053a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:48:27.735 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:27.735+0000 7fded3fff700 1 --2- 172.21.15.134:0/3071176366 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fded40f57d0 0x7fded41053a0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7fdec4013510 tx=0x7fdec400b040).stop 2024-01-21T11:48:27.735 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:27.735+0000 7fded3fff700 1 -- 172.21.15.134:0/3071176366 shutdown_connections 2024-01-21T11:48:27.735 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:27.735+0000 7fded3fff700 1 --2- 172.21.15.134:0/3071176366 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fdebc02ff60 0x7fdebc032420 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:48:27.735 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:27.735+0000 7fded3fff700 1 --2- 172.21.15.134:0/3071176366 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fded40f57d0 0x7fded41053a0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:48:27.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:27.735+0000 7fded3fff700 1 -- 172.21.15.134:0/3071176366 >> 172.21.15.134:0/3071176366 conn(0x7fded40f0120 msgr2=0x7fded40f0b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:48:27.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:27.735+0000 7fded3fff700 1 -- 172.21.15.134:0/3071176366 shutdown_connections 2024-01-21T11:48:27.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:27.735+0000 7fded3fff700 1 -- 172.21.15.134:0/3071176366 wait complete. 2024-01-21T11:48:27.736 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:48:27.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:48:27 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:48:25.863202+0000 mgr.smithi029.blhxxk (mgr.14164) 198 : cluster [DBG] pgmap v186: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:48:28.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:48:28 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:48:27.734385+0000 mon.smithi029 (mon.0) 267 : audit [DBG] from='client.? 172.21.15.134:0/3071176366' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:48:29.130 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:48:29.130 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:48:29.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:48:29 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:48:27.863591+0000 mgr.smithi029.blhxxk (mgr.14164) 199 : cluster [DBG] pgmap v187: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:48:30.297 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:30.295+0000 7f82d7b11700 1 -- 172.21.15.134:0/3875763893 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82d0060220 msgr2=0x7f82d00fb550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:48:30.297 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:30.295+0000 7f82d7b11700 1 --2- 172.21.15.134:0/3875763893 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82d0060220 0x7f82d00fb550 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7f82c0009fb0 tx=0x7f82c0009b30).stop 2024-01-21T11:48:30.297 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:30.295+0000 7f82d7b11700 1 -- 172.21.15.134:0/3875763893 shutdown_connections 2024-01-21T11:48:30.297 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:30.295+0000 7f82d7b11700 1 --2- 172.21.15.134:0/3875763893 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82d0060220 0x7f82d00fb550 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:48:30.298 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:30.295+0000 7f82d7b11700 1 -- 172.21.15.134:0/3875763893 >> 172.21.15.134:0/3875763893 conn(0x7f82d00f0b70 msgr2=0x7f82d00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:48:30.298 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:30.295+0000 7f82d7b11700 1 -- 172.21.15.134:0/3875763893 shutdown_connections 2024-01-21T11:48:30.298 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:30.295+0000 7f82d7b11700 1 -- 172.21.15.134:0/3875763893 wait complete. 2024-01-21T11:48:30.298 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:30.296+0000 7f82d7b11700 1 Processor -- start 2024-01-21T11:48:30.298 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:30.296+0000 7f82d7b11700 1 -- start start 2024-01-21T11:48:30.298 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:30.296+0000 7f82d7b11700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82d0060220 0x7f82d01088c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:48:30.298 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:30.296+0000 7f82d7b11700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82d0108db0 con 0x7f82d0060220 2024-01-21T11:48:30.298 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:30.296+0000 7f82d58ad700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82d0060220 0x7f82d01088c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:48:30.298 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:30.296+0000 7f82d58ad700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82d0060220 0x7f82d01088c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:38234/0 (socket says 172.21.15.134:38234) 2024-01-21T11:48:30.298 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:30.296+0000 7f82d58ad700 1 -- 172.21.15.134:0/69699015 learned_addr learned my addr 172.21.15.134:0/69699015 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:48:30.298 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:30.296+0000 7f82d58ad700 1 -- 172.21.15.134:0/69699015 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82c0005040 con 0x7f82d0060220 2024-01-21T11:48:30.298 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:30.296+0000 7f82d58ad700 1 --2- 172.21.15.134:0/69699015 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82d0060220 0x7f82d01088c0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7f82c0013510 tx=0x7f82c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:48:30.299 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:30.297+0000 7f82c67fc700 1 -- 172.21.15.134:0/69699015 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f82c0009320 con 0x7f82d0060220 2024-01-21T11:48:30.299 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:30.297+0000 7f82d7b11700 1 -- 172.21.15.134:0/69699015 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82d0108fb0 con 0x7f82d0060220 2024-01-21T11:48:30.299 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:30.297+0000 7f82c67fc700 1 -- 172.21.15.134:0/69699015 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f82c0013f00 con 0x7f82d0060220 2024-01-21T11:48:30.299 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:30.297+0000 7f82c67fc700 1 -- 172.21.15.134:0/69699015 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f82c0014dc0 con 0x7f82d0060220 2024-01-21T11:48:30.299 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:30.297+0000 7f82d7b11700 1 -- 172.21.15.134:0/69699015 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82d0109440 con 0x7f82d0060220 2024-01-21T11:48:30.300 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:30.297+0000 7f82c67fc700 1 -- 172.21.15.134:0/69699015 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f82c0014420 con 0x7f82d0060220 2024-01-21T11:48:30.300 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:30.297+0000 7f82d7b11700 1 -- 172.21.15.134:0/69699015 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82d0045ad0 con 0x7f82d0060220 2024-01-21T11:48:30.300 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:30.297+0000 7f82c67fc700 1 --2- 172.21.15.134:0/69699015 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f82bc02ff60 0x7f82bc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:48:30.300 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:30.298+0000 7f82c67fc700 1 -- 172.21.15.134:0/69699015 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f82c0041040 con 0x7f82d0060220 2024-01-21T11:48:30.300 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:30.298+0000 7f82d50ac700 1 --2- 172.21.15.134:0/69699015 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f82bc02ff60 0x7f82bc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:48:30.300 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:30.298+0000 7f82d50ac700 1 --2- 172.21.15.134:0/69699015 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f82bc02ff60 0x7f82bc032420 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f82cc0096a0 tx=0x7f82cc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:48:30.302 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:30.302+0000 7f82c67fc700 1 -- 172.21.15.134:0/69699015 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f82c00095f0 con 0x7f82d0060220 2024-01-21T11:48:30.509 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:30.509+0000 7f82d7b11700 1 -- 172.21.15.134:0/69699015 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f82d010a290 con 0x7f82d0060220 2024-01-21T11:48:30.510 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:30.510+0000 7f82c67fc700 1 -- 172.21.15.134:0/69699015 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f82c001cb10 con 0x7f82d0060220 2024-01-21T11:48:30.511 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:48:30.511 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:48:30.512 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:30.512+0000 7f82d7b11700 1 -- 172.21.15.134:0/69699015 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f82bc02ff60 msgr2=0x7f82bc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:48:30.513 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:30.512+0000 7f82d7b11700 1 --2- 172.21.15.134:0/69699015 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f82bc02ff60 0x7f82bc032420 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f82cc0096a0 tx=0x7f82cc006b40).stop 2024-01-21T11:48:30.513 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:30.512+0000 7f82d7b11700 1 -- 172.21.15.134:0/69699015 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82d0060220 msgr2=0x7f82d01088c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:48:30.513 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:30.512+0000 7f82d7b11700 1 --2- 172.21.15.134:0/69699015 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82d0060220 0x7f82d01088c0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7f82c0013510 tx=0x7f82c000b040).stop 2024-01-21T11:48:30.513 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:30.512+0000 7f82d7b11700 1 -- 172.21.15.134:0/69699015 shutdown_connections 2024-01-21T11:48:30.513 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:30.512+0000 7f82d7b11700 1 --2- 172.21.15.134:0/69699015 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f82bc02ff60 0x7f82bc032420 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:48:30.513 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:30.512+0000 7f82d7b11700 1 --2- 172.21.15.134:0/69699015 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82d0060220 0x7f82d01088c0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:48:30.513 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:30.512+0000 7f82d7b11700 1 -- 172.21.15.134:0/69699015 >> 172.21.15.134:0/69699015 conn(0x7f82d00f0b70 msgr2=0x7f82d00f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:48:30.513 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:30.512+0000 7f82d7b11700 1 -- 172.21.15.134:0/69699015 shutdown_connections 2024-01-21T11:48:30.513 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:30.512+0000 7f82d7b11700 1 -- 172.21.15.134:0/69699015 wait complete. 2024-01-21T11:48:30.513 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:48:30.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:48:30 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:48:30.511255+0000 mon.smithi029 (mon.0) 268 : audit [DBG] from='client.? 172.21.15.134:0/69699015' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:48:31.893 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:48:31.893 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:48:31.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:48:31 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:48:29.863976+0000 mgr.smithi029.blhxxk (mgr.14164) 200 : cluster [DBG] pgmap v188: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:48:33.043 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:33.042+0000 7f45135d9700 1 -- 172.21.15.134:0/1246014131 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f450c0f5410 msgr2=0x7f450c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:48:33.043 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:33.042+0000 7f45135d9700 1 --2- 172.21.15.134:0/1246014131 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f450c0f5410 0x7f450c0f5830 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7f44fc009fb0 tx=0x7f44fc009b30).stop 2024-01-21T11:48:33.043 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:33.042+0000 7f45135d9700 1 -- 172.21.15.134:0/1246014131 shutdown_connections 2024-01-21T11:48:33.044 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:33.042+0000 7f45135d9700 1 --2- 172.21.15.134:0/1246014131 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f450c0f5410 0x7f450c0f5830 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:48:33.044 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:33.042+0000 7f45135d9700 1 -- 172.21.15.134:0/1246014131 >> 172.21.15.134:0/1246014131 conn(0x7f450c0f0b70 msgr2=0x7f450c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:48:33.044 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:33.042+0000 7f45135d9700 1 -- 172.21.15.134:0/1246014131 shutdown_connections 2024-01-21T11:48:33.044 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:33.042+0000 7f45135d9700 1 -- 172.21.15.134:0/1246014131 wait complete. 2024-01-21T11:48:33.044 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:33.042+0000 7f45135d9700 1 Processor -- start 2024-01-21T11:48:33.044 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:33.042+0000 7f45135d9700 1 -- start start 2024-01-21T11:48:33.044 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:33.042+0000 7f45135d9700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f450c0f5410 0x7f450c0fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:48:33.044 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:33.042+0000 7f45135d9700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f450c0fb820 con 0x7f450c0f5410 2024-01-21T11:48:33.044 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:33.042+0000 7f4511375700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f450c0f5410 0x7f450c0fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:48:33.044 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:33.043+0000 7f4511375700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f450c0f5410 0x7f450c0fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:38258/0 (socket says 172.21.15.134:38258) 2024-01-21T11:48:33.044 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:33.043+0000 7f4511375700 1 -- 172.21.15.134:0/3536912813 learned_addr learned my addr 172.21.15.134:0/3536912813 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:48:33.044 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:33.043+0000 7f4511375700 1 -- 172.21.15.134:0/3536912813 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44fc005040 con 0x7f450c0f5410 2024-01-21T11:48:33.044 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:33.043+0000 7f4511375700 1 --2- 172.21.15.134:0/3536912813 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f450c0f5410 0x7f450c0fd3a0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7f44fc013510 tx=0x7f44fc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:48:33.045 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:33.043+0000 7f4501ffb700 1 -- 172.21.15.134:0/3536912813 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f44fc009320 con 0x7f450c0f5410 2024-01-21T11:48:33.045 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:33.043+0000 7f45135d9700 1 -- 172.21.15.134:0/3536912813 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f450c0fb9c0 con 0x7f450c0f5410 2024-01-21T11:48:33.045 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:33.043+0000 7f4501ffb700 1 -- 172.21.15.134:0/3536912813 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f44fc013f00 con 0x7f450c0f5410 2024-01-21T11:48:33.045 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:33.043+0000 7f4501ffb700 1 -- 172.21.15.134:0/3536912813 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f44fc014d90 con 0x7f450c0f5410 2024-01-21T11:48:33.045 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:33.043+0000 7f45135d9700 1 -- 172.21.15.134:0/3536912813 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f450c0fbe50 con 0x7f450c0f5410 2024-01-21T11:48:33.046 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:33.044+0000 7f45135d9700 1 -- 172.21.15.134:0/3536912813 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f450c059100 con 0x7f450c0f5410 2024-01-21T11:48:33.046 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:33.044+0000 7f4501ffb700 1 -- 172.21.15.134:0/3536912813 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f44fc0095f0 con 0x7f450c0f5410 2024-01-21T11:48:33.046 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:33.044+0000 7f4501ffb700 1 --2- 172.21.15.134:0/3536912813 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f44f802ff60 0x7f44f8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:48:33.046 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:33.044+0000 7f4501ffb700 1 -- 172.21.15.134:0/3536912813 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f44fc041000 con 0x7f450c0f5410 2024-01-21T11:48:33.046 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:33.045+0000 7f4510b74700 1 --2- 172.21.15.134:0/3536912813 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f44f802ff60 0x7f44f8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:48:33.046 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:33.045+0000 7f4510b74700 1 --2- 172.21.15.134:0/3536912813 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f44f802ff60 0x7f44f8032420 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f45080096a0 tx=0x7f4508006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:48:33.049 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:33.049+0000 7f4501ffb700 1 -- 172.21.15.134:0/3536912813 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f44fc014420 con 0x7f450c0f5410 2024-01-21T11:48:33.258 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:33.258+0000 7f45135d9700 1 -- 172.21.15.134:0/3536912813 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f450c0fae60 con 0x7f450c0f5410 2024-01-21T11:48:33.259 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:33.259+0000 7f4501ffb700 1 -- 172.21.15.134:0/3536912813 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f44fc01d7f0 con 0x7f450c0f5410 2024-01-21T11:48:33.259 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:48:33.259 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:48:33.261 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:33.261+0000 7f45135d9700 1 -- 172.21.15.134:0/3536912813 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f44f802ff60 msgr2=0x7f44f8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:48:33.261 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:33.261+0000 7f45135d9700 1 --2- 172.21.15.134:0/3536912813 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f44f802ff60 0x7f44f8032420 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f45080096a0 tx=0x7f4508006b40).stop 2024-01-21T11:48:33.261 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:33.261+0000 7f45135d9700 1 -- 172.21.15.134:0/3536912813 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f450c0f5410 msgr2=0x7f450c0fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:48:33.261 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:33.261+0000 7f45135d9700 1 --2- 172.21.15.134:0/3536912813 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f450c0f5410 0x7f450c0fd3a0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7f44fc013510 tx=0x7f44fc00b040).stop 2024-01-21T11:48:33.261 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:33.261+0000 7f45135d9700 1 -- 172.21.15.134:0/3536912813 shutdown_connections 2024-01-21T11:48:33.262 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:33.261+0000 7f45135d9700 1 --2- 172.21.15.134:0/3536912813 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f44f802ff60 0x7f44f8032420 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:48:33.262 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:33.261+0000 7f45135d9700 1 --2- 172.21.15.134:0/3536912813 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f450c0f5410 0x7f450c0fd3a0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:48:33.262 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:33.261+0000 7f45135d9700 1 -- 172.21.15.134:0/3536912813 >> 172.21.15.134:0/3536912813 conn(0x7f450c0f0b70 msgr2=0x7f450c05fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:48:33.262 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:33.261+0000 7f45135d9700 1 -- 172.21.15.134:0/3536912813 shutdown_connections 2024-01-21T11:48:33.262 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:33.261+0000 7f45135d9700 1 -- 172.21.15.134:0/3536912813 wait complete. 2024-01-21T11:48:33.262 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:48:33.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:48:33 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:48:31.864359+0000 mgr.smithi029.blhxxk (mgr.14164) 201 : cluster [DBG] pgmap v189: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:48:33.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:48:33 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:48:33.260083+0000 mon.smithi029 (mon.0) 269 : audit [DBG] from='client.? 172.21.15.134:0/3536912813' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:48:34.679 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:48:34.679 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:48:35.817 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:35.816+0000 7faac5a0f700 1 -- 172.21.15.134:0/3676257937 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faac00f6510 msgr2=0x7faac00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:48:35.817 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:35.816+0000 7faac5a0f700 1 --2- 172.21.15.134:0/3676257937 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faac00f6510 0x7faac00f6930 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7faab4009fb0 tx=0x7faab4009b30).stop 2024-01-21T11:48:35.817 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:35.816+0000 7faac5a0f700 1 -- 172.21.15.134:0/3676257937 shutdown_connections 2024-01-21T11:48:35.818 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:35.816+0000 7faac5a0f700 1 --2- 172.21.15.134:0/3676257937 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faac00f6510 0x7faac00f6930 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:48:35.818 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:35.816+0000 7faac5a0f700 1 -- 172.21.15.134:0/3676257937 >> 172.21.15.134:0/3676257937 conn(0x7faac00f1c70 msgr2=0x7faac00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:48:35.818 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:35.816+0000 7faac5a0f700 1 -- 172.21.15.134:0/3676257937 shutdown_connections 2024-01-21T11:48:35.818 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:35.816+0000 7faac5a0f700 1 -- 172.21.15.134:0/3676257937 wait complete. 2024-01-21T11:48:35.818 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:35.816+0000 7faac5a0f700 1 Processor -- start 2024-01-21T11:48:35.818 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:35.817+0000 7faac5a0f700 1 -- start start 2024-01-21T11:48:35.818 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:35.817+0000 7faac5a0f700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faac00f6510 0x7faac006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:48:35.818 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:35.817+0000 7faac5a0f700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faac0069e20 con 0x7faac00f6510 2024-01-21T11:48:35.818 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:35.817+0000 7faabeffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faac00f6510 0x7faac006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:48:35.818 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:35.817+0000 7faabeffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faac00f6510 0x7faac006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:38624/0 (socket says 172.21.15.134:38624) 2024-01-21T11:48:35.818 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:35.817+0000 7faabeffd700 1 -- 172.21.15.134:0/3633794019 learned_addr learned my addr 172.21.15.134:0/3633794019 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:48:35.818 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:35.817+0000 7faabeffd700 1 -- 172.21.15.134:0/3633794019 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faab4005040 con 0x7faac00f6510 2024-01-21T11:48:35.818 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:35.817+0000 7faabeffd700 1 --2- 172.21.15.134:0/3633794019 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faac00f6510 0x7faac006b9a0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7faab4013510 tx=0x7faab400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:48:35.820 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:35.817+0000 7faaa77fe700 1 -- 172.21.15.134:0/3633794019 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faab4009320 con 0x7faac00f6510 2024-01-21T11:48:35.820 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:35.818+0000 7faac5a0f700 1 -- 172.21.15.134:0/3633794019 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faac0069fc0 con 0x7faac00f6510 2024-01-21T11:48:35.820 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:35.818+0000 7faac5a0f700 1 -- 172.21.15.134:0/3633794019 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faac006a450 con 0x7faac00f6510 2024-01-21T11:48:35.820 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:35.818+0000 7faaa77fe700 1 -- 172.21.15.134:0/3633794019 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faab4013f00 con 0x7faac00f6510 2024-01-21T11:48:35.820 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:35.818+0000 7faaa77fe700 1 -- 172.21.15.134:0/3633794019 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faab4014d90 con 0x7faac00f6510 2024-01-21T11:48:35.820 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:35.818+0000 7faac5a0f700 1 -- 172.21.15.134:0/3633794019 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faac0059100 con 0x7faac00f6510 2024-01-21T11:48:35.820 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:35.818+0000 7faaa77fe700 1 -- 172.21.15.134:0/3633794019 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7faab40095f0 con 0x7faac00f6510 2024-01-21T11:48:35.820 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:35.819+0000 7faaa77fe700 1 --2- 172.21.15.134:0/3633794019 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7faaa802ff60 0x7faaa8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:48:35.820 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:35.819+0000 7faaa77fe700 1 -- 172.21.15.134:0/3633794019 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7faab4041050 con 0x7faac00f6510 2024-01-21T11:48:35.820 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:35.819+0000 7faabe7fc700 1 --2- 172.21.15.134:0/3633794019 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7faaa802ff60 0x7faaa8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:48:35.820 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:35.819+0000 7faabe7fc700 1 --2- 172.21.15.134:0/3633794019 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7faaa802ff60 0x7faaa8032420 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7faab00096a0 tx=0x7faab0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:48:35.823 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:35.823+0000 7faaa77fe700 1 -- 172.21.15.134:0/3633794019 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7faab4014420 con 0x7faac00f6510 2024-01-21T11:48:35.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:48:35 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:48:33.864739+0000 mgr.smithi029.blhxxk (mgr.14164) 202 : cluster [DBG] pgmap v190: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:48:36.028 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:36.028+0000 7faac5a0f700 1 -- 172.21.15.134:0/3633794019 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7faac0045ad0 con 0x7faac00f6510 2024-01-21T11:48:36.029 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:36.028+0000 7faaa77fe700 1 -- 172.21.15.134:0/3633794019 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7faab401d7f0 con 0x7faac00f6510 2024-01-21T11:48:36.029 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:48:36.029 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:48:36.030 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:36.030+0000 7faac5a0f700 1 -- 172.21.15.134:0/3633794019 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7faaa802ff60 msgr2=0x7faaa8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:48:36.030 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:36.030+0000 7faac5a0f700 1 --2- 172.21.15.134:0/3633794019 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7faaa802ff60 0x7faaa8032420 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7faab00096a0 tx=0x7faab0006b40).stop 2024-01-21T11:48:36.030 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:36.030+0000 7faac5a0f700 1 -- 172.21.15.134:0/3633794019 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faac00f6510 msgr2=0x7faac006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:48:36.031 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:36.030+0000 7faac5a0f700 1 --2- 172.21.15.134:0/3633794019 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faac00f6510 0x7faac006b9a0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7faab4013510 tx=0x7faab400b040).stop 2024-01-21T11:48:36.031 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:36.030+0000 7faac5a0f700 1 -- 172.21.15.134:0/3633794019 shutdown_connections 2024-01-21T11:48:36.031 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:36.030+0000 7faac5a0f700 1 --2- 172.21.15.134:0/3633794019 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7faaa802ff60 0x7faaa8032420 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:48:36.031 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:36.030+0000 7faac5a0f700 1 --2- 172.21.15.134:0/3633794019 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faac00f6510 0x7faac006b9a0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:48:36.031 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:36.030+0000 7faac5a0f700 1 -- 172.21.15.134:0/3633794019 >> 172.21.15.134:0/3633794019 conn(0x7faac00f1c70 msgr2=0x7faac00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:48:36.031 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:36.030+0000 7faac5a0f700 1 -- 172.21.15.134:0/3633794019 shutdown_connections 2024-01-21T11:48:36.031 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:36.030+0000 7faac5a0f700 1 -- 172.21.15.134:0/3633794019 wait complete. 2024-01-21T11:48:36.031 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:48:36.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:48:36 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:48:36.029773+0000 mon.smithi029 (mon.0) 270 : audit [DBG] from='client.? 172.21.15.134:0/3633794019' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:48:37.471 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:48:37.472 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:48:37.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:48:37 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:48:35.865056+0000 mgr.smithi029.blhxxk (mgr.14164) 203 : cluster [DBG] pgmap v191: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:48:38.641 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:38.640+0000 7f62418fc700 1 -- 172.21.15.134:0/2784434814 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f623c0f59d0 msgr2=0x7f623c0f5df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:48:38.641 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:38.640+0000 7f62418fc700 1 --2- 172.21.15.134:0/2784434814 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f623c0f59d0 0x7f623c0f5df0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7f6230009fb0 tx=0x7f6230009b30).stop 2024-01-21T11:48:38.641 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:38.640+0000 7f62418fc700 1 -- 172.21.15.134:0/2784434814 shutdown_connections 2024-01-21T11:48:38.641 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:38.640+0000 7f62418fc700 1 --2- 172.21.15.134:0/2784434814 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f623c0f59d0 0x7f623c0f5df0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:48:38.641 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:38.640+0000 7f62418fc700 1 -- 172.21.15.134:0/2784434814 >> 172.21.15.134:0/2784434814 conn(0x7f623c0f02e0 msgr2=0x7f623c0f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:48:38.641 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:38.640+0000 7f62418fc700 1 -- 172.21.15.134:0/2784434814 shutdown_connections 2024-01-21T11:48:38.641 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:38.640+0000 7f62418fc700 1 -- 172.21.15.134:0/2784434814 wait complete. 2024-01-21T11:48:38.642 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:38.641+0000 7f62418fc700 1 Processor -- start 2024-01-21T11:48:38.643 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:38.641+0000 7f62418fc700 1 -- start start 2024-01-21T11:48:38.643 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:38.641+0000 7f62418fc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f623c0f59d0 0x7f623c1093c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:48:38.643 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:38.641+0000 7f62418fc700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f623c0f53b0 con 0x7f623c0f59d0 2024-01-21T11:48:38.643 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:38.641+0000 7f623affd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f623c0f59d0 0x7f623c1093c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:48:38.643 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:38.641+0000 7f623affd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f623c0f59d0 0x7f623c1093c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:38636/0 (socket says 172.21.15.134:38636) 2024-01-21T11:48:38.643 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:38.641+0000 7f623affd700 1 -- 172.21.15.134:0/885716369 learned_addr learned my addr 172.21.15.134:0/885716369 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:48:38.643 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:38.642+0000 7f623affd700 1 -- 172.21.15.134:0/885716369 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6230005040 con 0x7f623c0f59d0 2024-01-21T11:48:38.643 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:38.642+0000 7f623affd700 1 --2- 172.21.15.134:0/885716369 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f623c0f59d0 0x7f623c1093c0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f6230013510 tx=0x7f623000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:48:38.643 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:38.642+0000 7f62237fe700 1 -- 172.21.15.134:0/885716369 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6230009320 con 0x7f623c0f59d0 2024-01-21T11:48:38.643 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:38.642+0000 7f62237fe700 1 -- 172.21.15.134:0/885716369 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6230013f00 con 0x7f623c0f59d0 2024-01-21T11:48:38.643 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:38.642+0000 7f62418fc700 1 -- 172.21.15.134:0/885716369 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f623c0f55b0 con 0x7f623c0f59d0 2024-01-21T11:48:38.643 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:38.642+0000 7f62237fe700 1 -- 172.21.15.134:0/885716369 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6230014d90 con 0x7f623c0f59d0 2024-01-21T11:48:38.643 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:38.642+0000 7f62418fc700 1 -- 172.21.15.134:0/885716369 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f623c109a70 con 0x7f623c0f59d0 2024-01-21T11:48:38.644 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:38.643+0000 7f62237fe700 1 -- 172.21.15.134:0/885716369 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f62300095f0 con 0x7f623c0f59d0 2024-01-21T11:48:38.644 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:38.643+0000 7f62418fc700 1 -- 172.21.15.134:0/885716369 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f623c045ad0 con 0x7f623c0f59d0 2024-01-21T11:48:38.644 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:38.643+0000 7f62237fe700 1 --2- 172.21.15.134:0/885716369 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f622402ff60 0x7f6224032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:48:38.645 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:38.643+0000 7f62237fe700 1 -- 172.21.15.134:0/885716369 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6230040fe0 con 0x7f623c0f59d0 2024-01-21T11:48:38.645 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:38.643+0000 7f623a7fc700 1 --2- 172.21.15.134:0/885716369 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f622402ff60 0x7f6224032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:48:38.645 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:38.644+0000 7f623a7fc700 1 --2- 172.21.15.134:0/885716369 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f622402ff60 0x7f6224032420 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f622c0096a0 tx=0x7f622c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:48:38.648 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:38.648+0000 7f62237fe700 1 -- 172.21.15.134:0/885716369 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f623001d7f0 con 0x7f623c0f59d0 2024-01-21T11:48:38.854 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:38.854+0000 7f62418fc700 1 -- 172.21.15.134:0/885716369 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f623c10a290 con 0x7f623c0f59d0 2024-01-21T11:48:38.855 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:38.855+0000 7f62237fe700 1 -- 172.21.15.134:0/885716369 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6230010230 con 0x7f623c0f59d0 2024-01-21T11:48:38.855 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:48:38.855 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:48:38.857 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:38.856+0000 7f62418fc700 1 -- 172.21.15.134:0/885716369 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f622402ff60 msgr2=0x7f6224032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:48:38.857 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:38.856+0000 7f62418fc700 1 --2- 172.21.15.134:0/885716369 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f622402ff60 0x7f6224032420 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f622c0096a0 tx=0x7f622c006b40).stop 2024-01-21T11:48:38.857 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:38.856+0000 7f62418fc700 1 -- 172.21.15.134:0/885716369 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f623c0f59d0 msgr2=0x7f623c1093c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:48:38.857 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:38.856+0000 7f62418fc700 1 --2- 172.21.15.134:0/885716369 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f623c0f59d0 0x7f623c1093c0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f6230013510 tx=0x7f623000b040).stop 2024-01-21T11:48:38.857 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:38.856+0000 7f62418fc700 1 -- 172.21.15.134:0/885716369 shutdown_connections 2024-01-21T11:48:38.857 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:38.856+0000 7f62418fc700 1 --2- 172.21.15.134:0/885716369 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f622402ff60 0x7f6224032420 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:48:38.857 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:38.856+0000 7f62418fc700 1 --2- 172.21.15.134:0/885716369 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f623c0f59d0 0x7f623c1093c0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:48:38.857 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:38.856+0000 7f62418fc700 1 -- 172.21.15.134:0/885716369 >> 172.21.15.134:0/885716369 conn(0x7f623c0f02e0 msgr2=0x7f623c0f0dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:48:38.857 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:38.856+0000 7f62418fc700 1 -- 172.21.15.134:0/885716369 shutdown_connections 2024-01-21T11:48:38.857 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:38.856+0000 7f62418fc700 1 -- 172.21.15.134:0/885716369 wait complete. 2024-01-21T11:48:38.858 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:48:39.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:48:39 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:48:37.865311+0000 mgr.smithi029.blhxxk (mgr.14164) 204 : cluster [DBG] pgmap v192: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:48:39.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:48:39 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:48:38.855822+0000 mon.smithi029 (mon.0) 271 : audit [DBG] from='client.? 172.21.15.134:0/885716369' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:48:40.294 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:48:40.295 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:48:41.462 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:41.461+0000 7f4901746700 1 -- 172.21.15.134:0/3052796749 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48fc0f47b0 msgr2=0x7f48fc0f4bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:48:41.463 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:41.461+0000 7f4901746700 1 --2- 172.21.15.134:0/3052796749 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48fc0f47b0 0x7f48fc0f4bd0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7f48ec009fb0 tx=0x7f48ec009b30).stop 2024-01-21T11:48:41.463 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:41.461+0000 7f4901746700 1 -- 172.21.15.134:0/3052796749 shutdown_connections 2024-01-21T11:48:41.463 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:41.461+0000 7f4901746700 1 --2- 172.21.15.134:0/3052796749 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48fc0f47b0 0x7f48fc0f4bd0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:48:41.463 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:41.461+0000 7f4901746700 1 -- 172.21.15.134:0/3052796749 >> 172.21.15.134:0/3052796749 conn(0x7f48fc0f02b0 msgr2=0x7f48fc0f26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:48:41.463 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:41.462+0000 7f4901746700 1 -- 172.21.15.134:0/3052796749 shutdown_connections 2024-01-21T11:48:41.463 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:41.462+0000 7f4901746700 1 -- 172.21.15.134:0/3052796749 wait complete. 2024-01-21T11:48:41.463 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:41.462+0000 7f4901746700 1 Processor -- start 2024-01-21T11:48:41.463 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:41.462+0000 7f4901746700 1 -- start start 2024-01-21T11:48:41.463 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:41.462+0000 7f4901746700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48fc0f47b0 0x7f48fc1066a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:48:41.463 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:41.462+0000 7f4901746700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48fc106b90 con 0x7f48fc0f47b0 2024-01-21T11:48:41.463 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:41.463+0000 7f48faffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48fc0f47b0 0x7f48fc1066a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:48:41.463 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:41.463+0000 7f48faffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48fc0f47b0 0x7f48fc1066a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:38666/0 (socket says 172.21.15.134:38666) 2024-01-21T11:48:41.464 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:41.463+0000 7f48faffd700 1 -- 172.21.15.134:0/3319194689 learned_addr learned my addr 172.21.15.134:0/3319194689 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:48:41.465 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:41.463+0000 7f48faffd700 1 -- 172.21.15.134:0/3319194689 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48ec005040 con 0x7f48fc0f47b0 2024-01-21T11:48:41.465 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:41.463+0000 7f48faffd700 1 --2- 172.21.15.134:0/3319194689 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48fc0f47b0 0x7f48fc1066a0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7f48ec013510 tx=0x7f48ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:48:41.465 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:41.463+0000 7f48e37fe700 1 -- 172.21.15.134:0/3319194689 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f48ec009320 con 0x7f48fc0f47b0 2024-01-21T11:48:41.465 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:41.463+0000 7f4901746700 1 -- 172.21.15.134:0/3319194689 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48fc106d90 con 0x7f48fc0f47b0 2024-01-21T11:48:41.465 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:41.463+0000 7f48e37fe700 1 -- 172.21.15.134:0/3319194689 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f48ec013f00 con 0x7f48fc0f47b0 2024-01-21T11:48:41.465 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:41.463+0000 7f48e37fe700 1 -- 172.21.15.134:0/3319194689 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f48ec014d90 con 0x7f48fc0f47b0 2024-01-21T11:48:41.465 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:41.463+0000 7f4901746700 1 -- 172.21.15.134:0/3319194689 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48fc107220 con 0x7f48fc0f47b0 2024-01-21T11:48:41.466 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:41.464+0000 7f48e37fe700 1 -- 172.21.15.134:0/3319194689 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f48ec0263a0 con 0x7f48fc0f47b0 2024-01-21T11:48:41.466 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:41.464+0000 7f4901746700 1 -- 172.21.15.134:0/3319194689 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48fc045ad0 con 0x7f48fc0f47b0 2024-01-21T11:48:41.466 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:41.464+0000 7f48e37fe700 1 --2- 172.21.15.134:0/3319194689 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f48e402ff60 0x7f48e4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:48:41.466 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:41.464+0000 7f48e37fe700 1 -- 172.21.15.134:0/3319194689 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f48ec0412f0 con 0x7f48fc0f47b0 2024-01-21T11:48:41.466 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:41.464+0000 7f48fa7fc700 1 --2- 172.21.15.134:0/3319194689 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f48e402ff60 0x7f48e4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:48:41.466 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:41.465+0000 7f48fa7fc700 1 --2- 172.21.15.134:0/3319194689 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f48e402ff60 0x7f48e4032420 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f48f00096a0 tx=0x7f48f0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:48:41.469 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:41.469+0000 7f48e37fe700 1 -- 172.21.15.134:0/3319194689 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f48ec01cac0 con 0x7f48fc0f47b0 2024-01-21T11:48:41.677 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:41.677+0000 7f4901746700 1 -- 172.21.15.134:0/3319194689 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f48fc1080d0 con 0x7f48fc0f47b0 2024-01-21T11:48:41.678 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:41.678+0000 7f48e37fe700 1 -- 172.21.15.134:0/3319194689 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f48ec01cac0 con 0x7f48fc0f47b0 2024-01-21T11:48:41.678 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:48:41.678 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:48:41.680 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:41.679+0000 7f4901746700 1 -- 172.21.15.134:0/3319194689 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f48e402ff60 msgr2=0x7f48e4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:48:41.680 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:41.679+0000 7f4901746700 1 --2- 172.21.15.134:0/3319194689 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f48e402ff60 0x7f48e4032420 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f48f00096a0 tx=0x7f48f0006b40).stop 2024-01-21T11:48:41.680 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:41.679+0000 7f4901746700 1 -- 172.21.15.134:0/3319194689 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48fc0f47b0 msgr2=0x7f48fc1066a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:48:41.680 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:41.679+0000 7f4901746700 1 --2- 172.21.15.134:0/3319194689 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48fc0f47b0 0x7f48fc1066a0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7f48ec013510 tx=0x7f48ec00b040).stop 2024-01-21T11:48:41.680 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:41.679+0000 7f4901746700 1 -- 172.21.15.134:0/3319194689 shutdown_connections 2024-01-21T11:48:41.680 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:41.679+0000 7f4901746700 1 --2- 172.21.15.134:0/3319194689 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f48e402ff60 0x7f48e4032420 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:48:41.680 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:41.679+0000 7f4901746700 1 --2- 172.21.15.134:0/3319194689 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48fc0f47b0 0x7f48fc1066a0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:48:41.680 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:41.679+0000 7f4901746700 1 -- 172.21.15.134:0/3319194689 >> 172.21.15.134:0/3319194689 conn(0x7f48fc0f02b0 msgr2=0x7f48fc0f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:48:41.680 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:41.679+0000 7f4901746700 1 -- 172.21.15.134:0/3319194689 shutdown_connections 2024-01-21T11:48:41.680 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:41.680+0000 7f4901746700 1 -- 172.21.15.134:0/3319194689 wait complete. 2024-01-21T11:48:41.681 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:48:41.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:48:41 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:48:39.865633+0000 mgr.smithi029.blhxxk (mgr.14164) 205 : cluster [DBG] pgmap v193: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:48:42.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:48:42 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:48:41.678923+0000 mon.smithi029 (mon.0) 272 : audit [DBG] from='client.? 172.21.15.134:0/3319194689' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:48:43.100 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:48:43.101 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:48:43.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:48:43 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:48:41.866003+0000 mgr.smithi029.blhxxk (mgr.14164) 206 : cluster [DBG] pgmap v194: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:48:44.241 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:44.240+0000 7f2d67d54700 1 -- 172.21.15.134:0/2215160781 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d600f6510 msgr2=0x7f2d600f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:48:44.242 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:44.240+0000 7f2d67d54700 1 --2- 172.21.15.134:0/2215160781 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d600f6510 0x7f2d600f6930 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7f2d5c009fb0 tx=0x7f2d5c009b30).stop 2024-01-21T11:48:44.242 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:44.240+0000 7f2d67d54700 1 -- 172.21.15.134:0/2215160781 shutdown_connections 2024-01-21T11:48:44.242 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:44.240+0000 7f2d67d54700 1 --2- 172.21.15.134:0/2215160781 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d600f6510 0x7f2d600f6930 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:48:44.242 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:44.240+0000 7f2d67d54700 1 -- 172.21.15.134:0/2215160781 >> 172.21.15.134:0/2215160781 conn(0x7f2d600f1c70 msgr2=0x7f2d600f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:48:44.242 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:44.240+0000 7f2d67d54700 1 -- 172.21.15.134:0/2215160781 shutdown_connections 2024-01-21T11:48:44.242 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:44.240+0000 7f2d67d54700 1 -- 172.21.15.134:0/2215160781 wait complete. 2024-01-21T11:48:44.242 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:44.240+0000 7f2d67d54700 1 Processor -- start 2024-01-21T11:48:44.242 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:44.241+0000 7f2d67d54700 1 -- start start 2024-01-21T11:48:44.242 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:44.241+0000 7f2d67d54700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d600f6510 0x7f2d60106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:48:44.242 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:44.241+0000 7f2d67d54700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d60106c20 con 0x7f2d600f6510 2024-01-21T11:48:44.242 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:44.241+0000 7f2d65af0700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d600f6510 0x7f2d60106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:48:44.242 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:44.241+0000 7f2d65af0700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d600f6510 0x7f2d60106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:38686/0 (socket says 172.21.15.134:38686) 2024-01-21T11:48:44.242 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:44.241+0000 7f2d65af0700 1 -- 172.21.15.134:0/1839778402 learned_addr learned my addr 172.21.15.134:0/1839778402 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:48:44.243 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:44.241+0000 7f2d65af0700 1 -- 172.21.15.134:0/1839778402 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d5c005040 con 0x7f2d600f6510 2024-01-21T11:48:44.243 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:44.241+0000 7f2d65af0700 1 --2- 172.21.15.134:0/1839778402 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d600f6510 0x7f2d60106730 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f2d5c013510 tx=0x7f2d5c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:48:44.243 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:44.241+0000 7f2d567fc700 1 -- 172.21.15.134:0/1839778402 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d5c009320 con 0x7f2d600f6510 2024-01-21T11:48:44.243 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:44.241+0000 7f2d67d54700 1 -- 172.21.15.134:0/1839778402 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d60106e20 con 0x7f2d600f6510 2024-01-21T11:48:44.243 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:44.241+0000 7f2d567fc700 1 -- 172.21.15.134:0/1839778402 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2d5c013f00 con 0x7f2d600f6510 2024-01-21T11:48:44.243 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:44.241+0000 7f2d567fc700 1 -- 172.21.15.134:0/1839778402 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d5c014d90 con 0x7f2d600f6510 2024-01-21T11:48:44.243 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:44.242+0000 7f2d67d54700 1 -- 172.21.15.134:0/1839778402 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d601072b0 con 0x7f2d600f6510 2024-01-21T11:48:44.244 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:44.242+0000 7f2d567fc700 1 -- 172.21.15.134:0/1839778402 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2d5c0095f0 con 0x7f2d600f6510 2024-01-21T11:48:44.244 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:44.242+0000 7f2d67d54700 1 -- 172.21.15.134:0/1839778402 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d60045ad0 con 0x7f2d600f6510 2024-01-21T11:48:44.244 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:44.242+0000 7f2d567fc700 1 --2- 172.21.15.134:0/1839778402 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f2d4c02ff60 0x7f2d4c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:48:44.244 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:44.242+0000 7f2d567fc700 1 -- 172.21.15.134:0/1839778402 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2d5c041000 con 0x7f2d600f6510 2024-01-21T11:48:44.244 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:44.243+0000 7f2d652ef700 1 --2- 172.21.15.134:0/1839778402 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f2d4c02ff60 0x7f2d4c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:48:44.244 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:44.243+0000 7f2d652ef700 1 --2- 172.21.15.134:0/1839778402 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f2d4c02ff60 0x7f2d4c032420 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f2d500096a0 tx=0x7f2d50006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:48:44.247 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:44.247+0000 7f2d567fc700 1 -- 172.21.15.134:0/1839778402 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2d5c01d7f0 con 0x7f2d600f6510 2024-01-21T11:48:44.456 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:44.455+0000 7f2d67d54700 1 -- 172.21.15.134:0/1839778402 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2d600fbf60 con 0x7f2d600f6510 2024-01-21T11:48:44.456 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:44.456+0000 7f2d567fc700 1 -- 172.21.15.134:0/1839778402 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2d5c010230 con 0x7f2d600f6510 2024-01-21T11:48:44.456 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:48:44.456 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:48:44.458 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:44.458+0000 7f2d67d54700 1 -- 172.21.15.134:0/1839778402 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f2d4c02ff60 msgr2=0x7f2d4c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:48:44.458 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:44.458+0000 7f2d67d54700 1 --2- 172.21.15.134:0/1839778402 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f2d4c02ff60 0x7f2d4c032420 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f2d500096a0 tx=0x7f2d50006b40).stop 2024-01-21T11:48:44.458 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:44.458+0000 7f2d67d54700 1 -- 172.21.15.134:0/1839778402 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d600f6510 msgr2=0x7f2d60106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:48:44.458 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:44.458+0000 7f2d67d54700 1 --2- 172.21.15.134:0/1839778402 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d600f6510 0x7f2d60106730 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f2d5c013510 tx=0x7f2d5c00b040).stop 2024-01-21T11:48:44.458 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:44.458+0000 7f2d67d54700 1 -- 172.21.15.134:0/1839778402 shutdown_connections 2024-01-21T11:48:44.458 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:44.458+0000 7f2d67d54700 1 --2- 172.21.15.134:0/1839778402 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f2d4c02ff60 0x7f2d4c032420 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:48:44.458 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:44.458+0000 7f2d67d54700 1 --2- 172.21.15.134:0/1839778402 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d600f6510 0x7f2d60106730 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:48:44.458 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:44.458+0000 7f2d67d54700 1 -- 172.21.15.134:0/1839778402 >> 172.21.15.134:0/1839778402 conn(0x7f2d600f1c70 msgr2=0x7f2d600f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:48:44.459 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:44.458+0000 7f2d67d54700 1 -- 172.21.15.134:0/1839778402 shutdown_connections 2024-01-21T11:48:44.459 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:44.458+0000 7f2d67d54700 1 -- 172.21.15.134:0/1839778402 wait complete. 2024-01-21T11:48:44.459 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:48:44.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:48:44 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:48:43.866301+0000 mgr.smithi029.blhxxk (mgr.14164) 207 : cluster [DBG] pgmap v195: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:48:44.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:48:44 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:48:44.457316+0000 mon.smithi029 (mon.0) 273 : audit [DBG] from='client.? 172.21.15.134:0/1839778402' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:48:45.845 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:48:45.846 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:48:46.999 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:46.997+0000 7f8a34e50700 1 -- 172.21.15.134:0/2066509599 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a300f6510 msgr2=0x7f8a300f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:48:47.000 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:46.997+0000 7f8a34e50700 1 --2- 172.21.15.134:0/2066509599 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a300f6510 0x7f8a300f6930 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7f8a20009fb0 tx=0x7f8a20009b30).stop 2024-01-21T11:48:47.000 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:46.997+0000 7f8a34e50700 1 -- 172.21.15.134:0/2066509599 shutdown_connections 2024-01-21T11:48:47.000 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:46.997+0000 7f8a34e50700 1 --2- 172.21.15.134:0/2066509599 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a300f6510 0x7f8a300f6930 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:48:47.000 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:46.997+0000 7f8a34e50700 1 -- 172.21.15.134:0/2066509599 >> 172.21.15.134:0/2066509599 conn(0x7f8a300f1c70 msgr2=0x7f8a300f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:48:47.000 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:46.997+0000 7f8a34e50700 1 -- 172.21.15.134:0/2066509599 shutdown_connections 2024-01-21T11:48:47.000 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:46.997+0000 7f8a34e50700 1 -- 172.21.15.134:0/2066509599 wait complete. 2024-01-21T11:48:47.000 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:46.998+0000 7f8a34e50700 1 Processor -- start 2024-01-21T11:48:47.000 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:46.998+0000 7f8a34e50700 1 -- start start 2024-01-21T11:48:47.000 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:46.998+0000 7f8a34e50700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a300f6510 0x7f8a3006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:48:47.000 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:46.998+0000 7f8a34e50700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a30069e20 con 0x7f8a300f6510 2024-01-21T11:48:47.000 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:46.998+0000 7f8a2e59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a300f6510 0x7f8a3006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:48:47.000 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:46.998+0000 7f8a2e59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a300f6510 0x7f8a3006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:50880/0 (socket says 172.21.15.134:50880) 2024-01-21T11:48:47.000 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:46.998+0000 7f8a2e59c700 1 -- 172.21.15.134:0/1703060639 learned_addr learned my addr 172.21.15.134:0/1703060639 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:48:47.001 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:46.999+0000 7f8a2e59c700 1 -- 172.21.15.134:0/1703060639 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a20005040 con 0x7f8a300f6510 2024-01-21T11:48:47.001 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:46.999+0000 7f8a2e59c700 1 --2- 172.21.15.134:0/1703060639 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a300f6510 0x7f8a3006b9a0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7f8a20013510 tx=0x7f8a2000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:48:47.001 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:46.999+0000 7f8a26ffd700 1 -- 172.21.15.134:0/1703060639 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8a20009320 con 0x7f8a300f6510 2024-01-21T11:48:47.001 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:46.999+0000 7f8a26ffd700 1 -- 172.21.15.134:0/1703060639 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8a20013f00 con 0x7f8a300f6510 2024-01-21T11:48:47.001 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:46.999+0000 7f8a34e50700 1 -- 172.21.15.134:0/1703060639 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a30069fc0 con 0x7f8a300f6510 2024-01-21T11:48:47.001 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:46.999+0000 7f8a26ffd700 1 -- 172.21.15.134:0/1703060639 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8a20014d90 con 0x7f8a300f6510 2024-01-21T11:48:47.001 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:46.999+0000 7f8a34e50700 1 -- 172.21.15.134:0/1703060639 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a3006a450 con 0x7f8a300f6510 2024-01-21T11:48:47.002 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:47.000+0000 7f8a34e50700 1 -- 172.21.15.134:0/1703060639 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a30059100 con 0x7f8a300f6510 2024-01-21T11:48:47.002 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:47.000+0000 7f8a26ffd700 1 -- 172.21.15.134:0/1703060639 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8a200095f0 con 0x7f8a300f6510 2024-01-21T11:48:47.002 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:47.000+0000 7f8a26ffd700 1 --2- 172.21.15.134:0/1703060639 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f8a1c02ff60 0x7f8a1c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:48:47.002 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:47.000+0000 7f8a26ffd700 1 -- 172.21.15.134:0/1703060639 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8a20040fe0 con 0x7f8a300f6510 2024-01-21T11:48:47.002 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:47.000+0000 7f8a2dd9b700 1 --2- 172.21.15.134:0/1703060639 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f8a1c02ff60 0x7f8a1c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:48:47.002 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:47.001+0000 7f8a2dd9b700 1 --2- 172.21.15.134:0/1703060639 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f8a1c02ff60 0x7f8a1c032420 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f8a180096a0 tx=0x7f8a18006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:48:47.005 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:47.005+0000 7f8a26ffd700 1 -- 172.21.15.134:0/1703060639 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8a2001d7f0 con 0x7f8a300f6510 2024-01-21T11:48:47.209 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:47.209+0000 7f8a34e50700 1 -- 172.21.15.134:0/1703060639 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8a300fbf60 con 0x7f8a300f6510 2024-01-21T11:48:47.210 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:47.210+0000 7f8a26ffd700 1 -- 172.21.15.134:0/1703060639 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8a20010230 con 0x7f8a300f6510 2024-01-21T11:48:47.210 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:48:47.210 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:48:47.211 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:47.211+0000 7f8a34e50700 1 -- 172.21.15.134:0/1703060639 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f8a1c02ff60 msgr2=0x7f8a1c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:48:47.212 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:47.211+0000 7f8a34e50700 1 --2- 172.21.15.134:0/1703060639 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f8a1c02ff60 0x7f8a1c032420 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f8a180096a0 tx=0x7f8a18006b40).stop 2024-01-21T11:48:47.212 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:47.211+0000 7f8a34e50700 1 -- 172.21.15.134:0/1703060639 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a300f6510 msgr2=0x7f8a3006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:48:47.212 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:47.211+0000 7f8a34e50700 1 --2- 172.21.15.134:0/1703060639 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a300f6510 0x7f8a3006b9a0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7f8a20013510 tx=0x7f8a2000b040).stop 2024-01-21T11:48:47.212 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:47.211+0000 7f8a34e50700 1 -- 172.21.15.134:0/1703060639 shutdown_connections 2024-01-21T11:48:47.212 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:47.211+0000 7f8a34e50700 1 --2- 172.21.15.134:0/1703060639 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f8a1c02ff60 0x7f8a1c032420 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:48:47.212 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:47.211+0000 7f8a34e50700 1 --2- 172.21.15.134:0/1703060639 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a300f6510 0x7f8a3006b9a0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:48:47.212 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:47.211+0000 7f8a34e50700 1 -- 172.21.15.134:0/1703060639 >> 172.21.15.134:0/1703060639 conn(0x7f8a300f1c70 msgr2=0x7f8a300f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:48:47.212 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:47.211+0000 7f8a34e50700 1 -- 172.21.15.134:0/1703060639 shutdown_connections 2024-01-21T11:48:47.212 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:47.211+0000 7f8a34e50700 1 -- 172.21.15.134:0/1703060639 wait complete. 2024-01-21T11:48:47.212 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:48:47.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:48:47 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:48:45.866586+0000 mgr.smithi029.blhxxk (mgr.14164) 208 : cluster [DBG] pgmap v196: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:48:47.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:48:47 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:48:47.210729+0000 mon.smithi029 (mon.0) 274 : audit [DBG] from='client.? 172.21.15.134:0/1703060639' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:48:48.618 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:48:48.619 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:48:49.793 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:49.792+0000 7efcaa49a700 1 -- 172.21.15.134:0/1912539255 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efca40f51a0 msgr2=0x7efca40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:48:49.794 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:49.792+0000 7efcaa49a700 1 --2- 172.21.15.134:0/1912539255 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efca40f51a0 0x7efca40f55c0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7efc94009fb0 tx=0x7efc94009b30).stop 2024-01-21T11:48:49.794 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:49.792+0000 7efcaa49a700 1 -- 172.21.15.134:0/1912539255 shutdown_connections 2024-01-21T11:48:49.794 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:49.792+0000 7efcaa49a700 1 --2- 172.21.15.134:0/1912539255 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efca40f51a0 0x7efca40f55c0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:48:49.794 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:49.792+0000 7efcaa49a700 1 -- 172.21.15.134:0/1912539255 >> 172.21.15.134:0/1912539255 conn(0x7efca40f09a0 msgr2=0x7efca40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:48:49.794 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:49.792+0000 7efcaa49a700 1 -- 172.21.15.134:0/1912539255 shutdown_connections 2024-01-21T11:48:49.794 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:49.792+0000 7efcaa49a700 1 -- 172.21.15.134:0/1912539255 wait complete. 2024-01-21T11:48:49.794 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:49.792+0000 7efcaa49a700 1 Processor -- start 2024-01-21T11:48:49.794 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:49.793+0000 7efcaa49a700 1 -- start start 2024-01-21T11:48:49.794 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:49.793+0000 7efcaa49a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efca40f51a0 0x7efca41053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:48:49.794 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:49.793+0000 7efcaa49a700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efca41058a0 con 0x7efca40f51a0 2024-01-21T11:48:49.794 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:49.793+0000 7efca9498700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efca40f51a0 0x7efca41053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:48:49.794 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:49.793+0000 7efca9498700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efca40f51a0 0x7efca41053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:50902/0 (socket says 172.21.15.134:50902) 2024-01-21T11:48:49.794 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:49.793+0000 7efca9498700 1 -- 172.21.15.134:0/872843232 learned_addr learned my addr 172.21.15.134:0/872843232 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:48:49.795 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:49.793+0000 7efca9498700 1 -- 172.21.15.134:0/872843232 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc94005040 con 0x7efca40f51a0 2024-01-21T11:48:49.795 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:49.794+0000 7efca9498700 1 --2- 172.21.15.134:0/872843232 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efca40f51a0 0x7efca41053b0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7efc94013510 tx=0x7efc9400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:48:49.796 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:49.794+0000 7efc99ffb700 1 -- 172.21.15.134:0/872843232 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efc94009320 con 0x7efca40f51a0 2024-01-21T11:48:49.796 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:49.794+0000 7efc99ffb700 1 -- 172.21.15.134:0/872843232 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efc94013f00 con 0x7efca40f51a0 2024-01-21T11:48:49.796 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:49.794+0000 7efcaa49a700 1 -- 172.21.15.134:0/872843232 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efca4105aa0 con 0x7efca40f51a0 2024-01-21T11:48:49.796 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:49.794+0000 7efc99ffb700 1 -- 172.21.15.134:0/872843232 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efc94014dc0 con 0x7efca40f51a0 2024-01-21T11:48:49.796 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:49.794+0000 7efcaa49a700 1 -- 172.21.15.134:0/872843232 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efca4105f30 con 0x7efca40f51a0 2024-01-21T11:48:49.796 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:49.795+0000 7efc99ffb700 1 -- 172.21.15.134:0/872843232 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efc940095f0 con 0x7efca40f51a0 2024-01-21T11:48:49.796 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:49.795+0000 7efcaa49a700 1 -- 172.21.15.134:0/872843232 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efca4045ad0 con 0x7efca40f51a0 2024-01-21T11:48:49.796 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:49.795+0000 7efc99ffb700 1 --2- 172.21.15.134:0/872843232 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7efc9002ff60 0x7efc90032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:48:49.796 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:49.795+0000 7efc99ffb700 1 -- 172.21.15.134:0/872843232 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efc94041100 con 0x7efca40f51a0 2024-01-21T11:48:49.797 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:49.795+0000 7efca8c97700 1 --2- 172.21.15.134:0/872843232 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7efc9002ff60 0x7efc90032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:48:49.797 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:49.796+0000 7efca8c97700 1 --2- 172.21.15.134:0/872843232 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7efc9002ff60 0x7efc90032420 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7efca00096a0 tx=0x7efca0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:48:49.800 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:49.800+0000 7efc99ffb700 1 -- 172.21.15.134:0/872843232 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efc94014420 con 0x7efca40f51a0 2024-01-21T11:48:49.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:48:49 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:48:47.866894+0000 mgr.smithi029.blhxxk (mgr.14164) 209 : cluster [DBG] pgmap v197: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:48:50.005 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:50.004+0000 7efcaa49a700 1 -- 172.21.15.134:0/872843232 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efca4106de0 con 0x7efca40f51a0 2024-01-21T11:48:50.005 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:50.005+0000 7efc99ffb700 1 -- 172.21.15.134:0/872843232 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7efc94010230 con 0x7efca40f51a0 2024-01-21T11:48:50.005 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:48:50.006 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:48:50.007 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:50.006+0000 7efcaa49a700 1 -- 172.21.15.134:0/872843232 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7efc9002ff60 msgr2=0x7efc90032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:48:50.007 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:50.006+0000 7efcaa49a700 1 --2- 172.21.15.134:0/872843232 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7efc9002ff60 0x7efc90032420 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7efca00096a0 tx=0x7efca0006b40).stop 2024-01-21T11:48:50.007 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:50.006+0000 7efcaa49a700 1 -- 172.21.15.134:0/872843232 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efca40f51a0 msgr2=0x7efca41053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:48:50.007 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:50.006+0000 7efcaa49a700 1 --2- 172.21.15.134:0/872843232 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efca40f51a0 0x7efca41053b0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7efc94013510 tx=0x7efc9400b040).stop 2024-01-21T11:48:50.007 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:50.006+0000 7efcaa49a700 1 -- 172.21.15.134:0/872843232 shutdown_connections 2024-01-21T11:48:50.007 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:50.007+0000 7efcaa49a700 1 --2- 172.21.15.134:0/872843232 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7efc9002ff60 0x7efc90032420 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:48:50.007 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:50.007+0000 7efcaa49a700 1 --2- 172.21.15.134:0/872843232 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efca40f51a0 0x7efca41053b0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:48:50.007 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:50.007+0000 7efcaa49a700 1 -- 172.21.15.134:0/872843232 >> 172.21.15.134:0/872843232 conn(0x7efca40f09a0 msgr2=0x7efca40f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:48:50.007 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:50.007+0000 7efcaa49a700 1 -- 172.21.15.134:0/872843232 shutdown_connections 2024-01-21T11:48:50.008 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:50.007+0000 7efcaa49a700 1 -- 172.21.15.134:0/872843232 wait complete. 2024-01-21T11:48:50.008 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:48:50.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:48:50 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:48:50.006281+0000 mon.smithi029 (mon.0) 275 : audit [DBG] from='client.? 172.21.15.134:0/872843232' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:48:51.393 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:48:51.393 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:48:51.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:48:51 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:48:49.867250+0000 mgr.smithi029.blhxxk (mgr.14164) 210 : cluster [DBG] pgmap v198: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:48:52.547 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:52.547+0000 7fe017fff700 1 -- 172.21.15.134:0/3440947675 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0180f51a0 msgr2=0x7fe0180f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:48:52.548 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:52.547+0000 7fe017fff700 1 --2- 172.21.15.134:0/3440947675 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0180f51a0 0x7fe0180f55c0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7fe00c009fb0 tx=0x7fe00c009b30).stop 2024-01-21T11:48:52.548 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:52.547+0000 7fe017fff700 1 -- 172.21.15.134:0/3440947675 shutdown_connections 2024-01-21T11:48:52.548 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:52.547+0000 7fe017fff700 1 --2- 172.21.15.134:0/3440947675 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0180f51a0 0x7fe0180f55c0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:48:52.548 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:52.547+0000 7fe017fff700 1 -- 172.21.15.134:0/3440947675 >> 172.21.15.134:0/3440947675 conn(0x7fe0180f09a0 msgr2=0x7fe0180f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:48:52.548 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:52.547+0000 7fe017fff700 1 -- 172.21.15.134:0/3440947675 shutdown_connections 2024-01-21T11:48:52.548 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:52.547+0000 7fe017fff700 1 -- 172.21.15.134:0/3440947675 wait complete. 2024-01-21T11:48:52.548 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:52.547+0000 7fe017fff700 1 Processor -- start 2024-01-21T11:48:52.548 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:52.547+0000 7fe017fff700 1 -- start start 2024-01-21T11:48:52.548 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:52.547+0000 7fe017fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0180f51a0 0x7fe0180fd120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:48:52.548 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:52.547+0000 7fe017fff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0180fb5a0 con 0x7fe0180f51a0 2024-01-21T11:48:52.548 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:52.548+0000 7fe016ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0180f51a0 0x7fe0180fd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:48:52.548 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:52.548+0000 7fe016ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0180f51a0 0x7fe0180fd120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:50932/0 (socket says 172.21.15.134:50932) 2024-01-21T11:48:52.549 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:52.548+0000 7fe016ffd700 1 -- 172.21.15.134:0/2867724384 learned_addr learned my addr 172.21.15.134:0/2867724384 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:48:52.549 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:52.548+0000 7fe016ffd700 1 -- 172.21.15.134:0/2867724384 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe00c005040 con 0x7fe0180f51a0 2024-01-21T11:48:52.549 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:52.548+0000 7fe016ffd700 1 --2- 172.21.15.134:0/2867724384 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0180f51a0 0x7fe0180fd120 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7fe00c013510 tx=0x7fe00c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:48:52.549 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:52.548+0000 7fdfff7fe700 1 -- 172.21.15.134:0/2867724384 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe00c009320 con 0x7fe0180f51a0 2024-01-21T11:48:52.549 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:52.548+0000 7fe017fff700 1 -- 172.21.15.134:0/2867724384 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0180fb740 con 0x7fe0180f51a0 2024-01-21T11:48:52.549 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:52.548+0000 7fe017fff700 1 -- 172.21.15.134:0/2867724384 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0180fbbd0 con 0x7fe0180f51a0 2024-01-21T11:48:52.549 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:52.548+0000 7fdfff7fe700 1 -- 172.21.15.134:0/2867724384 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe00c013f00 con 0x7fe0180f51a0 2024-01-21T11:48:52.549 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:52.548+0000 7fdfff7fe700 1 -- 172.21.15.134:0/2867724384 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe00c014d90 con 0x7fe0180f51a0 2024-01-21T11:48:52.550 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:52.549+0000 7fe017fff700 1 -- 172.21.15.134:0/2867724384 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe018059100 con 0x7fe0180f51a0 2024-01-21T11:48:52.550 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:52.549+0000 7fdfff7fe700 1 -- 172.21.15.134:0/2867724384 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe00c0095f0 con 0x7fe0180f51a0 2024-01-21T11:48:52.550 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:52.549+0000 7fdfff7fe700 1 --2- 172.21.15.134:0/2867724384 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fe00002ff60 0x7fe000032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:48:52.550 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:52.549+0000 7fdfff7fe700 1 -- 172.21.15.134:0/2867724384 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe00c0410b0 con 0x7fe0180f51a0 2024-01-21T11:48:52.550 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:52.550+0000 7fe0167fc700 1 --2- 172.21.15.134:0/2867724384 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fe00002ff60 0x7fe000032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:48:52.550 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:52.550+0000 7fe0167fc700 1 --2- 172.21.15.134:0/2867724384 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fe00002ff60 0x7fe000032420 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7fe0080096a0 tx=0x7fe008006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:48:52.553 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:52.554+0000 7fdfff7fe700 1 -- 172.21.15.134:0/2867724384 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe00c014420 con 0x7fe0180f51a0 2024-01-21T11:48:52.759 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:52.760+0000 7fe017fff700 1 -- 172.21.15.134:0/2867724384 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe0180fabe0 con 0x7fe0180f51a0 2024-01-21T11:48:52.759 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:52.760+0000 7fdfff7fe700 1 -- 172.21.15.134:0/2867724384 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe00c010230 con 0x7fe0180f51a0 2024-01-21T11:48:52.759 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:48:52.760 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:48:52.761 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:52.762+0000 7fe017fff700 1 -- 172.21.15.134:0/2867724384 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fe00002ff60 msgr2=0x7fe000032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:48:52.761 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:52.762+0000 7fe017fff700 1 --2- 172.21.15.134:0/2867724384 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fe00002ff60 0x7fe000032420 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7fe0080096a0 tx=0x7fe008006b40).stop 2024-01-21T11:48:52.761 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:52.762+0000 7fe017fff700 1 -- 172.21.15.134:0/2867724384 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0180f51a0 msgr2=0x7fe0180fd120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:48:52.762 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:52.762+0000 7fe017fff700 1 --2- 172.21.15.134:0/2867724384 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0180f51a0 0x7fe0180fd120 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7fe00c013510 tx=0x7fe00c00b040).stop 2024-01-21T11:48:52.762 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:52.762+0000 7fe017fff700 1 -- 172.21.15.134:0/2867724384 shutdown_connections 2024-01-21T11:48:52.762 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:52.762+0000 7fe017fff700 1 --2- 172.21.15.134:0/2867724384 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fe00002ff60 0x7fe000032420 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:48:52.762 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:52.762+0000 7fe017fff700 1 --2- 172.21.15.134:0/2867724384 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0180f51a0 0x7fe0180fd120 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:48:52.762 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:52.762+0000 7fe017fff700 1 -- 172.21.15.134:0/2867724384 >> 172.21.15.134:0/2867724384 conn(0x7fe0180f09a0 msgr2=0x7fe01805fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:48:52.762 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:52.762+0000 7fe017fff700 1 -- 172.21.15.134:0/2867724384 shutdown_connections 2024-01-21T11:48:52.762 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:52.762+0000 7fe017fff700 1 -- 172.21.15.134:0/2867724384 wait complete. 2024-01-21T11:48:52.762 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:48:53.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:48:53 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:48:51.867614+0000 mgr.smithi029.blhxxk (mgr.14164) 211 : cluster [DBG] pgmap v199: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:48:53.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:48:53 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:48:52.760511+0000 mon.smithi029 (mon.0) 276 : audit [DBG] from='client.? 172.21.15.134:0/2867724384' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:48:54.144 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:48:54.144 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:48:55.272 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:55.270+0000 7f1c82ccc700 1 -- 172.21.15.134:0/1356627655 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c7c0f5410 msgr2=0x7f1c7c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:48:55.272 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:55.270+0000 7f1c82ccc700 1 --2- 172.21.15.134:0/1356627655 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c7c0f5410 0x7f1c7c0f5830 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7f1c70009fb0 tx=0x7f1c70009b30).stop 2024-01-21T11:48:55.272 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:55.270+0000 7f1c82ccc700 1 -- 172.21.15.134:0/1356627655 shutdown_connections 2024-01-21T11:48:55.272 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:55.270+0000 7f1c82ccc700 1 --2- 172.21.15.134:0/1356627655 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c7c0f5410 0x7f1c7c0f5830 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:48:55.272 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:55.270+0000 7f1c82ccc700 1 -- 172.21.15.134:0/1356627655 >> 172.21.15.134:0/1356627655 conn(0x7f1c7c0f0b70 msgr2=0x7f1c7c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:48:55.273 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:55.270+0000 7f1c82ccc700 1 -- 172.21.15.134:0/1356627655 shutdown_connections 2024-01-21T11:48:55.273 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:55.270+0000 7f1c82ccc700 1 -- 172.21.15.134:0/1356627655 wait complete. 2024-01-21T11:48:55.273 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:55.271+0000 7f1c82ccc700 1 Processor -- start 2024-01-21T11:48:55.273 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:55.271+0000 7f1c82ccc700 1 -- start start 2024-01-21T11:48:55.273 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:55.271+0000 7f1c82ccc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c7c0f5410 0x7f1c7c0601b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:48:55.273 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:55.271+0000 7f1c82ccc700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c7c1093f0 con 0x7f1c7c0f5410 2024-01-21T11:48:55.273 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:55.271+0000 7f1c80a68700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c7c0f5410 0x7f1c7c0601b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:48:55.273 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:55.271+0000 7f1c80a68700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c7c0f5410 0x7f1c7c0601b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:33250/0 (socket says 172.21.15.134:33250) 2024-01-21T11:48:55.273 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:55.271+0000 7f1c80a68700 1 -- 172.21.15.134:0/44855666 learned_addr learned my addr 172.21.15.134:0/44855666 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:48:55.273 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:55.272+0000 7f1c80a68700 1 -- 172.21.15.134:0/44855666 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c70005040 con 0x7f1c7c0f5410 2024-01-21T11:48:55.273 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:55.272+0000 7f1c80a68700 1 --2- 172.21.15.134:0/44855666 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c7c0f5410 0x7f1c7c0601b0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f1c70013510 tx=0x7f1c7000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:48:55.273 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:55.272+0000 7f1c797fa700 1 -- 172.21.15.134:0/44855666 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1c70009320 con 0x7f1c7c0f5410 2024-01-21T11:48:55.273 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:55.272+0000 7f1c82ccc700 1 -- 172.21.15.134:0/44855666 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c7c109530 con 0x7f1c7c0f5410 2024-01-21T11:48:55.273 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:55.272+0000 7f1c797fa700 1 -- 172.21.15.134:0/44855666 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1c70013f00 con 0x7f1c7c0f5410 2024-01-21T11:48:55.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:55.272+0000 7f1c82ccc700 1 -- 172.21.15.134:0/44855666 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c7c109a20 con 0x7f1c7c0f5410 2024-01-21T11:48:55.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:55.272+0000 7f1c797fa700 1 -- 172.21.15.134:0/44855666 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1c70014d90 con 0x7f1c7c0f5410 2024-01-21T11:48:55.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:55.273+0000 7f1c797fa700 1 -- 172.21.15.134:0/44855666 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1c700263a0 con 0x7f1c7c0f5410 2024-01-21T11:48:55.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:55.273+0000 7f1c797fa700 1 --2- 172.21.15.134:0/44855666 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f1c6c02ff60 0x7f1c6c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:48:55.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:55.273+0000 7f1c66ffd700 1 -- 172.21.15.134:0/44855666 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c54004fa0 con 0x7f1c7c0f5410 2024-01-21T11:48:55.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:55.273+0000 7f1c797fa700 1 -- 172.21.15.134:0/44855666 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1c70041240 con 0x7f1c7c0f5410 2024-01-21T11:48:55.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:55.273+0000 7f1c7bfff700 1 --2- 172.21.15.134:0/44855666 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f1c6c02ff60 0x7f1c6c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:48:55.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:55.274+0000 7f1c7bfff700 1 --2- 172.21.15.134:0/44855666 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f1c6c02ff60 0x7f1c6c032420 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f1c680096a0 tx=0x7f1c68006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:48:55.278 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:55.278+0000 7f1c797fa700 1 -- 172.21.15.134:0/44855666 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1c70012920 con 0x7f1c7c0f5410 2024-01-21T11:48:55.486 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:55.486+0000 7f1c66ffd700 1 -- 172.21.15.134:0/44855666 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1c54005670 con 0x7f1c7c0f5410 2024-01-21T11:48:55.487 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:55.487+0000 7f1c797fa700 1 -- 172.21.15.134:0/44855666 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1c70012330 con 0x7f1c7c0f5410 2024-01-21T11:48:55.487 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:48:55.487 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:48:55.488 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:55.488+0000 7f1c66ffd700 1 -- 172.21.15.134:0/44855666 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f1c6c02ff60 msgr2=0x7f1c6c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:48:55.488 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:55.488+0000 7f1c66ffd700 1 --2- 172.21.15.134:0/44855666 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f1c6c02ff60 0x7f1c6c032420 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f1c680096a0 tx=0x7f1c68006b40).stop 2024-01-21T11:48:55.489 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:55.488+0000 7f1c66ffd700 1 -- 172.21.15.134:0/44855666 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c7c0f5410 msgr2=0x7f1c7c0601b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:48:55.489 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:55.488+0000 7f1c66ffd700 1 --2- 172.21.15.134:0/44855666 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c7c0f5410 0x7f1c7c0601b0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f1c70013510 tx=0x7f1c7000b040).stop 2024-01-21T11:48:55.489 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:55.488+0000 7f1c66ffd700 1 -- 172.21.15.134:0/44855666 shutdown_connections 2024-01-21T11:48:55.489 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:55.488+0000 7f1c66ffd700 1 --2- 172.21.15.134:0/44855666 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f1c6c02ff60 0x7f1c6c032420 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:48:55.489 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:55.488+0000 7f1c66ffd700 1 --2- 172.21.15.134:0/44855666 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c7c0f5410 0x7f1c7c0601b0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:48:55.489 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:55.488+0000 7f1c66ffd700 1 -- 172.21.15.134:0/44855666 >> 172.21.15.134:0/44855666 conn(0x7f1c7c0f0b70 msgr2=0x7f1c7c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:48:55.489 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:55.488+0000 7f1c66ffd700 1 -- 172.21.15.134:0/44855666 shutdown_connections 2024-01-21T11:48:55.489 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:55.488+0000 7f1c66ffd700 1 -- 172.21.15.134:0/44855666 wait complete. 2024-01-21T11:48:55.490 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:48:55.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:48:55 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:48:53.867974+0000 mgr.smithi029.blhxxk (mgr.14164) 212 : cluster [DBG] pgmap v200: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:48:55.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:48:55 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:48:55.487845+0000 mon.smithi029 (mon.0) 277 : audit [DBG] from='client.? 172.21.15.134:0/44855666' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:48:56.851 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:48:56.852 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:48:57.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:48:57 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:48:55.868329+0000 mgr.smithi029.blhxxk (mgr.14164) 213 : cluster [DBG] pgmap v201: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:48:57.947 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:57.946+0000 7fa74905f700 1 -- 172.21.15.134:0/2916993007 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa744060280 msgr2=0x7fa7440f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:48:57.947 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:57.946+0000 7fa74905f700 1 --2- 172.21.15.134:0/2916993007 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa744060280 0x7fa7440f90f0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7fa734009fb0 tx=0x7fa734009b30).stop 2024-01-21T11:48:57.947 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:57.946+0000 7fa74905f700 1 -- 172.21.15.134:0/2916993007 shutdown_connections 2024-01-21T11:48:57.947 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:57.946+0000 7fa74905f700 1 --2- 172.21.15.134:0/2916993007 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa744060280 0x7fa7440f90f0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:48:57.947 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:57.946+0000 7fa74905f700 1 -- 172.21.15.134:0/2916993007 >> 172.21.15.134:0/2916993007 conn(0x7fa7440f0b70 msgr2=0x7fa7440f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:48:57.947 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:57.946+0000 7fa74905f700 1 -- 172.21.15.134:0/2916993007 shutdown_connections 2024-01-21T11:48:57.947 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:57.946+0000 7fa74905f700 1 -- 172.21.15.134:0/2916993007 wait complete. 2024-01-21T11:48:57.948 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:57.947+0000 7fa74905f700 1 Processor -- start 2024-01-21T11:48:57.948 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:57.947+0000 7fa74905f700 1 -- start start 2024-01-21T11:48:57.948 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:57.948+0000 7fa74905f700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa744060280 0x7fa7441066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:48:57.949 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:57.948+0000 7fa74905f700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa744106bd0 con 0x7fa744060280 2024-01-21T11:48:57.949 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:57.948+0000 7fa742d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa744060280 0x7fa7441066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:48:57.949 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:57.948+0000 7fa742d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa744060280 0x7fa7441066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:33264/0 (socket says 172.21.15.134:33264) 2024-01-21T11:48:57.949 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:57.948+0000 7fa742d9d700 1 -- 172.21.15.134:0/1777321781 learned_addr learned my addr 172.21.15.134:0/1777321781 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:48:57.949 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:57.948+0000 7fa742d9d700 1 -- 172.21.15.134:0/1777321781 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa734005040 con 0x7fa744060280 2024-01-21T11:48:57.949 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:57.948+0000 7fa742d9d700 1 --2- 172.21.15.134:0/1777321781 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa744060280 0x7fa7441066e0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7fa734013510 tx=0x7fa73400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:48:57.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:57.949+0000 7fa72b7fe700 1 -- 172.21.15.134:0/1777321781 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa734009320 con 0x7fa744060280 2024-01-21T11:48:57.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:57.949+0000 7fa74905f700 1 -- 172.21.15.134:0/1777321781 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa744106dd0 con 0x7fa744060280 2024-01-21T11:48:57.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:57.949+0000 7fa72b7fe700 1 -- 172.21.15.134:0/1777321781 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa734013f00 con 0x7fa744060280 2024-01-21T11:48:57.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:57.949+0000 7fa72b7fe700 1 -- 172.21.15.134:0/1777321781 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa734014dc0 con 0x7fa744060280 2024-01-21T11:48:57.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:57.949+0000 7fa74905f700 1 -- 172.21.15.134:0/1777321781 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa744107260 con 0x7fa744060280 2024-01-21T11:48:57.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:57.949+0000 7fa74905f700 1 -- 172.21.15.134:0/1777321781 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa744045ad0 con 0x7fa744060280 2024-01-21T11:48:57.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:57.949+0000 7fa72b7fe700 1 -- 172.21.15.134:0/1777321781 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa7340095f0 con 0x7fa744060280 2024-01-21T11:48:57.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:57.950+0000 7fa72b7fe700 1 --2- 172.21.15.134:0/1777321781 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fa72c02ff60 0x7fa72c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:48:57.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:57.950+0000 7fa72b7fe700 1 -- 172.21.15.134:0/1777321781 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa734041030 con 0x7fa744060280 2024-01-21T11:48:57.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:57.950+0000 7fa74259c700 1 --2- 172.21.15.134:0/1777321781 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fa72c02ff60 0x7fa72c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:48:57.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:57.950+0000 7fa74259c700 1 --2- 172.21.15.134:0/1777321781 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fa72c02ff60 0x7fa72c032420 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7fa7380096a0 tx=0x7fa738006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:48:57.954 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:57.954+0000 7fa72b7fe700 1 -- 172.21.15.134:0/1777321781 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa734014420 con 0x7fa744060280 2024-01-21T11:48:58.161 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:58.160+0000 7fa74905f700 1 -- 172.21.15.134:0/1777321781 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa744108110 con 0x7fa744060280 2024-01-21T11:48:58.162 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:58.161+0000 7fa72b7fe700 1 -- 172.21.15.134:0/1777321781 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa734010230 con 0x7fa744060280 2024-01-21T11:48:58.162 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:48:58.162 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:48:58.163 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:58.163+0000 7fa74905f700 1 -- 172.21.15.134:0/1777321781 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fa72c02ff60 msgr2=0x7fa72c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:48:58.163 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:58.163+0000 7fa74905f700 1 --2- 172.21.15.134:0/1777321781 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fa72c02ff60 0x7fa72c032420 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7fa7380096a0 tx=0x7fa738006b40).stop 2024-01-21T11:48:58.163 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:58.163+0000 7fa74905f700 1 -- 172.21.15.134:0/1777321781 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa744060280 msgr2=0x7fa7441066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:48:58.163 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:58.163+0000 7fa74905f700 1 --2- 172.21.15.134:0/1777321781 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa744060280 0x7fa7441066e0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7fa734013510 tx=0x7fa73400b040).stop 2024-01-21T11:48:58.163 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:58.163+0000 7fa74905f700 1 -- 172.21.15.134:0/1777321781 shutdown_connections 2024-01-21T11:48:58.163 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:58.163+0000 7fa74905f700 1 --2- 172.21.15.134:0/1777321781 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fa72c02ff60 0x7fa72c032420 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:48:58.164 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:58.163+0000 7fa74905f700 1 --2- 172.21.15.134:0/1777321781 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa744060280 0x7fa7441066e0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:48:58.164 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:58.163+0000 7fa74905f700 1 -- 172.21.15.134:0/1777321781 >> 172.21.15.134:0/1777321781 conn(0x7fa7440f0b70 msgr2=0x7fa7440f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:48:58.164 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:58.163+0000 7fa74905f700 1 -- 172.21.15.134:0/1777321781 shutdown_connections 2024-01-21T11:48:58.164 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:48:58.163+0000 7fa74905f700 1 -- 172.21.15.134:0/1777321781 wait complete. 2024-01-21T11:48:58.164 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:48:58.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:48:58 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:48:58.162271+0000 mon.smithi029 (mon.0) 278 : audit [DBG] from='client.? 172.21.15.134:0/1777321781' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:48:59.543 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:48:59.543 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:48:59.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:48:59 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:48:57.868628+0000 mgr.smithi029.blhxxk (mgr.14164) 214 : cluster [DBG] pgmap v202: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:49:00.694 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:00.692+0000 7f2f5d322700 1 -- 172.21.15.134:0/558746986 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f580f6510 msgr2=0x7f2f580f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:49:00.694 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:00.692+0000 7f2f5d322700 1 --2- 172.21.15.134:0/558746986 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f580f6510 0x7f2f580f6930 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7f2f4c009fb0 tx=0x7f2f4c009b30).stop 2024-01-21T11:49:00.695 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:00.692+0000 7f2f5d322700 1 -- 172.21.15.134:0/558746986 shutdown_connections 2024-01-21T11:49:00.695 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:00.692+0000 7f2f5d322700 1 --2- 172.21.15.134:0/558746986 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f580f6510 0x7f2f580f6930 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:49:00.695 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:00.692+0000 7f2f5d322700 1 -- 172.21.15.134:0/558746986 >> 172.21.15.134:0/558746986 conn(0x7f2f580f1c70 msgr2=0x7f2f580f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:49:00.695 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:00.692+0000 7f2f5d322700 1 -- 172.21.15.134:0/558746986 shutdown_connections 2024-01-21T11:49:00.695 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:00.692+0000 7f2f5d322700 1 -- 172.21.15.134:0/558746986 wait complete. 2024-01-21T11:49:00.695 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:00.693+0000 7f2f5d322700 1 Processor -- start 2024-01-21T11:49:00.695 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:00.693+0000 7f2f5d322700 1 -- start start 2024-01-21T11:49:00.695 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:00.693+0000 7f2f5d322700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f580f6510 0x7f2f58106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:49:00.695 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:00.693+0000 7f2f5d322700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f58106c20 con 0x7f2f580f6510 2024-01-21T11:49:00.695 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:00.693+0000 7f2f56ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f580f6510 0x7f2f58106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:49:00.695 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:00.693+0000 7f2f56ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f580f6510 0x7f2f58106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:33282/0 (socket says 172.21.15.134:33282) 2024-01-21T11:49:00.695 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:00.693+0000 7f2f56ffd700 1 -- 172.21.15.134:0/4017561968 learned_addr learned my addr 172.21.15.134:0/4017561968 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:49:00.695 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:00.694+0000 7f2f56ffd700 1 -- 172.21.15.134:0/4017561968 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f4c005040 con 0x7f2f580f6510 2024-01-21T11:49:00.695 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:00.694+0000 7f2f56ffd700 1 --2- 172.21.15.134:0/4017561968 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f580f6510 0x7f2f58106730 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f2f4c013510 tx=0x7f2f4c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:49:00.696 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:00.694+0000 7f2f3f7fe700 1 -- 172.21.15.134:0/4017561968 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2f4c009320 con 0x7f2f580f6510 2024-01-21T11:49:00.696 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:00.694+0000 7f2f5d322700 1 -- 172.21.15.134:0/4017561968 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f58106e20 con 0x7f2f580f6510 2024-01-21T11:49:00.696 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:00.694+0000 7f2f3f7fe700 1 -- 172.21.15.134:0/4017561968 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2f4c013f00 con 0x7f2f580f6510 2024-01-21T11:49:00.696 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:00.694+0000 7f2f3f7fe700 1 -- 172.21.15.134:0/4017561968 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2f4c014d90 con 0x7f2f580f6510 2024-01-21T11:49:00.696 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:00.694+0000 7f2f5d322700 1 -- 172.21.15.134:0/4017561968 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f581072b0 con 0x7f2f580f6510 2024-01-21T11:49:00.697 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:00.695+0000 7f2f3f7fe700 1 -- 172.21.15.134:0/4017561968 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2f4c0095f0 con 0x7f2f580f6510 2024-01-21T11:49:00.697 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:00.695+0000 7f2f5d322700 1 -- 172.21.15.134:0/4017561968 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f58045ad0 con 0x7f2f580f6510 2024-01-21T11:49:00.697 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:00.695+0000 7f2f3f7fe700 1 --2- 172.21.15.134:0/4017561968 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f2f4002ff60 0x7f2f40032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:49:00.697 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:00.695+0000 7f2f3f7fe700 1 -- 172.21.15.134:0/4017561968 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2f4c040fe0 con 0x7f2f580f6510 2024-01-21T11:49:00.697 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:00.695+0000 7f2f567fc700 1 --2- 172.21.15.134:0/4017561968 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f2f4002ff60 0x7f2f40032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:49:00.697 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:00.695+0000 7f2f567fc700 1 --2- 172.21.15.134:0/4017561968 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f2f4002ff60 0x7f2f40032420 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f2f480096a0 tx=0x7f2f48006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:49:00.700 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:00.700+0000 7f2f3f7fe700 1 -- 172.21.15.134:0/4017561968 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2f4c01d7f0 con 0x7f2f580f6510 2024-01-21T11:49:00.903 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:00.903+0000 7f2f5d322700 1 -- 172.21.15.134:0/4017561968 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2f580fbf60 con 0x7f2f580f6510 2024-01-21T11:49:00.904 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:00.903+0000 7f2f3f7fe700 1 -- 172.21.15.134:0/4017561968 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2f4c010230 con 0x7f2f580f6510 2024-01-21T11:49:00.904 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:49:00.904 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:49:00.906 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:00.905+0000 7f2f5d322700 1 -- 172.21.15.134:0/4017561968 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f2f4002ff60 msgr2=0x7f2f40032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:49:00.906 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:00.905+0000 7f2f5d322700 1 --2- 172.21.15.134:0/4017561968 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f2f4002ff60 0x7f2f40032420 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f2f480096a0 tx=0x7f2f48006b40).stop 2024-01-21T11:49:00.906 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:00.905+0000 7f2f5d322700 1 -- 172.21.15.134:0/4017561968 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f580f6510 msgr2=0x7f2f58106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:49:00.906 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:00.905+0000 7f2f5d322700 1 --2- 172.21.15.134:0/4017561968 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f580f6510 0x7f2f58106730 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f2f4c013510 tx=0x7f2f4c00b040).stop 2024-01-21T11:49:00.906 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:00.905+0000 7f2f5d322700 1 -- 172.21.15.134:0/4017561968 shutdown_connections 2024-01-21T11:49:00.906 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:00.905+0000 7f2f5d322700 1 --2- 172.21.15.134:0/4017561968 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f2f4002ff60 0x7f2f40032420 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:49:00.906 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:00.905+0000 7f2f5d322700 1 --2- 172.21.15.134:0/4017561968 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f580f6510 0x7f2f58106730 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:49:00.906 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:00.905+0000 7f2f5d322700 1 -- 172.21.15.134:0/4017561968 >> 172.21.15.134:0/4017561968 conn(0x7f2f580f1c70 msgr2=0x7f2f580f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:49:00.906 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:00.905+0000 7f2f5d322700 1 -- 172.21.15.134:0/4017561968 shutdown_connections 2024-01-21T11:49:00.906 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:00.905+0000 7f2f5d322700 1 -- 172.21.15.134:0/4017561968 wait complete. 2024-01-21T11:49:00.906 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:49:01.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:49:01 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:48:59.868977+0000 mgr.smithi029.blhxxk (mgr.14164) 215 : cluster [DBG] pgmap v203: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:49:01.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:49:01 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:49:00.904748+0000 mon.smithi029 (mon.0) 279 : audit [DBG] from='client.? 172.21.15.134:0/4017561968' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:49:02.291 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:49:02.292 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:49:02.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:49:02 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:49:01.944064+0000 mon.smithi029 (mon.0) 280 : audit [INF] from='mgr.14164 172.21.15.29:0/3252518221' entity='mgr.smithi029.blhxxk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi029.blhxxk/mirror_snapshot_schedule"}]: dispatch 2024-01-21T11:49:02.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:49:02 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:49:01.945600+0000 mon.smithi029 (mon.0) 281 : audit [INF] from='mgr.14164 172.21.15.29:0/3252518221' entity='mgr.smithi029.blhxxk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi029.blhxxk/trash_purge_schedule"}]: dispatch 2024-01-21T11:49:03.499 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:03.497+0000 7fa240614700 1 -- 172.21.15.134:0/4143434186 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2380f5410 msgr2=0x7fa2380f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:49:03.500 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:03.497+0000 7fa240614700 1 --2- 172.21.15.134:0/4143434186 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2380f5410 0x7fa2380f5830 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7fa234009fb0 tx=0x7fa234009b30).stop 2024-01-21T11:49:03.500 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:03.497+0000 7fa240614700 1 -- 172.21.15.134:0/4143434186 shutdown_connections 2024-01-21T11:49:03.500 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:03.497+0000 7fa240614700 1 --2- 172.21.15.134:0/4143434186 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2380f5410 0x7fa2380f5830 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:49:03.500 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:03.497+0000 7fa240614700 1 -- 172.21.15.134:0/4143434186 >> 172.21.15.134:0/4143434186 conn(0x7fa2380f0b70 msgr2=0x7fa2380f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:49:03.500 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:03.497+0000 7fa240614700 1 -- 172.21.15.134:0/4143434186 shutdown_connections 2024-01-21T11:49:03.500 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:03.497+0000 7fa240614700 1 -- 172.21.15.134:0/4143434186 wait complete. 2024-01-21T11:49:03.500 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:03.498+0000 7fa240614700 1 Processor -- start 2024-01-21T11:49:03.500 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:03.498+0000 7fa240614700 1 -- start start 2024-01-21T11:49:03.500 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:03.498+0000 7fa240614700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2380f5410 0x7fa238109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:49:03.500 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:03.498+0000 7fa240614700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa238060240 con 0x7fa2380f5410 2024-01-21T11:49:03.500 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:03.498+0000 7fa23e3b0700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2380f5410 0x7fa238109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:49:03.500 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:03.498+0000 7fa23e3b0700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2380f5410 0x7fa238109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:33304/0 (socket says 172.21.15.134:33304) 2024-01-21T11:49:03.500 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:03.498+0000 7fa23e3b0700 1 -- 172.21.15.134:0/2183475642 learned_addr learned my addr 172.21.15.134:0/2183475642 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:49:03.501 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:03.499+0000 7fa23e3b0700 1 -- 172.21.15.134:0/2183475642 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa234005040 con 0x7fa2380f5410 2024-01-21T11:49:03.501 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:03.499+0000 7fa23e3b0700 1 --2- 172.21.15.134:0/2183475642 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2380f5410 0x7fa238109430 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7fa234013510 tx=0x7fa23400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:49:03.501 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:03.499+0000 7fa22effd700 1 -- 172.21.15.134:0/2183475642 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa234009320 con 0x7fa2380f5410 2024-01-21T11:49:03.501 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:03.499+0000 7fa240614700 1 -- 172.21.15.134:0/2183475642 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa238060440 con 0x7fa2380f5410 2024-01-21T11:49:03.501 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:03.499+0000 7fa22effd700 1 -- 172.21.15.134:0/2183475642 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa234013f00 con 0x7fa2380f5410 2024-01-21T11:49:03.501 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:03.499+0000 7fa22effd700 1 -- 172.21.15.134:0/2183475642 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa234014d90 con 0x7fa2380f5410 2024-01-21T11:49:03.501 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:03.499+0000 7fa240614700 1 -- 172.21.15.134:0/2183475642 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa238109ae0 con 0x7fa2380f5410 2024-01-21T11:49:03.502 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:03.500+0000 7fa22effd700 1 -- 172.21.15.134:0/2183475642 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa2340095f0 con 0x7fa2380f5410 2024-01-21T11:49:03.502 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:03.500+0000 7fa240614700 1 -- 172.21.15.134:0/2183475642 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa238045ad0 con 0x7fa2380f5410 2024-01-21T11:49:03.502 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:03.500+0000 7fa22effd700 1 --2- 172.21.15.134:0/2183475642 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fa22402ff60 0x7fa224032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:49:03.502 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:03.500+0000 7fa22effd700 1 -- 172.21.15.134:0/2183475642 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa234041000 con 0x7fa2380f5410 2024-01-21T11:49:03.502 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:03.500+0000 7fa23dbaf700 1 --2- 172.21.15.134:0/2183475642 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fa22402ff60 0x7fa224032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:49:03.502 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:03.501+0000 7fa23dbaf700 1 --2- 172.21.15.134:0/2183475642 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fa22402ff60 0x7fa224032420 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7fa2280096a0 tx=0x7fa228006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:49:03.505 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:03.505+0000 7fa22effd700 1 -- 172.21.15.134:0/2183475642 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa234014420 con 0x7fa2380f5410 2024-01-21T11:49:03.710 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:03.710+0000 7fa240614700 1 -- 172.21.15.134:0/2183475642 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa2380fae60 con 0x7fa2380f5410 2024-01-21T11:49:03.711 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:03.710+0000 7fa22effd700 1 -- 172.21.15.134:0/2183475642 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa23401d7f0 con 0x7fa2380f5410 2024-01-21T11:49:03.711 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:49:03.711 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:49:03.712 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:03.712+0000 7fa240614700 1 -- 172.21.15.134:0/2183475642 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fa22402ff60 msgr2=0x7fa224032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:49:03.712 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:03.712+0000 7fa240614700 1 --2- 172.21.15.134:0/2183475642 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fa22402ff60 0x7fa224032420 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7fa2280096a0 tx=0x7fa228006b40).stop 2024-01-21T11:49:03.713 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:03.712+0000 7fa240614700 1 -- 172.21.15.134:0/2183475642 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2380f5410 msgr2=0x7fa238109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:49:03.713 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:03.712+0000 7fa240614700 1 --2- 172.21.15.134:0/2183475642 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2380f5410 0x7fa238109430 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7fa234013510 tx=0x7fa23400b040).stop 2024-01-21T11:49:03.713 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:03.712+0000 7fa240614700 1 -- 172.21.15.134:0/2183475642 shutdown_connections 2024-01-21T11:49:03.713 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:03.712+0000 7fa240614700 1 --2- 172.21.15.134:0/2183475642 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fa22402ff60 0x7fa224032420 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:49:03.713 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:03.712+0000 7fa240614700 1 --2- 172.21.15.134:0/2183475642 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2380f5410 0x7fa238109430 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:49:03.713 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:03.712+0000 7fa240614700 1 -- 172.21.15.134:0/2183475642 >> 172.21.15.134:0/2183475642 conn(0x7fa2380f0b70 msgr2=0x7fa2380f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:49:03.713 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:03.712+0000 7fa240614700 1 -- 172.21.15.134:0/2183475642 shutdown_connections 2024-01-21T11:49:03.713 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:03.712+0000 7fa240614700 1 -- 172.21.15.134:0/2183475642 wait complete. 2024-01-21T11:49:03.713 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:49:03.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:49:03 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:49:01.869319+0000 mgr.smithi029.blhxxk (mgr.14164) 216 : cluster [DBG] pgmap v204: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:49:04.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:49:04 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:49:03.711780+0000 mon.smithi029 (mon.0) 282 : audit [DBG] from='client.? 172.21.15.134:0/2183475642' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:49:05.095 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:49:05.096 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:49:05.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:49:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:49:03.869671+0000 mgr.smithi029.blhxxk (mgr.14164) 217 : cluster [DBG] pgmap v205: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:49:06.316 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:06.314+0000 7f5896ca3700 1 -- 172.21.15.134:0/279201643 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58900f5410 msgr2=0x7f58900f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:49:06.317 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:06.314+0000 7f5896ca3700 1 --2- 172.21.15.134:0/279201643 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58900f5410 0x7f58900f5830 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7f5884009fb0 tx=0x7f5884009b30).stop 2024-01-21T11:49:06.317 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:06.314+0000 7f5896ca3700 1 -- 172.21.15.134:0/279201643 shutdown_connections 2024-01-21T11:49:06.317 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:06.314+0000 7f5896ca3700 1 --2- 172.21.15.134:0/279201643 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58900f5410 0x7f58900f5830 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:49:06.317 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:06.314+0000 7f5896ca3700 1 -- 172.21.15.134:0/279201643 >> 172.21.15.134:0/279201643 conn(0x7f58900f0b70 msgr2=0x7f58900f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:49:06.317 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:06.314+0000 7f5896ca3700 1 -- 172.21.15.134:0/279201643 shutdown_connections 2024-01-21T11:49:06.317 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:06.314+0000 7f5896ca3700 1 -- 172.21.15.134:0/279201643 wait complete. 2024-01-21T11:49:06.317 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:06.315+0000 7f5896ca3700 1 Processor -- start 2024-01-21T11:49:06.317 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:06.315+0000 7f5896ca3700 1 -- start start 2024-01-21T11:49:06.317 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:06.315+0000 7f5896ca3700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58900f5410 0x7f5890109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:49:06.317 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:06.315+0000 7f5896ca3700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5890060220 con 0x7f58900f5410 2024-01-21T11:49:06.317 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:06.315+0000 7f5894a3f700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58900f5410 0x7f5890109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:49:06.317 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:06.315+0000 7f5894a3f700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58900f5410 0x7f5890109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:32988/0 (socket says 172.21.15.134:32988) 2024-01-21T11:49:06.317 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:06.315+0000 7f5894a3f700 1 -- 172.21.15.134:0/3995059234 learned_addr learned my addr 172.21.15.134:0/3995059234 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:49:06.318 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:06.316+0000 7f5894a3f700 1 -- 172.21.15.134:0/3995059234 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5884005040 con 0x7f58900f5410 2024-01-21T11:49:06.318 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:06.316+0000 7f5894a3f700 1 --2- 172.21.15.134:0/3995059234 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58900f5410 0x7f5890109450 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f5884013510 tx=0x7f588400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:49:06.318 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:06.316+0000 7f588d7fa700 1 -- 172.21.15.134:0/3995059234 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5884009320 con 0x7f58900f5410 2024-01-21T11:49:06.318 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:06.316+0000 7f5896ca3700 1 -- 172.21.15.134:0/3995059234 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5890060420 con 0x7f58900f5410 2024-01-21T11:49:06.318 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:06.316+0000 7f588d7fa700 1 -- 172.21.15.134:0/3995059234 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5884013f00 con 0x7f58900f5410 2024-01-21T11:49:06.318 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:06.316+0000 7f588d7fa700 1 -- 172.21.15.134:0/3995059234 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5884014d90 con 0x7f58900f5410 2024-01-21T11:49:06.318 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:06.316+0000 7f5896ca3700 1 -- 172.21.15.134:0/3995059234 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5890109b00 con 0x7f58900f5410 2024-01-21T11:49:06.319 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:06.317+0000 7f588d7fa700 1 -- 172.21.15.134:0/3995059234 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5884014420 con 0x7f58900f5410 2024-01-21T11:49:06.319 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:06.317+0000 7f5896ca3700 1 -- 172.21.15.134:0/3995059234 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5890059100 con 0x7f58900f5410 2024-01-21T11:49:06.319 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:06.317+0000 7f588d7fa700 1 --2- 172.21.15.134:0/3995059234 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f588002ff60 0x7f5880032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:49:06.319 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:06.317+0000 7f588d7fa700 1 -- 172.21.15.134:0/3995059234 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f58840410c0 con 0x7f58900f5410 2024-01-21T11:49:06.319 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:06.317+0000 7f588ffff700 1 --2- 172.21.15.134:0/3995059234 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f588002ff60 0x7f5880032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:49:06.319 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:06.318+0000 7f588ffff700 1 --2- 172.21.15.134:0/3995059234 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f588002ff60 0x7f5880032420 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f587c0096a0 tx=0x7f587c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:49:06.322 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:06.322+0000 7f588d7fa700 1 -- 172.21.15.134:0/3995059234 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f58840095f0 con 0x7f58900f5410 2024-01-21T11:49:06.524 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:06.524+0000 7f5896ca3700 1 -- 172.21.15.134:0/3995059234 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5890045ad0 con 0x7f58900f5410 2024-01-21T11:49:06.525 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:06.525+0000 7f588d7fa700 1 -- 172.21.15.134:0/3995059234 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f588401db70 con 0x7f58900f5410 2024-01-21T11:49:06.525 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:49:06.526 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:49:06.527 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:06.526+0000 7f5896ca3700 1 -- 172.21.15.134:0/3995059234 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f588002ff60 msgr2=0x7f5880032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:49:06.527 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:06.526+0000 7f5896ca3700 1 --2- 172.21.15.134:0/3995059234 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f588002ff60 0x7f5880032420 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f587c0096a0 tx=0x7f587c006b40).stop 2024-01-21T11:49:06.527 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:06.526+0000 7f5896ca3700 1 -- 172.21.15.134:0/3995059234 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58900f5410 msgr2=0x7f5890109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:49:06.527 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:06.526+0000 7f5896ca3700 1 --2- 172.21.15.134:0/3995059234 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58900f5410 0x7f5890109450 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f5884013510 tx=0x7f588400b040).stop 2024-01-21T11:49:06.527 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:06.527+0000 7f5896ca3700 1 -- 172.21.15.134:0/3995059234 shutdown_connections 2024-01-21T11:49:06.528 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:06.527+0000 7f5896ca3700 1 --2- 172.21.15.134:0/3995059234 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f588002ff60 0x7f5880032420 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:49:06.528 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:06.527+0000 7f5896ca3700 1 --2- 172.21.15.134:0/3995059234 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58900f5410 0x7f5890109450 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:49:06.528 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:06.527+0000 7f5896ca3700 1 -- 172.21.15.134:0/3995059234 >> 172.21.15.134:0/3995059234 conn(0x7f58900f0b70 msgr2=0x7f58900f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:49:06.528 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:06.527+0000 7f5896ca3700 1 -- 172.21.15.134:0/3995059234 shutdown_connections 2024-01-21T11:49:06.528 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:06.527+0000 7f5896ca3700 1 -- 172.21.15.134:0/3995059234 wait complete. 2024-01-21T11:49:06.528 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:49:06.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:49:06 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:49:06.526062+0000 mon.smithi029 (mon.0) 283 : audit [DBG] from='client.? 172.21.15.134:0/3995059234' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:49:07.903 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:49:07.904 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:49:07.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:49:07 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:49:05.869944+0000 mgr.smithi029.blhxxk (mgr.14164) 218 : cluster [DBG] pgmap v206: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:49:09.024 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:09.022+0000 7f8d5337f700 1 -- 172.21.15.134:0/2465002461 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d4c0f5410 msgr2=0x7f8d4c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:49:09.024 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:09.022+0000 7f8d5337f700 1 --2- 172.21.15.134:0/2465002461 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d4c0f5410 0x7f8d4c0f5830 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7f8d3c009fb0 tx=0x7f8d3c009b30).stop 2024-01-21T11:49:09.024 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:09.023+0000 7f8d5337f700 1 -- 172.21.15.134:0/2465002461 shutdown_connections 2024-01-21T11:49:09.025 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:09.023+0000 7f8d5337f700 1 --2- 172.21.15.134:0/2465002461 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d4c0f5410 0x7f8d4c0f5830 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:49:09.025 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:09.023+0000 7f8d5337f700 1 -- 172.21.15.134:0/2465002461 >> 172.21.15.134:0/2465002461 conn(0x7f8d4c0f0b70 msgr2=0x7f8d4c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:49:09.025 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:09.023+0000 7f8d5337f700 1 -- 172.21.15.134:0/2465002461 shutdown_connections 2024-01-21T11:49:09.025 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:09.023+0000 7f8d5337f700 1 -- 172.21.15.134:0/2465002461 wait complete. 2024-01-21T11:49:09.025 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:09.023+0000 7f8d5337f700 1 Processor -- start 2024-01-21T11:49:09.025 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:09.023+0000 7f8d5337f700 1 -- start start 2024-01-21T11:49:09.025 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:09.023+0000 7f8d5337f700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d4c0f5410 0x7f8d4c109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:49:09.025 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:09.023+0000 7f8d5337f700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d4c060220 con 0x7f8d4c0f5410 2024-01-21T11:49:09.025 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:09.023+0000 7f8d5111b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d4c0f5410 0x7f8d4c109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:49:09.025 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:09.023+0000 7f8d5111b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d4c0f5410 0x7f8d4c109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:33004/0 (socket says 172.21.15.134:33004) 2024-01-21T11:49:09.025 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:09.023+0000 7f8d5111b700 1 -- 172.21.15.134:0/1638243748 learned_addr learned my addr 172.21.15.134:0/1638243748 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:49:09.025 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:09.024+0000 7f8d5111b700 1 -- 172.21.15.134:0/1638243748 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d3c005040 con 0x7f8d4c0f5410 2024-01-21T11:49:09.026 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:09.024+0000 7f8d5111b700 1 --2- 172.21.15.134:0/1638243748 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d4c0f5410 0x7f8d4c109450 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f8d3c013510 tx=0x7f8d3c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:49:09.026 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:09.024+0000 7f8d41ffb700 1 -- 172.21.15.134:0/1638243748 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8d3c009320 con 0x7f8d4c0f5410 2024-01-21T11:49:09.026 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:09.024+0000 7f8d5337f700 1 -- 172.21.15.134:0/1638243748 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d4c060420 con 0x7f8d4c0f5410 2024-01-21T11:49:09.026 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:09.024+0000 7f8d41ffb700 1 -- 172.21.15.134:0/1638243748 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8d3c013f00 con 0x7f8d4c0f5410 2024-01-21T11:49:09.026 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:09.024+0000 7f8d41ffb700 1 -- 172.21.15.134:0/1638243748 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8d3c014d90 con 0x7f8d4c0f5410 2024-01-21T11:49:09.026 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:09.024+0000 7f8d5337f700 1 -- 172.21.15.134:0/1638243748 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d4c109b00 con 0x7f8d4c0f5410 2024-01-21T11:49:09.027 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:09.025+0000 7f8d41ffb700 1 -- 172.21.15.134:0/1638243748 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8d3c0095f0 con 0x7f8d4c0f5410 2024-01-21T11:49:09.027 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:09.025+0000 7f8d5337f700 1 -- 172.21.15.134:0/1638243748 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d4c059100 con 0x7f8d4c0f5410 2024-01-21T11:49:09.027 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:09.025+0000 7f8d41ffb700 1 --2- 172.21.15.134:0/1638243748 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f8d3802ff60 0x7f8d38032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:49:09.027 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:09.025+0000 7f8d41ffb700 1 -- 172.21.15.134:0/1638243748 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8d3c0410c0 con 0x7f8d4c0f5410 2024-01-21T11:49:09.027 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:09.025+0000 7f8d5091a700 1 --2- 172.21.15.134:0/1638243748 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f8d3802ff60 0x7f8d38032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:49:09.027 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:09.026+0000 7f8d5091a700 1 --2- 172.21.15.134:0/1638243748 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f8d3802ff60 0x7f8d38032420 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f8d480096a0 tx=0x7f8d48006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:49:09.030 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:09.030+0000 7f8d41ffb700 1 -- 172.21.15.134:0/1638243748 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8d3c014420 con 0x7f8d4c0f5410 2024-01-21T11:49:09.236 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:09.236+0000 7f8d5337f700 1 -- 172.21.15.134:0/1638243748 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8d4c045ad0 con 0x7f8d4c0f5410 2024-01-21T11:49:09.237 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:09.237+0000 7f8d41ffb700 1 -- 172.21.15.134:0/1638243748 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8d3c01d7f0 con 0x7f8d4c0f5410 2024-01-21T11:49:09.237 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:49:09.237 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:49:09.239 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:09.238+0000 7f8d5337f700 1 -- 172.21.15.134:0/1638243748 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f8d3802ff60 msgr2=0x7f8d38032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:49:09.239 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:09.238+0000 7f8d5337f700 1 --2- 172.21.15.134:0/1638243748 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f8d3802ff60 0x7f8d38032420 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f8d480096a0 tx=0x7f8d48006b40).stop 2024-01-21T11:49:09.239 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:09.238+0000 7f8d5337f700 1 -- 172.21.15.134:0/1638243748 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d4c0f5410 msgr2=0x7f8d4c109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:49:09.239 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:09.238+0000 7f8d5337f700 1 --2- 172.21.15.134:0/1638243748 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d4c0f5410 0x7f8d4c109450 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f8d3c013510 tx=0x7f8d3c00b040).stop 2024-01-21T11:49:09.239 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:09.238+0000 7f8d5337f700 1 -- 172.21.15.134:0/1638243748 shutdown_connections 2024-01-21T11:49:09.239 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:09.238+0000 7f8d5337f700 1 --2- 172.21.15.134:0/1638243748 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f8d3802ff60 0x7f8d38032420 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:49:09.239 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:09.238+0000 7f8d5337f700 1 --2- 172.21.15.134:0/1638243748 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d4c0f5410 0x7f8d4c109450 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:49:09.239 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:09.238+0000 7f8d5337f700 1 -- 172.21.15.134:0/1638243748 >> 172.21.15.134:0/1638243748 conn(0x7f8d4c0f0b70 msgr2=0x7f8d4c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:49:09.239 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:09.239+0000 7f8d5337f700 1 -- 172.21.15.134:0/1638243748 shutdown_connections 2024-01-21T11:49:09.240 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:09.239+0000 7f8d5337f700 1 -- 172.21.15.134:0/1638243748 wait complete. 2024-01-21T11:49:09.240 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:49:09.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:49:09 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:49:07.870265+0000 mgr.smithi029.blhxxk (mgr.14164) 219 : cluster [DBG] pgmap v207: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:49:09.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:49:09 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:49:09.237876+0000 mon.smithi029 (mon.0) 284 : audit [DBG] from='client.? 172.21.15.134:0/1638243748' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:49:10.622 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:49:10.623 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:49:11.798 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:11.796+0000 7f777d5df700 1 -- 172.21.15.134:0/1377524811 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77780f6510 msgr2=0x7f77780f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:49:11.798 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:11.796+0000 7f777d5df700 1 --2- 172.21.15.134:0/1377524811 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77780f6510 0x7f77780f6930 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7f7768009fb0 tx=0x7f7768009b30).stop 2024-01-21T11:49:11.798 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:11.796+0000 7f777d5df700 1 -- 172.21.15.134:0/1377524811 shutdown_connections 2024-01-21T11:49:11.798 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:11.796+0000 7f777d5df700 1 --2- 172.21.15.134:0/1377524811 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77780f6510 0x7f77780f6930 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:49:11.798 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:11.796+0000 7f777d5df700 1 -- 172.21.15.134:0/1377524811 >> 172.21.15.134:0/1377524811 conn(0x7f77780f1c70 msgr2=0x7f77780f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:49:11.799 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:11.796+0000 7f777d5df700 1 -- 172.21.15.134:0/1377524811 shutdown_connections 2024-01-21T11:49:11.799 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:11.796+0000 7f777d5df700 1 -- 172.21.15.134:0/1377524811 wait complete. 2024-01-21T11:49:11.799 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:11.797+0000 7f777d5df700 1 Processor -- start 2024-01-21T11:49:11.799 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:11.797+0000 7f777d5df700 1 -- start start 2024-01-21T11:49:11.799 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:11.797+0000 7f777d5df700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77780f6510 0x7f7778106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:49:11.799 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:11.797+0000 7f777d5df700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7778106c20 con 0x7f77780f6510 2024-01-21T11:49:11.799 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:11.797+0000 7f7776ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77780f6510 0x7f7778106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:49:11.799 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:11.797+0000 7f7776ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77780f6510 0x7f7778106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:33020/0 (socket says 172.21.15.134:33020) 2024-01-21T11:49:11.799 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:11.797+0000 7f7776ffd700 1 -- 172.21.15.134:0/208770742 learned_addr learned my addr 172.21.15.134:0/208770742 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:49:11.799 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:11.797+0000 7f7776ffd700 1 -- 172.21.15.134:0/208770742 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7768005040 con 0x7f77780f6510 2024-01-21T11:49:11.799 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:11.798+0000 7f7776ffd700 1 --2- 172.21.15.134:0/208770742 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77780f6510 0x7f7778106730 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f7768013510 tx=0x7f776800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:49:11.799 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:11.798+0000 7f775f7fe700 1 -- 172.21.15.134:0/208770742 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7768009320 con 0x7f77780f6510 2024-01-21T11:49:11.799 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:11.798+0000 7f777d5df700 1 -- 172.21.15.134:0/208770742 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7778106e20 con 0x7f77780f6510 2024-01-21T11:49:11.799 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:11.798+0000 7f775f7fe700 1 -- 172.21.15.134:0/208770742 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7768013f00 con 0x7f77780f6510 2024-01-21T11:49:11.799 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:11.798+0000 7f777d5df700 1 -- 172.21.15.134:0/208770742 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77781072b0 con 0x7f77780f6510 2024-01-21T11:49:11.800 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:11.798+0000 7f775f7fe700 1 -- 172.21.15.134:0/208770742 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7768014d90 con 0x7f77780f6510 2024-01-21T11:49:11.800 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:11.799+0000 7f775f7fe700 1 -- 172.21.15.134:0/208770742 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f77680263a0 con 0x7f77780f6510 2024-01-21T11:49:11.800 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:11.799+0000 7f777d5df700 1 -- 172.21.15.134:0/208770742 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7778045ad0 con 0x7f77780f6510 2024-01-21T11:49:11.801 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:11.799+0000 7f775f7fe700 1 --2- 172.21.15.134:0/208770742 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f776002ff60 0x7f7760032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:49:11.801 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:11.799+0000 7f775f7fe700 1 -- 172.21.15.134:0/208770742 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7768041040 con 0x7f77780f6510 2024-01-21T11:49:11.801 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:11.799+0000 7f77767fc700 1 --2- 172.21.15.134:0/208770742 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f776002ff60 0x7f7760032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:49:11.801 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:11.799+0000 7f77767fc700 1 --2- 172.21.15.134:0/208770742 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f776002ff60 0x7f7760032420 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f776c0096a0 tx=0x7f776c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:49:11.804 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:11.804+0000 7f775f7fe700 1 -- 172.21.15.134:0/208770742 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f77680095f0 con 0x7f77780f6510 2024-01-21T11:49:11.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:49:11 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:49:09.870581+0000 mgr.smithi029.blhxxk (mgr.14164) 220 : cluster [DBG] pgmap v208: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:49:12.009 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:12.009+0000 7f777d5df700 1 -- 172.21.15.134:0/208770742 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f77780fbf60 con 0x7f77780f6510 2024-01-21T11:49:12.010 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:12.010+0000 7f775f7fe700 1 -- 172.21.15.134:0/208770742 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f776801db70 con 0x7f77780f6510 2024-01-21T11:49:12.010 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:49:12.010 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:49:12.012 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:12.011+0000 7f777d5df700 1 -- 172.21.15.134:0/208770742 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f776002ff60 msgr2=0x7f7760032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:49:12.012 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:12.011+0000 7f777d5df700 1 --2- 172.21.15.134:0/208770742 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f776002ff60 0x7f7760032420 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f776c0096a0 tx=0x7f776c006b40).stop 2024-01-21T11:49:12.012 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:12.011+0000 7f777d5df700 1 -- 172.21.15.134:0/208770742 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77780f6510 msgr2=0x7f7778106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:49:12.012 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:12.011+0000 7f777d5df700 1 --2- 172.21.15.134:0/208770742 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77780f6510 0x7f7778106730 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f7768013510 tx=0x7f776800b040).stop 2024-01-21T11:49:12.012 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:12.011+0000 7f777d5df700 1 -- 172.21.15.134:0/208770742 shutdown_connections 2024-01-21T11:49:12.012 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:12.011+0000 7f777d5df700 1 --2- 172.21.15.134:0/208770742 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f776002ff60 0x7f7760032420 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:49:12.013 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:12.011+0000 7f777d5df700 1 --2- 172.21.15.134:0/208770742 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77780f6510 0x7f7778106730 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:49:12.013 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:12.011+0000 7f777d5df700 1 -- 172.21.15.134:0/208770742 >> 172.21.15.134:0/208770742 conn(0x7f77780f1c70 msgr2=0x7f77780f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:49:12.013 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:12.011+0000 7f777d5df700 1 -- 172.21.15.134:0/208770742 shutdown_connections 2024-01-21T11:49:12.013 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:12.011+0000 7f777d5df700 1 -- 172.21.15.134:0/208770742 wait complete. 2024-01-21T11:49:12.013 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:49:12.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:49:12 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:49:12.011120+0000 mon.smithi029 (mon.0) 285 : audit [DBG] from='client.? 172.21.15.134:0/208770742' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:49:13.389 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:49:13.389 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:49:13.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:49:13 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:49:11.870957+0000 mgr.smithi029.blhxxk (mgr.14164) 221 : cluster [DBG] pgmap v209: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:49:14.531 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:14.529+0000 7f9aefe77700 1 -- 172.21.15.134:0/324852936 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ae80f6510 msgr2=0x7f9ae80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:49:14.531 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:14.529+0000 7f9aefe77700 1 --2- 172.21.15.134:0/324852936 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ae80f6510 0x7f9ae80f6930 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7f9ad8009fb0 tx=0x7f9ad8009b30).stop 2024-01-21T11:49:14.531 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:14.529+0000 7f9aefe77700 1 -- 172.21.15.134:0/324852936 shutdown_connections 2024-01-21T11:49:14.531 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:14.529+0000 7f9aefe77700 1 --2- 172.21.15.134:0/324852936 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ae80f6510 0x7f9ae80f6930 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:49:14.531 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:14.529+0000 7f9aefe77700 1 -- 172.21.15.134:0/324852936 >> 172.21.15.134:0/324852936 conn(0x7f9ae80f1c70 msgr2=0x7f9ae80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:49:14.531 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:14.529+0000 7f9aefe77700 1 -- 172.21.15.134:0/324852936 shutdown_connections 2024-01-21T11:49:14.531 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:14.530+0000 7f9aefe77700 1 -- 172.21.15.134:0/324852936 wait complete. 2024-01-21T11:49:14.531 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:14.530+0000 7f9aefe77700 1 Processor -- start 2024-01-21T11:49:14.532 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:14.530+0000 7f9aefe77700 1 -- start start 2024-01-21T11:49:14.532 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:14.530+0000 7f9aefe77700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ae80f6510 0x7f9ae8106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:49:14.532 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:14.530+0000 7f9aefe77700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae8106c20 con 0x7f9ae80f6510 2024-01-21T11:49:14.532 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:14.531+0000 7f9aedc13700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ae80f6510 0x7f9ae8106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:49:14.532 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:14.531+0000 7f9aedc13700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ae80f6510 0x7f9ae8106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:41538/0 (socket says 172.21.15.134:41538) 2024-01-21T11:49:14.532 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:14.531+0000 7f9aedc13700 1 -- 172.21.15.134:0/1800988402 learned_addr learned my addr 172.21.15.134:0/1800988402 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:49:14.533 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:14.531+0000 7f9aedc13700 1 -- 172.21.15.134:0/1800988402 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ad8005040 con 0x7f9ae80f6510 2024-01-21T11:49:14.534 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:14.531+0000 7f9aedc13700 1 --2- 172.21.15.134:0/1800988402 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ae80f6510 0x7f9ae8106730 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7f9ad8013510 tx=0x7f9ad800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:49:14.534 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:14.531+0000 7f9ade7fc700 1 -- 172.21.15.134:0/1800988402 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ad8009320 con 0x7f9ae80f6510 2024-01-21T11:49:14.534 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:14.531+0000 7f9aefe77700 1 -- 172.21.15.134:0/1800988402 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ae8106e20 con 0x7f9ae80f6510 2024-01-21T11:49:14.534 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:14.531+0000 7f9ade7fc700 1 -- 172.21.15.134:0/1800988402 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9ad8013f00 con 0x7f9ae80f6510 2024-01-21T11:49:14.534 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:14.531+0000 7f9ade7fc700 1 -- 172.21.15.134:0/1800988402 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ad8014d90 con 0x7f9ae80f6510 2024-01-21T11:49:14.534 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:14.531+0000 7f9aefe77700 1 -- 172.21.15.134:0/1800988402 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ae81072b0 con 0x7f9ae80f6510 2024-01-21T11:49:14.534 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:14.532+0000 7f9ade7fc700 1 -- 172.21.15.134:0/1800988402 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9ad80095f0 con 0x7f9ae80f6510 2024-01-21T11:49:14.534 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:14.532+0000 7f9aefe77700 1 -- 172.21.15.134:0/1800988402 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ae8045ad0 con 0x7f9ae80f6510 2024-01-21T11:49:14.534 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:14.532+0000 7f9ade7fc700 1 --2- 172.21.15.134:0/1800988402 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f9ad402ff60 0x7f9ad4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:49:14.534 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:14.532+0000 7f9ade7fc700 1 -- 172.21.15.134:0/1800988402 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9ad8041080 con 0x7f9ae80f6510 2024-01-21T11:49:14.534 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:14.533+0000 7f9aed412700 1 --2- 172.21.15.134:0/1800988402 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f9ad402ff60 0x7f9ad4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:49:14.534 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:14.533+0000 7f9aed412700 1 --2- 172.21.15.134:0/1800988402 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f9ad402ff60 0x7f9ad4032420 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f9ae40096a0 tx=0x7f9ae4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:49:14.537 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:14.537+0000 7f9ade7fc700 1 -- 172.21.15.134:0/1800988402 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9ad8014420 con 0x7f9ae80f6510 2024-01-21T11:49:14.743 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:14.743+0000 7f9aefe77700 1 -- 172.21.15.134:0/1800988402 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9ae80fbf60 con 0x7f9ae80f6510 2024-01-21T11:49:14.744 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:14.744+0000 7f9ade7fc700 1 -- 172.21.15.134:0/1800988402 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9ad8010230 con 0x7f9ae80f6510 2024-01-21T11:49:14.744 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:49:14.744 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:49:14.746 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:14.745+0000 7f9aefe77700 1 -- 172.21.15.134:0/1800988402 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f9ad402ff60 msgr2=0x7f9ad4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:49:14.746 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:14.745+0000 7f9aefe77700 1 --2- 172.21.15.134:0/1800988402 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f9ad402ff60 0x7f9ad4032420 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f9ae40096a0 tx=0x7f9ae4006b40).stop 2024-01-21T11:49:14.746 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:14.745+0000 7f9aefe77700 1 -- 172.21.15.134:0/1800988402 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ae80f6510 msgr2=0x7f9ae8106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:49:14.746 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:14.745+0000 7f9aefe77700 1 --2- 172.21.15.134:0/1800988402 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ae80f6510 0x7f9ae8106730 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7f9ad8013510 tx=0x7f9ad800b040).stop 2024-01-21T11:49:14.746 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:14.745+0000 7f9aefe77700 1 -- 172.21.15.134:0/1800988402 shutdown_connections 2024-01-21T11:49:14.746 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:14.745+0000 7f9aefe77700 1 --2- 172.21.15.134:0/1800988402 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f9ad402ff60 0x7f9ad4032420 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:49:14.746 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:14.745+0000 7f9aefe77700 1 --2- 172.21.15.134:0/1800988402 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ae80f6510 0x7f9ae8106730 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:49:14.746 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:14.745+0000 7f9aefe77700 1 -- 172.21.15.134:0/1800988402 >> 172.21.15.134:0/1800988402 conn(0x7f9ae80f1c70 msgr2=0x7f9ae80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:49:14.746 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:14.745+0000 7f9aefe77700 1 -- 172.21.15.134:0/1800988402 shutdown_connections 2024-01-21T11:49:14.746 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:14.745+0000 7f9aefe77700 1 -- 172.21.15.134:0/1800988402 wait complete. 2024-01-21T11:49:14.747 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:49:15.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:49:15 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:49:13.871211+0000 mgr.smithi029.blhxxk (mgr.14164) 222 : cluster [DBG] pgmap v210: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:49:15.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:49:15 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:49:14.744920+0000 mon.smithi029 (mon.0) 286 : audit [DBG] from='client.? 172.21.15.134:0/1800988402' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:49:16.188 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:49:16.188 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:49:17.365 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:17.360+0000 7f6c91b0e700 1 -- 172.21.15.134:0/4252108039 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c8c0f51a0 msgr2=0x7f6c8c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:49:17.365 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:17.360+0000 7f6c91b0e700 1 --2- 172.21.15.134:0/4252108039 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c8c0f51a0 0x7f6c8c0f55c0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7f6c80009fb0 tx=0x7f6c80009b30).stop 2024-01-21T11:49:17.366 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:17.360+0000 7f6c91b0e700 1 -- 172.21.15.134:0/4252108039 shutdown_connections 2024-01-21T11:49:17.366 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:17.360+0000 7f6c91b0e700 1 --2- 172.21.15.134:0/4252108039 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c8c0f51a0 0x7f6c8c0f55c0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:49:17.366 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:17.360+0000 7f6c91b0e700 1 -- 172.21.15.134:0/4252108039 >> 172.21.15.134:0/4252108039 conn(0x7f6c8c0f09a0 msgr2=0x7f6c8c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:49:17.366 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:17.360+0000 7f6c91b0e700 1 -- 172.21.15.134:0/4252108039 shutdown_connections 2024-01-21T11:49:17.366 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:17.360+0000 7f6c91b0e700 1 -- 172.21.15.134:0/4252108039 wait complete. 2024-01-21T11:49:17.366 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:17.360+0000 7f6c91b0e700 1 Processor -- start 2024-01-21T11:49:17.366 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:17.360+0000 7f6c91b0e700 1 -- start start 2024-01-21T11:49:17.366 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:17.361+0000 7f6c91b0e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c8c0f51a0 0x7f6c8c1091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:49:17.366 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:17.361+0000 7f6c91b0e700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c8c060220 con 0x7f6c8c0f51a0 2024-01-21T11:49:17.366 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:17.361+0000 7f6c90b0c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c8c0f51a0 0x7f6c8c1091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:49:17.366 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:17.361+0000 7f6c90b0c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c8c0f51a0 0x7f6c8c1091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:41560/0 (socket says 172.21.15.134:41560) 2024-01-21T11:49:17.366 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:17.361+0000 7f6c90b0c700 1 -- 172.21.15.134:0/3853153942 learned_addr learned my addr 172.21.15.134:0/3853153942 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:49:17.366 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:17.361+0000 7f6c90b0c700 1 -- 172.21.15.134:0/3853153942 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c80005040 con 0x7f6c8c0f51a0 2024-01-21T11:49:17.367 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:17.361+0000 7f6c90b0c700 1 --2- 172.21.15.134:0/3853153942 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c8c0f51a0 0x7f6c8c1091d0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f6c80009830 tx=0x7f6c8000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:49:17.367 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:17.361+0000 7f6c897fa700 1 -- 172.21.15.134:0/3853153942 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c80009320 con 0x7f6c8c0f51a0 2024-01-21T11:49:17.367 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:17.361+0000 7f6c91b0e700 1 -- 172.21.15.134:0/3853153942 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c8c060420 con 0x7f6c8c0f51a0 2024-01-21T11:49:17.367 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:17.362+0000 7f6c897fa700 1 -- 172.21.15.134:0/3853153942 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6c800136c0 con 0x7f6c8c0f51a0 2024-01-21T11:49:17.367 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:17.362+0000 7f6c897fa700 1 -- 172.21.15.134:0/3853153942 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c8001ee90 con 0x7f6c8c0f51a0 2024-01-21T11:49:17.367 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:17.362+0000 7f6c91b0e700 1 -- 172.21.15.134:0/3853153942 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c8c109880 con 0x7f6c8c0f51a0 2024-01-21T11:49:17.367 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:17.362+0000 7f6c897fa700 1 -- 172.21.15.134:0/3853153942 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6c8001e420 con 0x7f6c8c0f51a0 2024-01-21T11:49:17.367 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:17.362+0000 7f6c91b0e700 1 -- 172.21.15.134:0/3853153942 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c8c059100 con 0x7f6c8c0f51a0 2024-01-21T11:49:17.367 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:17.362+0000 7f6c897fa700 1 --2- 172.21.15.134:0/3853153942 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f6c7c02ff10 0x7f6c7c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:49:17.367 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:17.363+0000 7f6c897fa700 1 -- 172.21.15.134:0/3853153942 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6c80041010 con 0x7f6c8c0f51a0 2024-01-21T11:49:17.367 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:17.363+0000 7f6c8bfff700 1 --2- 172.21.15.134:0/3853153942 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f6c7c02ff10 0x7f6c7c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:49:17.367 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:17.363+0000 7f6c8bfff700 1 --2- 172.21.15.134:0/3853153942 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f6c7c02ff10 0x7f6c7c0323d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f6c780096a0 tx=0x7f6c78006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:49:17.368 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:17.367+0000 7f6c897fa700 1 -- 172.21.15.134:0/3853153942 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6c800095f0 con 0x7f6c8c0f51a0 2024-01-21T11:49:17.574 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:17.573+0000 7f6c91b0e700 1 -- 172.21.15.134:0/3853153942 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6c8c0fabe0 con 0x7f6c8c0f51a0 2024-01-21T11:49:17.574 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:17.574+0000 7f6c897fa700 1 -- 172.21.15.134:0/3853153942 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6c8001cb10 con 0x7f6c8c0f51a0 2024-01-21T11:49:17.574 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:49:17.574 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:49:17.576 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:17.576+0000 7f6c91b0e700 1 -- 172.21.15.134:0/3853153942 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f6c7c02ff10 msgr2=0x7f6c7c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:49:17.576 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:17.576+0000 7f6c91b0e700 1 --2- 172.21.15.134:0/3853153942 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f6c7c02ff10 0x7f6c7c0323d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f6c780096a0 tx=0x7f6c78006b40).stop 2024-01-21T11:49:17.576 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:17.576+0000 7f6c91b0e700 1 -- 172.21.15.134:0/3853153942 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c8c0f51a0 msgr2=0x7f6c8c1091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:49:17.576 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:17.576+0000 7f6c91b0e700 1 --2- 172.21.15.134:0/3853153942 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c8c0f51a0 0x7f6c8c1091d0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f6c80009830 tx=0x7f6c8000b040).stop 2024-01-21T11:49:17.576 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:17.576+0000 7f6c91b0e700 1 -- 172.21.15.134:0/3853153942 shutdown_connections 2024-01-21T11:49:17.576 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:17.576+0000 7f6c91b0e700 1 --2- 172.21.15.134:0/3853153942 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f6c7c02ff10 0x7f6c7c0323d0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:49:17.576 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:17.576+0000 7f6c91b0e700 1 --2- 172.21.15.134:0/3853153942 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c8c0f51a0 0x7f6c8c1091d0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:49:17.577 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:17.576+0000 7f6c91b0e700 1 -- 172.21.15.134:0/3853153942 >> 172.21.15.134:0/3853153942 conn(0x7f6c8c0f09a0 msgr2=0x7f6c8c0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:49:17.577 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:17.576+0000 7f6c91b0e700 1 -- 172.21.15.134:0/3853153942 shutdown_connections 2024-01-21T11:49:17.577 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:17.576+0000 7f6c91b0e700 1 -- 172.21.15.134:0/3853153942 wait complete. 2024-01-21T11:49:17.577 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:49:17.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:49:17 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:49:15.871452+0000 mgr.smithi029.blhxxk (mgr.14164) 223 : cluster [DBG] pgmap v211: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:49:17.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:49:17 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:49:17.575238+0000 mon.smithi029 (mon.0) 287 : audit [DBG] from='client.? 172.21.15.134:0/3853153942' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:49:19.024 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:49:19.024 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:49:19.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:49:19 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:49:17.871685+0000 mgr.smithi029.blhxxk (mgr.14164) 224 : cluster [DBG] pgmap v212: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:49:20.169 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:20.167+0000 7f154abf4700 1 -- 172.21.15.134:0/1090734306 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15440f5410 msgr2=0x7f15440f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:49:20.169 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:20.167+0000 7f154abf4700 1 --2- 172.21.15.134:0/1090734306 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15440f5410 0x7f15440f5830 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7f1538009fb0 tx=0x7f1538009b30).stop 2024-01-21T11:49:20.169 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:20.167+0000 7f154abf4700 1 -- 172.21.15.134:0/1090734306 shutdown_connections 2024-01-21T11:49:20.169 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:20.167+0000 7f154abf4700 1 --2- 172.21.15.134:0/1090734306 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15440f5410 0x7f15440f5830 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:49:20.169 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:20.167+0000 7f154abf4700 1 -- 172.21.15.134:0/1090734306 >> 172.21.15.134:0/1090734306 conn(0x7f15440f0b70 msgr2=0x7f15440f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:49:20.169 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:20.167+0000 7f154abf4700 1 -- 172.21.15.134:0/1090734306 shutdown_connections 2024-01-21T11:49:20.169 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:20.167+0000 7f154abf4700 1 -- 172.21.15.134:0/1090734306 wait complete. 2024-01-21T11:49:20.170 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:20.168+0000 7f154abf4700 1 Processor -- start 2024-01-21T11:49:20.170 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:20.168+0000 7f154abf4700 1 -- start start 2024-01-21T11:49:20.170 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:20.168+0000 7f154abf4700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15440f5410 0x7f1544109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:49:20.170 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:20.168+0000 7f154abf4700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1544060220 con 0x7f15440f5410 2024-01-21T11:49:20.170 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:20.168+0000 7f1548990700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15440f5410 0x7f1544109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:49:20.170 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:20.168+0000 7f1548990700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15440f5410 0x7f1544109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:41578/0 (socket says 172.21.15.134:41578) 2024-01-21T11:49:20.170 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:20.168+0000 7f1548990700 1 -- 172.21.15.134:0/387994540 learned_addr learned my addr 172.21.15.134:0/387994540 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:49:20.170 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:20.168+0000 7f1548990700 1 -- 172.21.15.134:0/387994540 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1538005040 con 0x7f15440f5410 2024-01-21T11:49:20.170 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:20.168+0000 7f1548990700 1 --2- 172.21.15.134:0/387994540 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15440f5410 0x7f1544109450 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f1538013510 tx=0x7f153800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:49:20.170 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:20.169+0000 7f15417fa700 1 -- 172.21.15.134:0/387994540 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1538009320 con 0x7f15440f5410 2024-01-21T11:49:20.170 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:20.169+0000 7f154abf4700 1 -- 172.21.15.134:0/387994540 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1544060420 con 0x7f15440f5410 2024-01-21T11:49:20.170 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:20.169+0000 7f15417fa700 1 -- 172.21.15.134:0/387994540 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1538013f00 con 0x7f15440f5410 2024-01-21T11:49:20.170 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:20.169+0000 7f15417fa700 1 -- 172.21.15.134:0/387994540 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1538014d90 con 0x7f15440f5410 2024-01-21T11:49:20.170 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:20.169+0000 7f154abf4700 1 -- 172.21.15.134:0/387994540 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1544109b00 con 0x7f15440f5410 2024-01-21T11:49:20.171 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:20.170+0000 7f15417fa700 1 -- 172.21.15.134:0/387994540 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f15380095f0 con 0x7f15440f5410 2024-01-21T11:49:20.171 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:20.170+0000 7f154abf4700 1 -- 172.21.15.134:0/387994540 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1544059100 con 0x7f15440f5410 2024-01-21T11:49:20.172 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:20.170+0000 7f15417fa700 1 --2- 172.21.15.134:0/387994540 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f153402ff60 0x7f1534032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:49:20.172 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:20.170+0000 7f15417fa700 1 -- 172.21.15.134:0/387994540 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1538041000 con 0x7f15440f5410 2024-01-21T11:49:20.172 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:20.170+0000 7f1543fff700 1 --2- 172.21.15.134:0/387994540 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f153402ff60 0x7f1534032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:49:20.172 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:20.171+0000 7f1543fff700 1 --2- 172.21.15.134:0/387994540 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f153402ff60 0x7f1534032420 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f15300096a0 tx=0x7f1530006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:49:20.174 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:20.174+0000 7f15417fa700 1 -- 172.21.15.134:0/387994540 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1538014420 con 0x7f15440f5410 2024-01-21T11:49:20.381 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:20.381+0000 7f154abf4700 1 -- 172.21.15.134:0/387994540 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1544045ad0 con 0x7f15440f5410 2024-01-21T11:49:20.382 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:20.382+0000 7f15417fa700 1 -- 172.21.15.134:0/387994540 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1538010230 con 0x7f15440f5410 2024-01-21T11:49:20.382 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:49:20.383 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:49:20.384 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:20.383+0000 7f154abf4700 1 -- 172.21.15.134:0/387994540 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f153402ff60 msgr2=0x7f1534032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:49:20.384 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:20.383+0000 7f154abf4700 1 --2- 172.21.15.134:0/387994540 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f153402ff60 0x7f1534032420 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f15300096a0 tx=0x7f1530006b40).stop 2024-01-21T11:49:20.384 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:20.383+0000 7f154abf4700 1 -- 172.21.15.134:0/387994540 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15440f5410 msgr2=0x7f1544109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:49:20.384 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:20.383+0000 7f154abf4700 1 --2- 172.21.15.134:0/387994540 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15440f5410 0x7f1544109450 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f1538013510 tx=0x7f153800b040).stop 2024-01-21T11:49:20.384 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:20.384+0000 7f154abf4700 1 -- 172.21.15.134:0/387994540 shutdown_connections 2024-01-21T11:49:20.385 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:20.384+0000 7f154abf4700 1 --2- 172.21.15.134:0/387994540 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f153402ff60 0x7f1534032420 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:49:20.385 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:20.384+0000 7f154abf4700 1 --2- 172.21.15.134:0/387994540 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15440f5410 0x7f1544109450 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:49:20.385 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:20.384+0000 7f154abf4700 1 -- 172.21.15.134:0/387994540 >> 172.21.15.134:0/387994540 conn(0x7f15440f0b70 msgr2=0x7f15440f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:49:20.385 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:20.384+0000 7f154abf4700 1 -- 172.21.15.134:0/387994540 shutdown_connections 2024-01-21T11:49:20.385 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:20.384+0000 7f154abf4700 1 -- 172.21.15.134:0/387994540 wait complete. 2024-01-21T11:49:20.385 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:49:20.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:49:20 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:49:20.383058+0000 mon.smithi029 (mon.0) 288 : audit [DBG] from='client.? 172.21.15.134:0/387994540' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:49:21.785 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:49:21.785 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:49:21.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:49:21 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:49:19.872045+0000 mgr.smithi029.blhxxk (mgr.14164) 225 : cluster [DBG] pgmap v213: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:49:22.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:22.942+0000 7f2aa0292700 1 -- 172.21.15.134:0/207513453 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a980f6510 msgr2=0x7f2a980f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:49:22.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:22.942+0000 7f2aa0292700 1 --2- 172.21.15.134:0/207513453 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a980f6510 0x7f2a980f6930 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7f2a90009fb0 tx=0x7f2a90009b30).stop 2024-01-21T11:49:22.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:22.942+0000 7f2aa0292700 1 -- 172.21.15.134:0/207513453 shutdown_connections 2024-01-21T11:49:22.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:22.942+0000 7f2aa0292700 1 --2- 172.21.15.134:0/207513453 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a980f6510 0x7f2a980f6930 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:49:22.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:22.942+0000 7f2aa0292700 1 -- 172.21.15.134:0/207513453 >> 172.21.15.134:0/207513453 conn(0x7f2a980f1c70 msgr2=0x7f2a980f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:49:22.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:22.942+0000 7f2aa0292700 1 -- 172.21.15.134:0/207513453 shutdown_connections 2024-01-21T11:49:22.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:22.942+0000 7f2aa0292700 1 -- 172.21.15.134:0/207513453 wait complete. 2024-01-21T11:49:22.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:22.943+0000 7f2aa0292700 1 Processor -- start 2024-01-21T11:49:22.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:22.943+0000 7f2aa0292700 1 -- start start 2024-01-21T11:49:22.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:22.943+0000 7f2aa0292700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a980f6510 0x7f2a98106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:49:22.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:22.943+0000 7f2aa0292700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a98106c20 con 0x7f2a980f6510 2024-01-21T11:49:22.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:22.943+0000 7f2a9e02e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a980f6510 0x7f2a98106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:49:22.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:22.943+0000 7f2a9e02e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a980f6510 0x7f2a98106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:41596/0 (socket says 172.21.15.134:41596) 2024-01-21T11:49:22.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:22.943+0000 7f2a9e02e700 1 -- 172.21.15.134:0/4156227780 learned_addr learned my addr 172.21.15.134:0/4156227780 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:49:22.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:22.944+0000 7f2a9e02e700 1 -- 172.21.15.134:0/4156227780 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a90005040 con 0x7f2a980f6510 2024-01-21T11:49:22.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:22.944+0000 7f2a9e02e700 1 --2- 172.21.15.134:0/4156227780 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a980f6510 0x7f2a98106730 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7f2a90013510 tx=0x7f2a9000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:49:22.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:22.944+0000 7f2a8effd700 1 -- 172.21.15.134:0/4156227780 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2a90009320 con 0x7f2a980f6510 2024-01-21T11:49:22.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:22.944+0000 7f2aa0292700 1 -- 172.21.15.134:0/4156227780 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a98106e20 con 0x7f2a980f6510 2024-01-21T11:49:22.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:22.944+0000 7f2a8effd700 1 -- 172.21.15.134:0/4156227780 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2a90013f00 con 0x7f2a980f6510 2024-01-21T11:49:22.947 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:22.944+0000 7f2a8effd700 1 -- 172.21.15.134:0/4156227780 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2a90014d90 con 0x7f2a980f6510 2024-01-21T11:49:22.947 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:22.944+0000 7f2aa0292700 1 -- 172.21.15.134:0/4156227780 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a981072b0 con 0x7f2a980f6510 2024-01-21T11:49:22.947 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:22.945+0000 7f2a8effd700 1 -- 172.21.15.134:0/4156227780 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2a900095f0 con 0x7f2a980f6510 2024-01-21T11:49:22.947 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:22.945+0000 7f2aa0292700 1 -- 172.21.15.134:0/4156227780 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a98045ad0 con 0x7f2a980f6510 2024-01-21T11:49:22.947 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:22.945+0000 7f2a8effd700 1 --2- 172.21.15.134:0/4156227780 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f2a8402ff60 0x7f2a84032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:49:22.947 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:22.945+0000 7f2a8effd700 1 -- 172.21.15.134:0/4156227780 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2a90040fe0 con 0x7f2a980f6510 2024-01-21T11:49:22.947 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:22.945+0000 7f2a9d82d700 1 --2- 172.21.15.134:0/4156227780 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f2a8402ff60 0x7f2a84032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:49:22.947 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:22.946+0000 7f2a9d82d700 1 --2- 172.21.15.134:0/4156227780 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f2a8402ff60 0x7f2a84032420 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f2a880096a0 tx=0x7f2a88006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:49:22.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:22.950+0000 7f2a8effd700 1 -- 172.21.15.134:0/4156227780 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2a9001d7f0 con 0x7f2a980f6510 2024-01-21T11:49:23.155 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:23.155+0000 7f2aa0292700 1 -- 172.21.15.134:0/4156227780 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2a980fbf60 con 0x7f2a980f6510 2024-01-21T11:49:23.156 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:23.155+0000 7f2a8effd700 1 -- 172.21.15.134:0/4156227780 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2a90010230 con 0x7f2a980f6510 2024-01-21T11:49:23.156 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:49:23.156 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:49:23.157 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:23.157+0000 7f2aa0292700 1 -- 172.21.15.134:0/4156227780 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f2a8402ff60 msgr2=0x7f2a84032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:49:23.157 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:23.157+0000 7f2aa0292700 1 --2- 172.21.15.134:0/4156227780 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f2a8402ff60 0x7f2a84032420 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f2a880096a0 tx=0x7f2a88006b40).stop 2024-01-21T11:49:23.158 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:23.157+0000 7f2aa0292700 1 -- 172.21.15.134:0/4156227780 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a980f6510 msgr2=0x7f2a98106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:49:23.158 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:23.157+0000 7f2aa0292700 1 --2- 172.21.15.134:0/4156227780 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a980f6510 0x7f2a98106730 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7f2a90013510 tx=0x7f2a9000b040).stop 2024-01-21T11:49:23.158 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:23.157+0000 7f2aa0292700 1 -- 172.21.15.134:0/4156227780 shutdown_connections 2024-01-21T11:49:23.158 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:23.157+0000 7f2aa0292700 1 --2- 172.21.15.134:0/4156227780 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f2a8402ff60 0x7f2a84032420 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:49:23.158 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:23.157+0000 7f2aa0292700 1 --2- 172.21.15.134:0/4156227780 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a980f6510 0x7f2a98106730 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:49:23.158 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:23.157+0000 7f2aa0292700 1 -- 172.21.15.134:0/4156227780 >> 172.21.15.134:0/4156227780 conn(0x7f2a980f1c70 msgr2=0x7f2a980f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:49:23.158 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:23.157+0000 7f2aa0292700 1 -- 172.21.15.134:0/4156227780 shutdown_connections 2024-01-21T11:49:23.158 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:23.157+0000 7f2aa0292700 1 -- 172.21.15.134:0/4156227780 wait complete. 2024-01-21T11:49:23.158 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:49:23.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:49:23 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:49:21.872348+0000 mgr.smithi029.blhxxk (mgr.14164) 226 : cluster [DBG] pgmap v214: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:49:23.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:49:23 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:49:23.156570+0000 mon.smithi029 (mon.0) 289 : audit [DBG] from='client.? 172.21.15.134:0/4156227780' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:49:24.536 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:49:24.537 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:49:25.675 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:25.675+0000 7fd5c0a42700 1 -- 172.21.15.134:0/4142730458 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5bc0f51a0 msgr2=0x7fd5bc0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:49:25.675 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:25.675+0000 7fd5c0a42700 1 --2- 172.21.15.134:0/4142730458 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5bc0f51a0 0x7fd5bc0f55c0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7fd5ac009fb0 tx=0x7fd5ac009b30).stop 2024-01-21T11:49:25.676 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:25.675+0000 7fd5c0a42700 1 -- 172.21.15.134:0/4142730458 shutdown_connections 2024-01-21T11:49:25.676 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:25.675+0000 7fd5c0a42700 1 --2- 172.21.15.134:0/4142730458 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5bc0f51a0 0x7fd5bc0f55c0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:49:25.676 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:25.675+0000 7fd5c0a42700 1 -- 172.21.15.134:0/4142730458 >> 172.21.15.134:0/4142730458 conn(0x7fd5bc0f09a0 msgr2=0x7fd5bc0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:49:25.676 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:25.675+0000 7fd5c0a42700 1 -- 172.21.15.134:0/4142730458 shutdown_connections 2024-01-21T11:49:25.676 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:25.675+0000 7fd5c0a42700 1 -- 172.21.15.134:0/4142730458 wait complete. 2024-01-21T11:49:25.678 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:25.675+0000 7fd5c0a42700 1 Processor -- start 2024-01-21T11:49:25.678 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:25.676+0000 7fd5c0a42700 1 -- start start 2024-01-21T11:49:25.678 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:25.676+0000 7fd5c0a42700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5bc0f51a0 0x7fd5bc1064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:49:25.678 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:25.676+0000 7fd5c0a42700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5bc1069a0 con 0x7fd5bc0f51a0 2024-01-21T11:49:25.678 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:25.676+0000 7fd5bad9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5bc0f51a0 0x7fd5bc1064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:49:25.678 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:25.676+0000 7fd5bad9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5bc0f51a0 0x7fd5bc1064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:60522/0 (socket says 172.21.15.134:60522) 2024-01-21T11:49:25.678 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:25.676+0000 7fd5bad9d700 1 -- 172.21.15.134:0/895508633 learned_addr learned my addr 172.21.15.134:0/895508633 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:49:25.678 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:25.676+0000 7fd5bad9d700 1 -- 172.21.15.134:0/895508633 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5ac005040 con 0x7fd5bc0f51a0 2024-01-21T11:49:25.678 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:25.676+0000 7fd5bad9d700 1 --2- 172.21.15.134:0/895508633 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5bc0f51a0 0x7fd5bc1064b0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7fd5ac013510 tx=0x7fd5ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:49:25.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:25.677+0000 7fd5a37fe700 1 -- 172.21.15.134:0/895508633 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd5ac009320 con 0x7fd5bc0f51a0 2024-01-21T11:49:25.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:25.677+0000 7fd5c0a42700 1 -- 172.21.15.134:0/895508633 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5bc106ba0 con 0x7fd5bc0f51a0 2024-01-21T11:49:25.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:25.677+0000 7fd5a37fe700 1 -- 172.21.15.134:0/895508633 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd5ac013f00 con 0x7fd5bc0f51a0 2024-01-21T11:49:25.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:25.677+0000 7fd5a37fe700 1 -- 172.21.15.134:0/895508633 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd5ac014dc0 con 0x7fd5bc0f51a0 2024-01-21T11:49:25.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:25.677+0000 7fd5c0a42700 1 -- 172.21.15.134:0/895508633 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5bc107030 con 0x7fd5bc0f51a0 2024-01-21T11:49:25.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:25.677+0000 7fd5c0a42700 1 -- 172.21.15.134:0/895508633 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5bc045ad0 con 0x7fd5bc0f51a0 2024-01-21T11:49:25.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:25.678+0000 7fd5a37fe700 1 -- 172.21.15.134:0/895508633 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd5ac014420 con 0x7fd5bc0f51a0 2024-01-21T11:49:25.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:25.678+0000 7fd5a37fe700 1 --2- 172.21.15.134:0/895508633 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fd5a402ff60 0x7fd5a4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:49:25.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:25.678+0000 7fd5a37fe700 1 -- 172.21.15.134:0/895508633 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd5ac041050 con 0x7fd5bc0f51a0 2024-01-21T11:49:25.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:25.678+0000 7fd5ba59c700 1 --2- 172.21.15.134:0/895508633 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fd5a402ff60 0x7fd5a4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:49:25.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:25.678+0000 7fd5ba59c700 1 --2- 172.21.15.134:0/895508633 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fd5a402ff60 0x7fd5a4032420 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7fd5b00096a0 tx=0x7fd5b0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:49:25.683 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:25.683+0000 7fd5a37fe700 1 -- 172.21.15.134:0/895508633 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd5ac0095f0 con 0x7fd5bc0f51a0 2024-01-21T11:49:25.887 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:25.887+0000 7fd5c0a42700 1 -- 172.21.15.134:0/895508633 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd5bc107ee0 con 0x7fd5bc0f51a0 2024-01-21T11:49:25.888 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:25.888+0000 7fd5a37fe700 1 -- 172.21.15.134:0/895508633 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd5ac01cb10 con 0x7fd5bc0f51a0 2024-01-21T11:49:25.888 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:49:25.888 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:49:25.890 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:25.890+0000 7fd5c0a42700 1 -- 172.21.15.134:0/895508633 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fd5a402ff60 msgr2=0x7fd5a4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:49:25.890 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:25.890+0000 7fd5c0a42700 1 --2- 172.21.15.134:0/895508633 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fd5a402ff60 0x7fd5a4032420 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7fd5b00096a0 tx=0x7fd5b0006b40).stop 2024-01-21T11:49:25.890 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:25.890+0000 7fd5c0a42700 1 -- 172.21.15.134:0/895508633 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5bc0f51a0 msgr2=0x7fd5bc1064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:49:25.890 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:25.890+0000 7fd5c0a42700 1 --2- 172.21.15.134:0/895508633 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5bc0f51a0 0x7fd5bc1064b0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7fd5ac013510 tx=0x7fd5ac00b040).stop 2024-01-21T11:49:25.890 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:25.890+0000 7fd5c0a42700 1 -- 172.21.15.134:0/895508633 shutdown_connections 2024-01-21T11:49:25.890 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:25.890+0000 7fd5c0a42700 1 --2- 172.21.15.134:0/895508633 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fd5a402ff60 0x7fd5a4032420 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:49:25.890 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:25.890+0000 7fd5c0a42700 1 --2- 172.21.15.134:0/895508633 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5bc0f51a0 0x7fd5bc1064b0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:49:25.890 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:25.890+0000 7fd5c0a42700 1 -- 172.21.15.134:0/895508633 >> 172.21.15.134:0/895508633 conn(0x7fd5bc0f09a0 msgr2=0x7fd5bc0f1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:49:25.890 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:25.890+0000 7fd5c0a42700 1 -- 172.21.15.134:0/895508633 shutdown_connections 2024-01-21T11:49:25.891 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:25.890+0000 7fd5c0a42700 1 -- 172.21.15.134:0/895508633 wait complete. 2024-01-21T11:49:25.891 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:49:25.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:49:25 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:49:23.872683+0000 mgr.smithi029.blhxxk (mgr.14164) 227 : cluster [DBG] pgmap v215: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:49:26.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:49:26 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:49:25.889311+0000 mon.smithi029 (mon.0) 290 : audit [DBG] from='client.? 172.21.15.134:0/895508633' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:49:27.280 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:49:27.280 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:49:27.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:49:27 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:49:25.873037+0000 mgr.smithi029.blhxxk (mgr.14164) 228 : cluster [DBG] pgmap v216: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:49:28.467 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:28.467+0000 7f01555b9700 1 -- 172.21.15.134:0/2354771726 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f01500f6510 msgr2=0x7f01500f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:49:28.467 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:28.467+0000 7f01555b9700 1 --2- 172.21.15.134:0/2354771726 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f01500f6510 0x7f01500f6930 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7f0140009fb0 tx=0x7f0140009b30).stop 2024-01-21T11:49:28.467 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:28.467+0000 7f01555b9700 1 -- 172.21.15.134:0/2354771726 shutdown_connections 2024-01-21T11:49:28.467 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:28.467+0000 7f01555b9700 1 --2- 172.21.15.134:0/2354771726 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f01500f6510 0x7f01500f6930 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:49:28.468 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:28.467+0000 7f01555b9700 1 -- 172.21.15.134:0/2354771726 >> 172.21.15.134:0/2354771726 conn(0x7f01500f1c70 msgr2=0x7f01500f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:49:28.468 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:28.467+0000 7f01555b9700 1 -- 172.21.15.134:0/2354771726 shutdown_connections 2024-01-21T11:49:28.468 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:28.467+0000 7f01555b9700 1 -- 172.21.15.134:0/2354771726 wait complete. 2024-01-21T11:49:28.469 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:28.467+0000 7f01555b9700 1 Processor -- start 2024-01-21T11:49:28.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:28.467+0000 7f01555b9700 1 -- start start 2024-01-21T11:49:28.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:28.468+0000 7f01555b9700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f01500f6510 0x7f0150106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:49:28.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:28.468+0000 7f01555b9700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0150106c20 con 0x7f01500f6510 2024-01-21T11:49:28.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:28.468+0000 7f014effd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f01500f6510 0x7f0150106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:49:28.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:28.468+0000 7f014effd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f01500f6510 0x7f0150106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:60554/0 (socket says 172.21.15.134:60554) 2024-01-21T11:49:28.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:28.468+0000 7f014effd700 1 -- 172.21.15.134:0/3951639182 learned_addr learned my addr 172.21.15.134:0/3951639182 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:49:28.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:28.468+0000 7f014effd700 1 -- 172.21.15.134:0/3951639182 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0140005040 con 0x7f01500f6510 2024-01-21T11:49:28.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:28.468+0000 7f014effd700 1 --2- 172.21.15.134:0/3951639182 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f01500f6510 0x7f0150106730 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7f0140013510 tx=0x7f014000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:49:28.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:28.468+0000 7f01377fe700 1 -- 172.21.15.134:0/3951639182 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0140009320 con 0x7f01500f6510 2024-01-21T11:49:28.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:28.468+0000 7f01555b9700 1 -- 172.21.15.134:0/3951639182 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0150106e20 con 0x7f01500f6510 2024-01-21T11:49:28.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:28.468+0000 7f01377fe700 1 -- 172.21.15.134:0/3951639182 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0140013f00 con 0x7f01500f6510 2024-01-21T11:49:28.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:28.468+0000 7f01377fe700 1 -- 172.21.15.134:0/3951639182 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0140014d90 con 0x7f01500f6510 2024-01-21T11:49:28.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:28.469+0000 7f01555b9700 1 -- 172.21.15.134:0/3951639182 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01501072b0 con 0x7f01500f6510 2024-01-21T11:49:28.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:28.469+0000 7f01377fe700 1 -- 172.21.15.134:0/3951639182 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f01400095f0 con 0x7f01500f6510 2024-01-21T11:49:28.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:28.469+0000 7f01555b9700 1 -- 172.21.15.134:0/3951639182 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0150045ad0 con 0x7f01500f6510 2024-01-21T11:49:28.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:28.469+0000 7f01377fe700 1 --2- 172.21.15.134:0/3951639182 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f013802ff60 0x7f0138032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:49:28.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:28.469+0000 7f01377fe700 1 -- 172.21.15.134:0/3951639182 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0140040fe0 con 0x7f01500f6510 2024-01-21T11:49:28.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:28.470+0000 7f014e7fc700 1 --2- 172.21.15.134:0/3951639182 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f013802ff60 0x7f0138032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:49:28.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:28.470+0000 7f014e7fc700 1 --2- 172.21.15.134:0/3951639182 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f013802ff60 0x7f0138032420 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f01440096a0 tx=0x7f0144006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:49:28.474 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:28.474+0000 7f01377fe700 1 -- 172.21.15.134:0/3951639182 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0140014420 con 0x7f01500f6510 2024-01-21T11:49:28.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:28.679+0000 7f01555b9700 1 -- 172.21.15.134:0/3951639182 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f01500fbf60 con 0x7f01500f6510 2024-01-21T11:49:28.680 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:28.681+0000 7f01377fe700 1 -- 172.21.15.134:0/3951639182 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0140010230 con 0x7f01500f6510 2024-01-21T11:49:28.680 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:49:28.680 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:49:28.682 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:28.681+0000 7f01555b9700 1 -- 172.21.15.134:0/3951639182 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f013802ff60 msgr2=0x7f0138032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:49:28.682 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:28.681+0000 7f01555b9700 1 --2- 172.21.15.134:0/3951639182 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f013802ff60 0x7f0138032420 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f01440096a0 tx=0x7f0144006b40).stop 2024-01-21T11:49:28.682 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:28.681+0000 7f01555b9700 1 -- 172.21.15.134:0/3951639182 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f01500f6510 msgr2=0x7f0150106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:49:28.682 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:28.681+0000 7f01555b9700 1 --2- 172.21.15.134:0/3951639182 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f01500f6510 0x7f0150106730 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7f0140013510 tx=0x7f014000b040).stop 2024-01-21T11:49:28.682 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:28.682+0000 7f01555b9700 1 -- 172.21.15.134:0/3951639182 shutdown_connections 2024-01-21T11:49:28.682 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:28.682+0000 7f01555b9700 1 --2- 172.21.15.134:0/3951639182 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f013802ff60 0x7f0138032420 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:49:28.682 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:28.682+0000 7f01555b9700 1 --2- 172.21.15.134:0/3951639182 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f01500f6510 0x7f0150106730 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:49:28.682 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:28.682+0000 7f01555b9700 1 -- 172.21.15.134:0/3951639182 >> 172.21.15.134:0/3951639182 conn(0x7f01500f1c70 msgr2=0x7f01500f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:49:28.682 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:28.682+0000 7f01555b9700 1 -- 172.21.15.134:0/3951639182 shutdown_connections 2024-01-21T11:49:28.682 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:28.682+0000 7f01555b9700 1 -- 172.21.15.134:0/3951639182 wait complete. 2024-01-21T11:49:28.683 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:49:29.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:49:29 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:49:27.873423+0000 mgr.smithi029.blhxxk (mgr.14164) 229 : cluster [DBG] pgmap v217: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:49:29.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:49:29 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:49:28.681132+0000 mon.smithi029 (mon.0) 291 : audit [DBG] from='client.? 172.21.15.134:0/3951639182' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:49:30.052 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:49:30.052 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:49:31.307 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:31.305+0000 7f2d8f1cf700 1 -- 172.21.15.134:0/1594222275 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d880f6510 msgr2=0x7f2d880f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:49:31.307 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:31.305+0000 7f2d8f1cf700 1 --2- 172.21.15.134:0/1594222275 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d880f6510 0x7f2d880f6930 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7f2d7c009fb0 tx=0x7f2d7c009b30).stop 2024-01-21T11:49:31.307 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:31.305+0000 7f2d8f1cf700 1 -- 172.21.15.134:0/1594222275 shutdown_connections 2024-01-21T11:49:31.307 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:31.305+0000 7f2d8f1cf700 1 --2- 172.21.15.134:0/1594222275 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d880f6510 0x7f2d880f6930 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:49:31.307 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:31.305+0000 7f2d8f1cf700 1 -- 172.21.15.134:0/1594222275 >> 172.21.15.134:0/1594222275 conn(0x7f2d880f1c70 msgr2=0x7f2d880f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:49:31.307 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:31.305+0000 7f2d8f1cf700 1 -- 172.21.15.134:0/1594222275 shutdown_connections 2024-01-21T11:49:31.307 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:31.305+0000 7f2d8f1cf700 1 -- 172.21.15.134:0/1594222275 wait complete. 2024-01-21T11:49:31.307 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:31.305+0000 7f2d8f1cf700 1 Processor -- start 2024-01-21T11:49:31.308 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:31.305+0000 7f2d8f1cf700 1 -- start start 2024-01-21T11:49:31.308 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:31.306+0000 7f2d8f1cf700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d880f6510 0x7f2d8806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:49:31.308 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:31.306+0000 7f2d8f1cf700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d88069e20 con 0x7f2d880f6510 2024-01-21T11:49:31.308 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:31.306+0000 7f2d8cf6b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d880f6510 0x7f2d8806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:49:31.308 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:31.306+0000 7f2d8cf6b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d880f6510 0x7f2d8806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:60574/0 (socket says 172.21.15.134:60574) 2024-01-21T11:49:31.308 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:31.306+0000 7f2d8cf6b700 1 -- 172.21.15.134:0/1995022366 learned_addr learned my addr 172.21.15.134:0/1995022366 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:49:31.308 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:31.306+0000 7f2d8cf6b700 1 -- 172.21.15.134:0/1995022366 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d7c005040 con 0x7f2d880f6510 2024-01-21T11:49:31.308 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:31.306+0000 7f2d8cf6b700 1 --2- 172.21.15.134:0/1995022366 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d880f6510 0x7f2d8806b9a0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f2d7c013510 tx=0x7f2d7c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:49:31.308 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:31.307+0000 7f2d857fa700 1 -- 172.21.15.134:0/1995022366 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d7c009320 con 0x7f2d880f6510 2024-01-21T11:49:31.308 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:31.307+0000 7f2d8f1cf700 1 -- 172.21.15.134:0/1995022366 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d88069fc0 con 0x7f2d880f6510 2024-01-21T11:49:31.308 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:31.307+0000 7f2d857fa700 1 -- 172.21.15.134:0/1995022366 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2d7c013f00 con 0x7f2d880f6510 2024-01-21T11:49:31.308 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:31.307+0000 7f2d857fa700 1 -- 172.21.15.134:0/1995022366 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d7c014d90 con 0x7f2d880f6510 2024-01-21T11:49:31.309 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:31.307+0000 7f2d8f1cf700 1 -- 172.21.15.134:0/1995022366 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d8806a450 con 0x7f2d880f6510 2024-01-21T11:49:31.309 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:31.307+0000 7f2d857fa700 1 -- 172.21.15.134:0/1995022366 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2d7c014420 con 0x7f2d880f6510 2024-01-21T11:49:31.309 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:31.307+0000 7f2d8f1cf700 1 -- 172.21.15.134:0/1995022366 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d88059100 con 0x7f2d880f6510 2024-01-21T11:49:31.310 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:31.308+0000 7f2d857fa700 1 --2- 172.21.15.134:0/1995022366 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f2d7802ff60 0x7f2d78032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:49:31.310 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:31.308+0000 7f2d857fa700 1 -- 172.21.15.134:0/1995022366 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2d7c041000 con 0x7f2d880f6510 2024-01-21T11:49:31.310 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:31.308+0000 7f2d87fff700 1 --2- 172.21.15.134:0/1995022366 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f2d7802ff60 0x7f2d78032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:49:31.310 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:31.308+0000 7f2d87fff700 1 --2- 172.21.15.134:0/1995022366 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f2d7802ff60 0x7f2d78032420 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f2d740096a0 tx=0x7f2d74006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:49:31.312 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:31.313+0000 7f2d857fa700 1 -- 172.21.15.134:0/1995022366 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2d7c0095f0 con 0x7f2d880f6510 2024-01-21T11:49:31.516 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:31.516+0000 7f2d8f1cf700 1 -- 172.21.15.134:0/1995022366 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2d880fbf60 con 0x7f2d880f6510 2024-01-21T11:49:31.517 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:31.516+0000 7f2d857fa700 1 -- 172.21.15.134:0/1995022366 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2d7c01db70 con 0x7f2d880f6510 2024-01-21T11:49:31.517 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:49:31.517 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:49:31.519 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:31.518+0000 7f2d8f1cf700 1 -- 172.21.15.134:0/1995022366 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f2d7802ff60 msgr2=0x7f2d78032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:49:31.519 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:31.518+0000 7f2d8f1cf700 1 --2- 172.21.15.134:0/1995022366 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f2d7802ff60 0x7f2d78032420 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f2d740096a0 tx=0x7f2d74006b40).stop 2024-01-21T11:49:31.519 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:31.518+0000 7f2d8f1cf700 1 -- 172.21.15.134:0/1995022366 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d880f6510 msgr2=0x7f2d8806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:49:31.519 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:31.518+0000 7f2d8f1cf700 1 --2- 172.21.15.134:0/1995022366 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d880f6510 0x7f2d8806b9a0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f2d7c013510 tx=0x7f2d7c00b040).stop 2024-01-21T11:49:31.519 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:31.518+0000 7f2d8f1cf700 1 -- 172.21.15.134:0/1995022366 shutdown_connections 2024-01-21T11:49:31.519 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:31.518+0000 7f2d8f1cf700 1 --2- 172.21.15.134:0/1995022366 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f2d7802ff60 0x7f2d78032420 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:49:31.519 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:31.518+0000 7f2d8f1cf700 1 --2- 172.21.15.134:0/1995022366 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d880f6510 0x7f2d8806b9a0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:49:31.519 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:31.518+0000 7f2d8f1cf700 1 -- 172.21.15.134:0/1995022366 >> 172.21.15.134:0/1995022366 conn(0x7f2d880f1c70 msgr2=0x7f2d880f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:49:31.519 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:31.518+0000 7f2d8f1cf700 1 -- 172.21.15.134:0/1995022366 shutdown_connections 2024-01-21T11:49:31.519 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:31.518+0000 7f2d8f1cf700 1 -- 172.21.15.134:0/1995022366 wait complete. 2024-01-21T11:49:31.520 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:49:31.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:49:31 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:49:29.873737+0000 mgr.smithi029.blhxxk (mgr.14164) 230 : cluster [DBG] pgmap v218: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:49:31.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:49:31 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:49:31.517645+0000 mon.smithi029 (mon.0) 292 : audit [DBG] from='client.? 172.21.15.134:0/1995022366' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:49:32.920 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:49:32.921 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:49:33.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:49:33 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:49:31.874107+0000 mgr.smithi029.blhxxk (mgr.14164) 231 : cluster [DBG] pgmap v219: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:49:34.111 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:34.111+0000 7f4e01e65700 1 -- 172.21.15.134:0/3675269408 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4dfc0f62a0 msgr2=0x7f4dfc0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:49:34.111 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:34.111+0000 7f4e01e65700 1 --2- 172.21.15.134:0/3675269408 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4dfc0f62a0 0x7f4dfc0f66c0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7f4df0009fb0 tx=0x7f4df0009b30).stop 2024-01-21T11:49:34.111 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:34.111+0000 7f4e01e65700 1 -- 172.21.15.134:0/3675269408 shutdown_connections 2024-01-21T11:49:34.111 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:34.111+0000 7f4e01e65700 1 --2- 172.21.15.134:0/3675269408 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4dfc0f62a0 0x7f4dfc0f66c0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:49:34.111 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:34.111+0000 7f4e01e65700 1 -- 172.21.15.134:0/3675269408 >> 172.21.15.134:0/3675269408 conn(0x7f4dfc0f1aa0 msgr2=0x7f4dfc0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:49:34.111 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:34.111+0000 7f4e01e65700 1 -- 172.21.15.134:0/3675269408 shutdown_connections 2024-01-21T11:49:34.111 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:34.111+0000 7f4e01e65700 1 -- 172.21.15.134:0/3675269408 wait complete. 2024-01-21T11:49:34.111 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:34.111+0000 7f4e01e65700 1 Processor -- start 2024-01-21T11:49:34.111 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:34.112+0000 7f4e01e65700 1 -- start start 2024-01-21T11:49:34.113 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:34.112+0000 7f4e01e65700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4dfc0f62a0 0x7f4dfc1064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:49:34.113 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:34.112+0000 7f4e01e65700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dfc1069a0 con 0x7f4dfc0f62a0 2024-01-21T11:49:34.113 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:34.112+0000 7f4e00e63700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4dfc0f62a0 0x7f4dfc1064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:49:34.113 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:34.112+0000 7f4e00e63700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4dfc0f62a0 0x7f4dfc1064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:60598/0 (socket says 172.21.15.134:60598) 2024-01-21T11:49:34.113 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:34.112+0000 7f4e00e63700 1 -- 172.21.15.134:0/1828172581 learned_addr learned my addr 172.21.15.134:0/1828172581 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:49:34.114 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:34.112+0000 7f4e00e63700 1 -- 172.21.15.134:0/1828172581 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4df0005040 con 0x7f4dfc0f62a0 2024-01-21T11:49:34.114 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:34.113+0000 7f4e00e63700 1 --2- 172.21.15.134:0/1828172581 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4dfc0f62a0 0x7f4dfc1064b0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f4df0013510 tx=0x7f4df000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:49:34.114 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:34.113+0000 7f4df97fa700 1 -- 172.21.15.134:0/1828172581 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4df0009320 con 0x7f4dfc0f62a0 2024-01-21T11:49:34.114 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:34.113+0000 7f4e01e65700 1 -- 172.21.15.134:0/1828172581 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dfc106ba0 con 0x7f4dfc0f62a0 2024-01-21T11:49:34.114 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:34.113+0000 7f4df97fa700 1 -- 172.21.15.134:0/1828172581 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4df0013f00 con 0x7f4dfc0f62a0 2024-01-21T11:49:34.114 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:34.113+0000 7f4df97fa700 1 -- 172.21.15.134:0/1828172581 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4df0014dc0 con 0x7f4dfc0f62a0 2024-01-21T11:49:34.114 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:34.113+0000 7f4e01e65700 1 -- 172.21.15.134:0/1828172581 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dfc107030 con 0x7f4dfc0f62a0 2024-01-21T11:49:34.114 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:34.114+0000 7f4df97fa700 1 -- 172.21.15.134:0/1828172581 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4df00095f0 con 0x7f4dfc0f62a0 2024-01-21T11:49:34.114 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:34.114+0000 7f4e01e65700 1 -- 172.21.15.134:0/1828172581 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dfc045ad0 con 0x7f4dfc0f62a0 2024-01-21T11:49:34.114 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:34.114+0000 7f4df97fa700 1 --2- 172.21.15.134:0/1828172581 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f4dec02ff60 0x7f4dec032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:49:34.114 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:34.114+0000 7f4df97fa700 1 -- 172.21.15.134:0/1828172581 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4df00410c0 con 0x7f4dfc0f62a0 2024-01-21T11:49:34.114 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:34.114+0000 7f4dfbfff700 1 --2- 172.21.15.134:0/1828172581 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f4dec02ff60 0x7f4dec032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:49:34.115 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:34.114+0000 7f4dfbfff700 1 --2- 172.21.15.134:0/1828172581 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f4dec02ff60 0x7f4dec032420 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f4de80096a0 tx=0x7f4de8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:49:34.118 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:34.119+0000 7f4df97fa700 1 -- 172.21.15.134:0/1828172581 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4df001c790 con 0x7f4dfc0f62a0 2024-01-21T11:49:34.322 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:34.323+0000 7f4e01e65700 1 -- 172.21.15.134:0/1828172581 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4dfc107ee0 con 0x7f4dfc0f62a0 2024-01-21T11:49:34.323 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:34.324+0000 7f4df97fa700 1 -- 172.21.15.134:0/1828172581 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4df0010230 con 0x7f4dfc0f62a0 2024-01-21T11:49:34.323 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:49:34.323 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:49:34.325 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:34.325+0000 7f4e01e65700 1 -- 172.21.15.134:0/1828172581 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f4dec02ff60 msgr2=0x7f4dec032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:49:34.325 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:34.325+0000 7f4e01e65700 1 --2- 172.21.15.134:0/1828172581 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f4dec02ff60 0x7f4dec032420 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f4de80096a0 tx=0x7f4de8006b40).stop 2024-01-21T11:49:34.325 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:34.325+0000 7f4e01e65700 1 -- 172.21.15.134:0/1828172581 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4dfc0f62a0 msgr2=0x7f4dfc1064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:49:34.325 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:34.325+0000 7f4e01e65700 1 --2- 172.21.15.134:0/1828172581 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4dfc0f62a0 0x7f4dfc1064b0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f4df0013510 tx=0x7f4df000b040).stop 2024-01-21T11:49:34.325 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:34.326+0000 7f4e01e65700 1 -- 172.21.15.134:0/1828172581 shutdown_connections 2024-01-21T11:49:34.325 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:34.326+0000 7f4e01e65700 1 --2- 172.21.15.134:0/1828172581 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f4dec02ff60 0x7f4dec032420 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:49:34.325 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:34.326+0000 7f4e01e65700 1 --2- 172.21.15.134:0/1828172581 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4dfc0f62a0 0x7f4dfc1064b0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:49:34.325 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:34.326+0000 7f4e01e65700 1 -- 172.21.15.134:0/1828172581 >> 172.21.15.134:0/1828172581 conn(0x7f4dfc0f1aa0 msgr2=0x7f4dfc0f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:49:34.325 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:34.326+0000 7f4e01e65700 1 -- 172.21.15.134:0/1828172581 shutdown_connections 2024-01-21T11:49:34.325 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:34.326+0000 7f4e01e65700 1 -- 172.21.15.134:0/1828172581 wait complete. 2024-01-21T11:49:34.326 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:49:34.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:49:34 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:49:34.324140+0000 mon.smithi029 (mon.0) 293 : audit [DBG] from='client.? 172.21.15.134:0/1828172581' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:49:35.684 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:49:35.684 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:49:35.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:49:35 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:49:33.874370+0000 mgr.smithi029.blhxxk (mgr.14164) 232 : cluster [DBG] pgmap v220: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:49:36.884 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:36.882+0000 7fb977fff700 1 -- 172.21.15.134:0/2179810758 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9780f62a0 msgr2=0x7fb9780f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:49:36.885 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:36.882+0000 7fb977fff700 1 --2- 172.21.15.134:0/2179810758 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9780f62a0 0x7fb9780f66c0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7fb968009fb0 tx=0x7fb968009b30).stop 2024-01-21T11:49:36.885 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:36.883+0000 7fb977fff700 1 -- 172.21.15.134:0/2179810758 shutdown_connections 2024-01-21T11:49:36.885 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:36.883+0000 7fb977fff700 1 --2- 172.21.15.134:0/2179810758 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9780f62a0 0x7fb9780f66c0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:49:36.885 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:36.883+0000 7fb977fff700 1 -- 172.21.15.134:0/2179810758 >> 172.21.15.134:0/2179810758 conn(0x7fb9780f1ac0 msgr2=0x7fb9780f3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:49:36.885 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:36.883+0000 7fb977fff700 1 -- 172.21.15.134:0/2179810758 shutdown_connections 2024-01-21T11:49:36.885 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:36.883+0000 7fb977fff700 1 -- 172.21.15.134:0/2179810758 wait complete. 2024-01-21T11:49:36.885 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:36.883+0000 7fb977fff700 1 Processor -- start 2024-01-21T11:49:36.885 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:36.883+0000 7fb977fff700 1 -- start start 2024-01-21T11:49:36.885 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:36.883+0000 7fb977fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9780f62a0 0x7fb9781064a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:49:36.885 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:36.883+0000 7fb977fff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb978106990 con 0x7fb9780f62a0 2024-01-21T11:49:36.885 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:36.884+0000 7fb976ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9780f62a0 0x7fb9781064a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:49:36.885 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:36.884+0000 7fb976ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9780f62a0 0x7fb9781064a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:43874/0 (socket says 172.21.15.134:43874) 2024-01-21T11:49:36.885 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:36.884+0000 7fb976ffd700 1 -- 172.21.15.134:0/3937523684 learned_addr learned my addr 172.21.15.134:0/3937523684 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:49:36.886 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:36.884+0000 7fb976ffd700 1 -- 172.21.15.134:0/3937523684 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb968005040 con 0x7fb9780f62a0 2024-01-21T11:49:36.886 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:36.884+0000 7fb976ffd700 1 --2- 172.21.15.134:0/3937523684 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9780f62a0 0x7fb9781064a0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7fb968013510 tx=0x7fb96800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:49:36.886 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:36.884+0000 7fb95f7fe700 1 -- 172.21.15.134:0/3937523684 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb968009320 con 0x7fb9780f62a0 2024-01-21T11:49:36.886 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:36.884+0000 7fb977fff700 1 -- 172.21.15.134:0/3937523684 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb978106b90 con 0x7fb9780f62a0 2024-01-21T11:49:36.886 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:36.884+0000 7fb95f7fe700 1 -- 172.21.15.134:0/3937523684 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb968013f00 con 0x7fb9780f62a0 2024-01-21T11:49:36.886 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:36.884+0000 7fb95f7fe700 1 -- 172.21.15.134:0/3937523684 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb968014dc0 con 0x7fb9780f62a0 2024-01-21T11:49:36.886 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:36.884+0000 7fb977fff700 1 -- 172.21.15.134:0/3937523684 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb978107020 con 0x7fb9780f62a0 2024-01-21T11:49:36.887 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:36.885+0000 7fb95f7fe700 1 -- 172.21.15.134:0/3937523684 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb9680095f0 con 0x7fb9780f62a0 2024-01-21T11:49:36.887 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:36.885+0000 7fb977fff700 1 -- 172.21.15.134:0/3937523684 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb978045ad0 con 0x7fb9780f62a0 2024-01-21T11:49:36.887 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:36.886+0000 7fb95f7fe700 1 --2- 172.21.15.134:0/3937523684 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fb96002ff60 0x7fb960032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:49:36.887 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:36.886+0000 7fb95f7fe700 1 -- 172.21.15.134:0/3937523684 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb9680410c0 con 0x7fb9780f62a0 2024-01-21T11:49:36.887 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:36.886+0000 7fb9767fc700 1 --2- 172.21.15.134:0/3937523684 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fb96002ff60 0x7fb960032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:49:36.887 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:36.886+0000 7fb9767fc700 1 --2- 172.21.15.134:0/3937523684 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fb96002ff60 0x7fb960032420 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7fb96c0096a0 tx=0x7fb96c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:49:36.890 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:36.890+0000 7fb95f7fe700 1 -- 172.21.15.134:0/3937523684 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb96801c790 con 0x7fb9780f62a0 2024-01-21T11:49:37.098 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:37.098+0000 7fb977fff700 1 -- 172.21.15.134:0/3937523684 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb978107ed0 con 0x7fb9780f62a0 2024-01-21T11:49:37.099 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:37.099+0000 7fb95f7fe700 1 -- 172.21.15.134:0/3937523684 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb968010230 con 0x7fb9780f62a0 2024-01-21T11:49:37.099 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:49:37.099 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:49:37.101 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:37.100+0000 7fb977fff700 1 -- 172.21.15.134:0/3937523684 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fb96002ff60 msgr2=0x7fb960032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:49:37.101 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:37.100+0000 7fb977fff700 1 --2- 172.21.15.134:0/3937523684 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fb96002ff60 0x7fb960032420 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7fb96c0096a0 tx=0x7fb96c006b40).stop 2024-01-21T11:49:37.101 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:37.100+0000 7fb977fff700 1 -- 172.21.15.134:0/3937523684 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9780f62a0 msgr2=0x7fb9781064a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:49:37.101 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:37.100+0000 7fb977fff700 1 --2- 172.21.15.134:0/3937523684 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9780f62a0 0x7fb9781064a0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7fb968013510 tx=0x7fb96800b040).stop 2024-01-21T11:49:37.101 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:37.100+0000 7fb977fff700 1 -- 172.21.15.134:0/3937523684 shutdown_connections 2024-01-21T11:49:37.101 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:37.100+0000 7fb977fff700 1 --2- 172.21.15.134:0/3937523684 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fb96002ff60 0x7fb960032420 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:49:37.101 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:37.100+0000 7fb977fff700 1 --2- 172.21.15.134:0/3937523684 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9780f62a0 0x7fb9781064a0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:49:37.101 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:37.100+0000 7fb977fff700 1 -- 172.21.15.134:0/3937523684 >> 172.21.15.134:0/3937523684 conn(0x7fb9780f1ac0 msgr2=0x7fb9780f2560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:49:37.101 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:37.101+0000 7fb977fff700 1 -- 172.21.15.134:0/3937523684 shutdown_connections 2024-01-21T11:49:37.101 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:37.101+0000 7fb977fff700 1 -- 172.21.15.134:0/3937523684 wait complete. 2024-01-21T11:49:37.102 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:49:37.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:49:37 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:49:35.874631+0000 mgr.smithi029.blhxxk (mgr.14164) 233 : cluster [DBG] pgmap v221: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:49:37.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:49:37 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:49:37.099932+0000 mon.smithi029 (mon.0) 294 : audit [DBG] from='client.? 172.21.15.134:0/3937523684' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:49:38.543 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:49:38.543 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:49:39.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:39.735+0000 7fa828484700 1 -- 172.21.15.134:0/3959420263 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8200f5410 msgr2=0x7fa8200f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:49:39.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:39.735+0000 7fa828484700 1 --2- 172.21.15.134:0/3959420263 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8200f5410 0x7fa8200f5830 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7fa810009fb0 tx=0x7fa810009b30).stop 2024-01-21T11:49:39.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:39.735+0000 7fa828484700 1 -- 172.21.15.134:0/3959420263 shutdown_connections 2024-01-21T11:49:39.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:39.735+0000 7fa828484700 1 --2- 172.21.15.134:0/3959420263 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8200f5410 0x7fa8200f5830 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:49:39.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:39.735+0000 7fa828484700 1 -- 172.21.15.134:0/3959420263 >> 172.21.15.134:0/3959420263 conn(0x7fa8200f0b70 msgr2=0x7fa8200f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:49:39.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:39.735+0000 7fa828484700 1 -- 172.21.15.134:0/3959420263 shutdown_connections 2024-01-21T11:49:39.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:39.735+0000 7fa828484700 1 -- 172.21.15.134:0/3959420263 wait complete. 2024-01-21T11:49:39.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:39.735+0000 7fa828484700 1 Processor -- start 2024-01-21T11:49:39.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:39.735+0000 7fa828484700 1 -- start start 2024-01-21T11:49:39.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:39.735+0000 7fa828484700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8200f5410 0x7fa8200fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:49:39.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:39.736+0000 7fa828484700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8200fb820 con 0x7fa8200f5410 2024-01-21T11:49:39.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:39.736+0000 7fa826220700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8200f5410 0x7fa8200fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:49:39.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:39.736+0000 7fa826220700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8200f5410 0x7fa8200fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:43880/0 (socket says 172.21.15.134:43880) 2024-01-21T11:49:39.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:39.736+0000 7fa826220700 1 -- 172.21.15.134:0/2615634353 learned_addr learned my addr 172.21.15.134:0/2615634353 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:49:39.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:39.736+0000 7fa826220700 1 -- 172.21.15.134:0/2615634353 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa810005040 con 0x7fa8200f5410 2024-01-21T11:49:39.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:39.736+0000 7fa826220700 1 --2- 172.21.15.134:0/2615634353 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8200f5410 0x7fa8200fd3a0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7fa810013510 tx=0x7fa81000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:49:39.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:39.736+0000 7fa816ffd700 1 -- 172.21.15.134:0/2615634353 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa810009320 con 0x7fa8200f5410 2024-01-21T11:49:39.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:39.736+0000 7fa828484700 1 -- 172.21.15.134:0/2615634353 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8200fb9c0 con 0x7fa8200f5410 2024-01-21T11:49:39.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:39.736+0000 7fa816ffd700 1 -- 172.21.15.134:0/2615634353 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa810013f00 con 0x7fa8200f5410 2024-01-21T11:49:39.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:39.736+0000 7fa816ffd700 1 -- 172.21.15.134:0/2615634353 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa810014d90 con 0x7fa8200f5410 2024-01-21T11:49:39.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:39.736+0000 7fa828484700 1 -- 172.21.15.134:0/2615634353 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8200fbe50 con 0x7fa8200f5410 2024-01-21T11:49:39.739 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:39.737+0000 7fa828484700 1 -- 172.21.15.134:0/2615634353 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa820059100 con 0x7fa8200f5410 2024-01-21T11:49:39.739 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:39.737+0000 7fa816ffd700 1 -- 172.21.15.134:0/2615634353 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa8100095f0 con 0x7fa8200f5410 2024-01-21T11:49:39.739 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:39.738+0000 7fa816ffd700 1 --2- 172.21.15.134:0/2615634353 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fa80c02ff60 0x7fa80c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:49:39.739 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:39.738+0000 7fa816ffd700 1 -- 172.21.15.134:0/2615634353 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa8100410c0 con 0x7fa8200f5410 2024-01-21T11:49:39.739 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:39.738+0000 7fa825a1f700 1 --2- 172.21.15.134:0/2615634353 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fa80c02ff60 0x7fa80c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:49:39.739 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:39.738+0000 7fa825a1f700 1 --2- 172.21.15.134:0/2615634353 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fa80c02ff60 0x7fa80c032420 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7fa81c0096a0 tx=0x7fa81c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:49:39.742 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:39.742+0000 7fa816ffd700 1 -- 172.21.15.134:0/2615634353 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa81001d7f0 con 0x7fa8200f5410 2024-01-21T11:49:39.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:49:39 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:49:37.874998+0000 mgr.smithi029.blhxxk (mgr.14164) 234 : cluster [DBG] pgmap v222: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:49:39.948 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:39.948+0000 7fa828484700 1 -- 172.21.15.134:0/2615634353 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa8200fae60 con 0x7fa8200f5410 2024-01-21T11:49:39.949 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:39.948+0000 7fa816ffd700 1 -- 172.21.15.134:0/2615634353 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa810010230 con 0x7fa8200f5410 2024-01-21T11:49:39.949 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:49:39.949 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:49:39.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:39.950+0000 7fa828484700 1 -- 172.21.15.134:0/2615634353 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fa80c02ff60 msgr2=0x7fa80c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:49:39.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:39.950+0000 7fa828484700 1 --2- 172.21.15.134:0/2615634353 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fa80c02ff60 0x7fa80c032420 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7fa81c0096a0 tx=0x7fa81c006b40).stop 2024-01-21T11:49:39.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:39.950+0000 7fa828484700 1 -- 172.21.15.134:0/2615634353 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8200f5410 msgr2=0x7fa8200fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:49:39.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:39.950+0000 7fa828484700 1 --2- 172.21.15.134:0/2615634353 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8200f5410 0x7fa8200fd3a0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7fa810013510 tx=0x7fa81000b040).stop 2024-01-21T11:49:39.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:39.950+0000 7fa828484700 1 -- 172.21.15.134:0/2615634353 shutdown_connections 2024-01-21T11:49:39.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:39.950+0000 7fa828484700 1 --2- 172.21.15.134:0/2615634353 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fa80c02ff60 0x7fa80c032420 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:49:39.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:39.950+0000 7fa828484700 1 --2- 172.21.15.134:0/2615634353 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8200f5410 0x7fa8200fd3a0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:49:39.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:39.950+0000 7fa828484700 1 -- 172.21.15.134:0/2615634353 >> 172.21.15.134:0/2615634353 conn(0x7fa8200f0b70 msgr2=0x7fa82005fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:49:39.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:39.950+0000 7fa828484700 1 -- 172.21.15.134:0/2615634353 shutdown_connections 2024-01-21T11:49:39.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:39.950+0000 7fa828484700 1 -- 172.21.15.134:0/2615634353 wait complete. 2024-01-21T11:49:39.951 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:49:40.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:49:40 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:49:39.949685+0000 mon.smithi029 (mon.0) 295 : audit [DBG] from='client.? 172.21.15.134:0/2615634353' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:49:41.315 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:49:41.315 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:49:41.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:49:41 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:49:39.875354+0000 mgr.smithi029.blhxxk (mgr.14164) 235 : cluster [DBG] pgmap v223: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:49:42.555 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:42.553+0000 7f0ae7579700 1 -- 172.21.15.134:0/1707017156 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ae00f5410 msgr2=0x7f0ae00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:49:42.555 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:42.553+0000 7f0ae7579700 1 --2- 172.21.15.134:0/1707017156 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ae00f5410 0x7f0ae00f5830 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7f0ad0009fb0 tx=0x7f0ad0009b30).stop 2024-01-21T11:49:42.555 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:42.553+0000 7f0ae7579700 1 -- 172.21.15.134:0/1707017156 shutdown_connections 2024-01-21T11:49:42.555 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:42.553+0000 7f0ae7579700 1 --2- 172.21.15.134:0/1707017156 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ae00f5410 0x7f0ae00f5830 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:49:42.555 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:42.553+0000 7f0ae7579700 1 -- 172.21.15.134:0/1707017156 >> 172.21.15.134:0/1707017156 conn(0x7f0ae00f0b70 msgr2=0x7f0ae00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:49:42.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:42.553+0000 7f0ae7579700 1 -- 172.21.15.134:0/1707017156 shutdown_connections 2024-01-21T11:49:42.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:42.553+0000 7f0ae7579700 1 -- 172.21.15.134:0/1707017156 wait complete. 2024-01-21T11:49:42.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:42.554+0000 7f0ae7579700 1 Processor -- start 2024-01-21T11:49:42.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:42.554+0000 7f0ae7579700 1 -- start start 2024-01-21T11:49:42.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:42.554+0000 7f0ae7579700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ae00f5410 0x7f0ae0109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:49:42.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:42.554+0000 7f0ae7579700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ae0060220 con 0x7f0ae00f5410 2024-01-21T11:49:42.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:42.554+0000 7f0ae5315700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ae00f5410 0x7f0ae0109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:49:42.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:42.554+0000 7f0ae5315700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ae00f5410 0x7f0ae0109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:43900/0 (socket says 172.21.15.134:43900) 2024-01-21T11:49:42.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:42.554+0000 7f0ae5315700 1 -- 172.21.15.134:0/802735165 learned_addr learned my addr 172.21.15.134:0/802735165 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:49:42.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:42.555+0000 7f0ae5315700 1 -- 172.21.15.134:0/802735165 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ad0005040 con 0x7f0ae00f5410 2024-01-21T11:49:42.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:42.555+0000 7f0ae5315700 1 --2- 172.21.15.134:0/802735165 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ae00f5410 0x7f0ae0109450 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7f0ad0013510 tx=0x7f0ad000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:49:42.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:42.555+0000 7f0ad5ffb700 1 -- 172.21.15.134:0/802735165 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0ad0009320 con 0x7f0ae00f5410 2024-01-21T11:49:42.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:42.555+0000 7f0ae7579700 1 -- 172.21.15.134:0/802735165 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ae0060420 con 0x7f0ae00f5410 2024-01-21T11:49:42.557 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:42.555+0000 7f0ad5ffb700 1 -- 172.21.15.134:0/802735165 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0ad0013f00 con 0x7f0ae00f5410 2024-01-21T11:49:42.557 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:42.555+0000 7f0ad5ffb700 1 -- 172.21.15.134:0/802735165 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0ad0014d90 con 0x7f0ae00f5410 2024-01-21T11:49:42.557 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:42.555+0000 7f0ae7579700 1 -- 172.21.15.134:0/802735165 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ae0109b00 con 0x7f0ae00f5410 2024-01-21T11:49:42.557 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:42.556+0000 7f0ae7579700 1 -- 172.21.15.134:0/802735165 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ae0059100 con 0x7f0ae00f5410 2024-01-21T11:49:42.557 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:42.556+0000 7f0ad5ffb700 1 -- 172.21.15.134:0/802735165 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0ad00095f0 con 0x7f0ae00f5410 2024-01-21T11:49:42.558 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:42.556+0000 7f0ad5ffb700 1 --2- 172.21.15.134:0/802735165 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f0acc02ff60 0x7f0acc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:49:42.558 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:42.556+0000 7f0ad5ffb700 1 -- 172.21.15.134:0/802735165 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0ad0041000 con 0x7f0ae00f5410 2024-01-21T11:49:42.558 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:42.556+0000 7f0ae4b14700 1 --2- 172.21.15.134:0/802735165 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f0acc02ff60 0x7f0acc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:49:42.558 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:42.557+0000 7f0ae4b14700 1 --2- 172.21.15.134:0/802735165 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f0acc02ff60 0x7f0acc032420 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f0adc0096a0 tx=0x7f0adc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:49:42.561 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:42.561+0000 7f0ad5ffb700 1 -- 172.21.15.134:0/802735165 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0ad0014420 con 0x7f0ae00f5410 2024-01-21T11:49:42.766 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:42.765+0000 7f0ae7579700 1 -- 172.21.15.134:0/802735165 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0ae0045ad0 con 0x7f0ae00f5410 2024-01-21T11:49:42.766 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:42.766+0000 7f0ad5ffb700 1 -- 172.21.15.134:0/802735165 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0ad001d7f0 con 0x7f0ae00f5410 2024-01-21T11:49:42.766 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:49:42.766 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:49:42.768 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:42.768+0000 7f0ae7579700 1 -- 172.21.15.134:0/802735165 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f0acc02ff60 msgr2=0x7f0acc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:49:42.769 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:42.768+0000 7f0ae7579700 1 --2- 172.21.15.134:0/802735165 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f0acc02ff60 0x7f0acc032420 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f0adc0096a0 tx=0x7f0adc006b40).stop 2024-01-21T11:49:42.769 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:42.768+0000 7f0ae7579700 1 -- 172.21.15.134:0/802735165 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ae00f5410 msgr2=0x7f0ae0109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:49:42.769 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:42.768+0000 7f0ae7579700 1 --2- 172.21.15.134:0/802735165 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ae00f5410 0x7f0ae0109450 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7f0ad0013510 tx=0x7f0ad000b040).stop 2024-01-21T11:49:42.769 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:42.768+0000 7f0ae7579700 1 -- 172.21.15.134:0/802735165 shutdown_connections 2024-01-21T11:49:42.769 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:42.768+0000 7f0ae7579700 1 --2- 172.21.15.134:0/802735165 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f0acc02ff60 0x7f0acc032420 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:49:42.769 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:42.768+0000 7f0ae7579700 1 --2- 172.21.15.134:0/802735165 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ae00f5410 0x7f0ae0109450 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:49:42.769 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:42.768+0000 7f0ae7579700 1 -- 172.21.15.134:0/802735165 >> 172.21.15.134:0/802735165 conn(0x7f0ae00f0b70 msgr2=0x7f0ae00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:49:42.769 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:42.768+0000 7f0ae7579700 1 -- 172.21.15.134:0/802735165 shutdown_connections 2024-01-21T11:49:42.769 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:42.768+0000 7f0ae7579700 1 -- 172.21.15.134:0/802735165 wait complete. 2024-01-21T11:49:42.769 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:49:43.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:49:43 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:49:41.875749+0000 mgr.smithi029.blhxxk (mgr.14164) 236 : cluster [DBG] pgmap v224: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:49:43.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:49:43 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:49:42.767345+0000 mon.smithi029 (mon.0) 296 : audit [DBG] from='client.? 172.21.15.134:0/802735165' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:49:44.159 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:49:44.159 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:49:45.384 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:45.383+0000 7fb7254a0700 1 -- 172.21.15.134:0/3686250269 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7200f59d0 msgr2=0x7fb7200f5df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:49:45.384 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:45.383+0000 7fb7254a0700 1 --2- 172.21.15.134:0/3686250269 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7200f59d0 0x7fb7200f5df0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7fb708009fb0 tx=0x7fb708009b30).stop 2024-01-21T11:49:45.384 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:45.383+0000 7fb7254a0700 1 -- 172.21.15.134:0/3686250269 shutdown_connections 2024-01-21T11:49:45.384 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:45.383+0000 7fb7254a0700 1 --2- 172.21.15.134:0/3686250269 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7200f59d0 0x7fb7200f5df0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:49:45.385 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:45.383+0000 7fb7254a0700 1 -- 172.21.15.134:0/3686250269 >> 172.21.15.134:0/3686250269 conn(0x7fb7200f02e0 msgr2=0x7fb7200f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:49:45.385 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:45.384+0000 7fb7254a0700 1 -- 172.21.15.134:0/3686250269 shutdown_connections 2024-01-21T11:49:45.385 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:45.384+0000 7fb7254a0700 1 -- 172.21.15.134:0/3686250269 wait complete. 2024-01-21T11:49:45.386 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:45.384+0000 7fb7254a0700 1 Processor -- start 2024-01-21T11:49:45.386 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:45.385+0000 7fb7254a0700 1 -- start start 2024-01-21T11:49:45.386 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:45.385+0000 7fb7254a0700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7200f59d0 0x7fb7201055c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:49:45.387 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:45.385+0000 7fb7254a0700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb720105ab0 con 0x7fb7200f59d0 2024-01-21T11:49:45.387 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:45.385+0000 7fb71effd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7200f59d0 0x7fb7201055c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:49:45.387 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:45.385+0000 7fb71effd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7200f59d0 0x7fb7201055c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:52610/0 (socket says 172.21.15.134:52610) 2024-01-21T11:49:45.387 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:45.385+0000 7fb71effd700 1 -- 172.21.15.134:0/4053620986 learned_addr learned my addr 172.21.15.134:0/4053620986 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:49:45.387 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:45.385+0000 7fb71effd700 1 -- 172.21.15.134:0/4053620986 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb708005040 con 0x7fb7200f59d0 2024-01-21T11:49:45.387 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:45.386+0000 7fb71effd700 1 --2- 172.21.15.134:0/4053620986 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7200f59d0 0x7fb7201055c0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7fb708013510 tx=0x7fb70800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:49:45.387 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:45.386+0000 7fb7177fe700 1 -- 172.21.15.134:0/4053620986 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb708009320 con 0x7fb7200f59d0 2024-01-21T11:49:45.387 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:45.386+0000 7fb7254a0700 1 -- 172.21.15.134:0/4053620986 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb720105cb0 con 0x7fb7200f59d0 2024-01-21T11:49:45.387 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:45.386+0000 7fb7177fe700 1 -- 172.21.15.134:0/4053620986 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb708013f00 con 0x7fb7200f59d0 2024-01-21T11:49:45.387 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:45.386+0000 7fb7177fe700 1 -- 172.21.15.134:0/4053620986 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb708014d90 con 0x7fb7200f59d0 2024-01-21T11:49:45.387 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:45.386+0000 7fb7254a0700 1 -- 172.21.15.134:0/4053620986 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb720106140 con 0x7fb7200f59d0 2024-01-21T11:49:45.388 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:45.387+0000 7fb7177fe700 1 -- 172.21.15.134:0/4053620986 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb7080095f0 con 0x7fb7200f59d0 2024-01-21T11:49:45.388 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:45.387+0000 7fb7254a0700 1 -- 172.21.15.134:0/4053620986 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb720045ad0 con 0x7fb7200f59d0 2024-01-21T11:49:45.388 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:45.387+0000 7fb7177fe700 1 --2- 172.21.15.134:0/4053620986 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fb70c02ff60 0x7fb70c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:49:45.389 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:45.387+0000 7fb7177fe700 1 -- 172.21.15.134:0/4053620986 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb708040fc0 con 0x7fb7200f59d0 2024-01-21T11:49:45.389 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:45.387+0000 7fb71e7fc700 1 --2- 172.21.15.134:0/4053620986 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fb70c02ff60 0x7fb70c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:49:45.389 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:45.388+0000 7fb71e7fc700 1 --2- 172.21.15.134:0/4053620986 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fb70c02ff60 0x7fb70c032420 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7fb7100096a0 tx=0x7fb710006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:49:45.392 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:45.392+0000 7fb7177fe700 1 -- 172.21.15.134:0/4053620986 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb708014420 con 0x7fb7200f59d0 2024-01-21T11:49:45.597 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:45.597+0000 7fb7254a0700 1 -- 172.21.15.134:0/4053620986 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb720106f90 con 0x7fb7200f59d0 2024-01-21T11:49:45.598 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:45.598+0000 7fb7177fe700 1 -- 172.21.15.134:0/4053620986 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb708010230 con 0x7fb7200f59d0 2024-01-21T11:49:45.598 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:49:45.598 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:49:45.600 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:45.599+0000 7fb7254a0700 1 -- 172.21.15.134:0/4053620986 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fb70c02ff60 msgr2=0x7fb70c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:49:45.600 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:45.599+0000 7fb7254a0700 1 --2- 172.21.15.134:0/4053620986 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fb70c02ff60 0x7fb70c032420 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7fb7100096a0 tx=0x7fb710006b40).stop 2024-01-21T11:49:45.600 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:45.599+0000 7fb7254a0700 1 -- 172.21.15.134:0/4053620986 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7200f59d0 msgr2=0x7fb7201055c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:49:45.600 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:45.599+0000 7fb7254a0700 1 --2- 172.21.15.134:0/4053620986 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7200f59d0 0x7fb7201055c0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7fb708013510 tx=0x7fb70800b040).stop 2024-01-21T11:49:45.600 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:45.599+0000 7fb7254a0700 1 -- 172.21.15.134:0/4053620986 shutdown_connections 2024-01-21T11:49:45.600 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:45.599+0000 7fb7254a0700 1 --2- 172.21.15.134:0/4053620986 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fb70c02ff60 0x7fb70c032420 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:49:45.600 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:45.599+0000 7fb7254a0700 1 --2- 172.21.15.134:0/4053620986 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7200f59d0 0x7fb7201055c0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:49:45.600 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:45.599+0000 7fb7254a0700 1 -- 172.21.15.134:0/4053620986 >> 172.21.15.134:0/4053620986 conn(0x7fb7200f02e0 msgr2=0x7fb7200f0dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:49:45.600 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:45.599+0000 7fb7254a0700 1 -- 172.21.15.134:0/4053620986 shutdown_connections 2024-01-21T11:49:45.600 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:45.599+0000 7fb7254a0700 1 -- 172.21.15.134:0/4053620986 wait complete. 2024-01-21T11:49:45.601 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:49:45.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:49:45 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:49:43.876172+0000 mgr.smithi029.blhxxk (mgr.14164) 237 : cluster [DBG] pgmap v225: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:49:45.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:49:45 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:49:45.599002+0000 mon.smithi029 (mon.0) 297 : audit [DBG] from='client.? 172.21.15.134:0/4053620986' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:49:46.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:49:46 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:49:45.876516+0000 mgr.smithi029.blhxxk (mgr.14164) 238 : cluster [DBG] pgmap v226: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:49:46.961 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:49:46.961 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:49:48.150 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:48.148+0000 7ff9ab2e5700 1 -- 172.21.15.134:0/3699786517 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9a40f47c0 msgr2=0x7ff9a40f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:49:48.151 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:48.148+0000 7ff9ab2e5700 1 --2- 172.21.15.134:0/3699786517 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9a40f47c0 0x7ff9a40f4be0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7ff9a0009fb0 tx=0x7ff9a0009b30).stop 2024-01-21T11:49:48.151 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:48.149+0000 7ff9ab2e5700 1 -- 172.21.15.134:0/3699786517 shutdown_connections 2024-01-21T11:49:48.151 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:48.149+0000 7ff9ab2e5700 1 --2- 172.21.15.134:0/3699786517 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9a40f47c0 0x7ff9a40f4be0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:49:48.151 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:48.149+0000 7ff9ab2e5700 1 -- 172.21.15.134:0/3699786517 >> 172.21.15.134:0/3699786517 conn(0x7ff9a40f02e0 msgr2=0x7ff9a40f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:49:48.151 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:48.149+0000 7ff9ab2e5700 1 -- 172.21.15.134:0/3699786517 shutdown_connections 2024-01-21T11:49:48.151 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:48.149+0000 7ff9ab2e5700 1 -- 172.21.15.134:0/3699786517 wait complete. 2024-01-21T11:49:48.151 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:48.149+0000 7ff9ab2e5700 1 Processor -- start 2024-01-21T11:49:48.151 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:48.149+0000 7ff9ab2e5700 1 -- start start 2024-01-21T11:49:48.151 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:48.149+0000 7ff9ab2e5700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9a40f47c0 0x7ff9a4106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:49:48.151 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:48.149+0000 7ff9ab2e5700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9a4106bf0 con 0x7ff9a40f47c0 2024-01-21T11:49:48.151 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:48.149+0000 7ff9a9081700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9a40f47c0 0x7ff9a4106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:49:48.151 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:48.149+0000 7ff9a9081700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9a40f47c0 0x7ff9a4106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:52620/0 (socket says 172.21.15.134:52620) 2024-01-21T11:49:48.151 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:48.149+0000 7ff9a9081700 1 -- 172.21.15.134:0/3089118797 learned_addr learned my addr 172.21.15.134:0/3089118797 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:49:48.152 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:48.150+0000 7ff9a9081700 1 -- 172.21.15.134:0/3089118797 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9a0005040 con 0x7ff9a40f47c0 2024-01-21T11:49:48.152 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:48.150+0000 7ff9a9081700 1 --2- 172.21.15.134:0/3089118797 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9a40f47c0 0x7ff9a4106700 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7ff9a0013510 tx=0x7ff9a000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:49:48.152 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:48.150+0000 7ff995ffb700 1 -- 172.21.15.134:0/3089118797 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff9a0009320 con 0x7ff9a40f47c0 2024-01-21T11:49:48.152 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:48.150+0000 7ff9ab2e5700 1 -- 172.21.15.134:0/3089118797 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9a4106df0 con 0x7ff9a40f47c0 2024-01-21T11:49:48.152 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:48.150+0000 7ff995ffb700 1 -- 172.21.15.134:0/3089118797 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff9a0013f00 con 0x7ff9a40f47c0 2024-01-21T11:49:48.152 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:48.150+0000 7ff995ffb700 1 -- 172.21.15.134:0/3089118797 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff9a0014d90 con 0x7ff9a40f47c0 2024-01-21T11:49:48.152 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:48.150+0000 7ff9ab2e5700 1 -- 172.21.15.134:0/3089118797 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9a4107280 con 0x7ff9a40f47c0 2024-01-21T11:49:48.153 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:48.151+0000 7ff995ffb700 1 -- 172.21.15.134:0/3089118797 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff9a00095f0 con 0x7ff9a40f47c0 2024-01-21T11:49:48.153 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:48.151+0000 7ff9ab2e5700 1 -- 172.21.15.134:0/3089118797 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9a4045ad0 con 0x7ff9a40f47c0 2024-01-21T11:49:48.153 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:48.151+0000 7ff995ffb700 1 --2- 172.21.15.134:0/3089118797 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7ff99002ff60 0x7ff990032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:49:48.153 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:48.151+0000 7ff995ffb700 1 -- 172.21.15.134:0/3089118797 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff9a0040fe0 con 0x7ff9a40f47c0 2024-01-21T11:49:48.153 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:48.152+0000 7ff9a8880700 1 --2- 172.21.15.134:0/3089118797 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7ff99002ff60 0x7ff990032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:49:48.153 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:48.152+0000 7ff9a8880700 1 --2- 172.21.15.134:0/3089118797 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7ff99002ff60 0x7ff990032420 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7ff9980096a0 tx=0x7ff998006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:49:48.156 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:48.156+0000 7ff995ffb700 1 -- 172.21.15.134:0/3089118797 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff9a0014420 con 0x7ff9a40f47c0 2024-01-21T11:49:48.363 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:48.363+0000 7ff9ab2e5700 1 -- 172.21.15.134:0/3089118797 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff9a41080d0 con 0x7ff9a40f47c0 2024-01-21T11:49:48.363 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:48.363+0000 7ff995ffb700 1 -- 172.21.15.134:0/3089118797 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff9a0010230 con 0x7ff9a40f47c0 2024-01-21T11:49:48.364 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:49:48.364 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:49:48.365 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:48.365+0000 7ff9ab2e5700 1 -- 172.21.15.134:0/3089118797 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7ff99002ff60 msgr2=0x7ff990032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:49:48.366 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:48.365+0000 7ff9ab2e5700 1 --2- 172.21.15.134:0/3089118797 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7ff99002ff60 0x7ff990032420 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7ff9980096a0 tx=0x7ff998006b40).stop 2024-01-21T11:49:48.366 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:48.365+0000 7ff9ab2e5700 1 -- 172.21.15.134:0/3089118797 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9a40f47c0 msgr2=0x7ff9a4106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:49:48.366 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:48.365+0000 7ff9ab2e5700 1 --2- 172.21.15.134:0/3089118797 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9a40f47c0 0x7ff9a4106700 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7ff9a0013510 tx=0x7ff9a000b040).stop 2024-01-21T11:49:48.366 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:48.365+0000 7ff9ab2e5700 1 -- 172.21.15.134:0/3089118797 shutdown_connections 2024-01-21T11:49:48.366 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:48.365+0000 7ff9ab2e5700 1 --2- 172.21.15.134:0/3089118797 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7ff99002ff60 0x7ff990032420 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:49:48.366 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:48.365+0000 7ff9ab2e5700 1 --2- 172.21.15.134:0/3089118797 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9a40f47c0 0x7ff9a4106700 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:49:48.366 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:48.365+0000 7ff9ab2e5700 1 -- 172.21.15.134:0/3089118797 >> 172.21.15.134:0/3089118797 conn(0x7ff9a40f02e0 msgr2=0x7ff9a40f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:49:48.366 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:48.365+0000 7ff9ab2e5700 1 -- 172.21.15.134:0/3089118797 shutdown_connections 2024-01-21T11:49:48.366 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:48.365+0000 7ff9ab2e5700 1 -- 172.21.15.134:0/3089118797 wait complete. 2024-01-21T11:49:48.367 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:49:49.418 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:49:48 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:49:47.876939+0000 mgr.smithi029.blhxxk (mgr.14164) 239 : cluster [DBG] pgmap v227: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:49:49.418 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:49:48 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:49:48.364691+0000 mon.smithi029 (mon.0) 298 : audit [DBG] from='client.? 172.21.15.134:0/3089118797' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:49:49.698 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:49:49.698 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:49:50.861 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:50.861+0000 7fa1650d4700 1 -- 172.21.15.134:0/3255650426 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1600f51a0 msgr2=0x7fa1600f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:49:50.862 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:50.861+0000 7fa1650d4700 1 --2- 172.21.15.134:0/3255650426 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1600f51a0 0x7fa1600f55c0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7fa150009fb0 tx=0x7fa150009b30).stop 2024-01-21T11:49:50.862 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:50.861+0000 7fa1650d4700 1 -- 172.21.15.134:0/3255650426 shutdown_connections 2024-01-21T11:49:50.862 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:50.861+0000 7fa1650d4700 1 --2- 172.21.15.134:0/3255650426 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1600f51a0 0x7fa1600f55c0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:49:50.862 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:50.861+0000 7fa1650d4700 1 -- 172.21.15.134:0/3255650426 >> 172.21.15.134:0/3255650426 conn(0x7fa1600f09a0 msgr2=0x7fa1600f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:49:50.862 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:50.861+0000 7fa1650d4700 1 -- 172.21.15.134:0/3255650426 shutdown_connections 2024-01-21T11:49:50.862 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:50.861+0000 7fa1650d4700 1 -- 172.21.15.134:0/3255650426 wait complete. 2024-01-21T11:49:50.862 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:50.861+0000 7fa1650d4700 1 Processor -- start 2024-01-21T11:49:50.862 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:50.862+0000 7fa1650d4700 1 -- start start 2024-01-21T11:49:50.864 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:50.862+0000 7fa1650d4700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1600f51a0 0x7fa1601064a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:49:50.864 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:50.862+0000 7fa1650d4700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa160106990 con 0x7fa1600f51a0 2024-01-21T11:49:50.864 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:50.862+0000 7fa15ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1600f51a0 0x7fa1601064a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:49:50.864 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:50.862+0000 7fa15ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1600f51a0 0x7fa1601064a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:52650/0 (socket says 172.21.15.134:52650) 2024-01-21T11:49:50.864 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:50.862+0000 7fa15ffff700 1 -- 172.21.15.134:0/3775377934 learned_addr learned my addr 172.21.15.134:0/3775377934 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:49:50.864 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:50.862+0000 7fa15ffff700 1 -- 172.21.15.134:0/3775377934 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa150005040 con 0x7fa1600f51a0 2024-01-21T11:49:50.864 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:50.862+0000 7fa15ffff700 1 --2- 172.21.15.134:0/3775377934 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1600f51a0 0x7fa1601064a0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7fa150013510 tx=0x7fa15000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:49:50.864 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:50.863+0000 7fa15cff9700 1 -- 172.21.15.134:0/3775377934 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa150009320 con 0x7fa1600f51a0 2024-01-21T11:49:50.864 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:50.863+0000 7fa1650d4700 1 -- 172.21.15.134:0/3775377934 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa160106b90 con 0x7fa1600f51a0 2024-01-21T11:49:50.864 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:50.863+0000 7fa15cff9700 1 -- 172.21.15.134:0/3775377934 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa150013f00 con 0x7fa1600f51a0 2024-01-21T11:49:50.865 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:50.863+0000 7fa15cff9700 1 -- 172.21.15.134:0/3775377934 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa150014dc0 con 0x7fa1600f51a0 2024-01-21T11:49:50.865 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:50.863+0000 7fa1650d4700 1 -- 172.21.15.134:0/3775377934 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa160107020 con 0x7fa1600f51a0 2024-01-21T11:49:50.865 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:50.863+0000 7fa15cff9700 1 -- 172.21.15.134:0/3775377934 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa1500095f0 con 0x7fa1600f51a0 2024-01-21T11:49:50.865 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:50.863+0000 7fa1650d4700 1 -- 172.21.15.134:0/3775377934 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa160045ad0 con 0x7fa1600f51a0 2024-01-21T11:49:50.865 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:50.864+0000 7fa15cff9700 1 --2- 172.21.15.134:0/3775377934 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fa14c02ff60 0x7fa14c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:49:50.865 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:50.864+0000 7fa15cff9700 1 -- 172.21.15.134:0/3775377934 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa1500410c0 con 0x7fa1600f51a0 2024-01-21T11:49:50.865 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:50.864+0000 7fa15f7fe700 1 --2- 172.21.15.134:0/3775377934 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fa14c02ff60 0x7fa14c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:49:50.865 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:50.864+0000 7fa15f7fe700 1 --2- 172.21.15.134:0/3775377934 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fa14c02ff60 0x7fa14c032420 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fa1480096a0 tx=0x7fa148006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:49:50.868 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:50.868+0000 7fa15cff9700 1 -- 172.21.15.134:0/3775377934 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa150014420 con 0x7fa1600f51a0 2024-01-21T11:49:51.075 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:51.075+0000 7fa1650d4700 1 -- 172.21.15.134:0/3775377934 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa160107ed0 con 0x7fa1600f51a0 2024-01-21T11:49:51.076 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:51.076+0000 7fa15cff9700 1 -- 172.21.15.134:0/3775377934 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa150010230 con 0x7fa1600f51a0 2024-01-21T11:49:51.076 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:49:51.077 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:49:51.077 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:51.077+0000 7fa1650d4700 1 -- 172.21.15.134:0/3775377934 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fa14c02ff60 msgr2=0x7fa14c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:49:51.078 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:51.077+0000 7fa1650d4700 1 --2- 172.21.15.134:0/3775377934 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fa14c02ff60 0x7fa14c032420 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fa1480096a0 tx=0x7fa148006b40).stop 2024-01-21T11:49:51.078 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:51.077+0000 7fa1650d4700 1 -- 172.21.15.134:0/3775377934 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1600f51a0 msgr2=0x7fa1601064a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:49:51.078 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:51.077+0000 7fa1650d4700 1 --2- 172.21.15.134:0/3775377934 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1600f51a0 0x7fa1601064a0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7fa150013510 tx=0x7fa15000b040).stop 2024-01-21T11:49:51.078 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:51.077+0000 7fa1650d4700 1 -- 172.21.15.134:0/3775377934 shutdown_connections 2024-01-21T11:49:51.078 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:51.077+0000 7fa1650d4700 1 --2- 172.21.15.134:0/3775377934 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fa14c02ff60 0x7fa14c032420 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:49:51.078 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:51.077+0000 7fa1650d4700 1 --2- 172.21.15.134:0/3775377934 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1600f51a0 0x7fa1601064a0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:49:51.078 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:51.077+0000 7fa1650d4700 1 -- 172.21.15.134:0/3775377934 >> 172.21.15.134:0/3775377934 conn(0x7fa1600f09a0 msgr2=0x7fa16005fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:49:51.078 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:51.077+0000 7fa1650d4700 1 -- 172.21.15.134:0/3775377934 shutdown_connections 2024-01-21T11:49:51.078 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:51.077+0000 7fa1650d4700 1 -- 172.21.15.134:0/3775377934 wait complete. 2024-01-21T11:49:51.079 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:49:51.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:49:51 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:49:49.877311+0000 mgr.smithi029.blhxxk (mgr.14164) 240 : cluster [DBG] pgmap v228: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:49:51.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:49:51 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:49:51.076719+0000 mon.smithi029 (mon.0) 299 : audit [DBG] from='client.? 172.21.15.134:0/3775377934' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:49:52.561 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:49:52.561 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:49:53.694 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:53.693+0000 7f30f9f64700 1 -- 172.21.15.134:0/2643925255 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30f40f6510 msgr2=0x7f30f40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:49:53.694 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:53.693+0000 7f30f9f64700 1 --2- 172.21.15.134:0/2643925255 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30f40f6510 0x7f30f40f6930 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7f30e8009fb0 tx=0x7f30e8009b30).stop 2024-01-21T11:49:53.695 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:53.693+0000 7f30f9f64700 1 -- 172.21.15.134:0/2643925255 shutdown_connections 2024-01-21T11:49:53.695 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:53.693+0000 7f30f9f64700 1 --2- 172.21.15.134:0/2643925255 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30f40f6510 0x7f30f40f6930 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:49:53.695 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:53.693+0000 7f30f9f64700 1 -- 172.21.15.134:0/2643925255 >> 172.21.15.134:0/2643925255 conn(0x7f30f40f1c70 msgr2=0x7f30f40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:49:53.695 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:53.693+0000 7f30f9f64700 1 -- 172.21.15.134:0/2643925255 shutdown_connections 2024-01-21T11:49:53.695 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:53.693+0000 7f30f9f64700 1 -- 172.21.15.134:0/2643925255 wait complete. 2024-01-21T11:49:53.695 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:53.694+0000 7f30f9f64700 1 Processor -- start 2024-01-21T11:49:53.695 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:53.694+0000 7f30f9f64700 1 -- start start 2024-01-21T11:49:53.695 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:53.694+0000 7f30f9f64700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30f4108a10 0x7f30f4108e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:49:53.695 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:53.694+0000 7f30f9f64700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30f4109320 con 0x7f30f4108a10 2024-01-21T11:49:53.697 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:53.695+0000 7f30f37fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30f4108a10 0x7f30f4108e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:49:53.697 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:53.695+0000 7f30f37fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30f4108a10 0x7f30f4108e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:52660/0 (socket says 172.21.15.134:52660) 2024-01-21T11:49:53.697 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:53.695+0000 7f30f37fe700 1 -- 172.21.15.134:0/1353405309 learned_addr learned my addr 172.21.15.134:0/1353405309 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:49:53.697 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:53.695+0000 7f30f37fe700 1 -- 172.21.15.134:0/1353405309 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30e8005040 con 0x7f30f4108a10 2024-01-21T11:49:53.697 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:53.695+0000 7f30f37fe700 1 --2- 172.21.15.134:0/1353405309 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30f4108a10 0x7f30f4108e30 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7f30e8009f80 tx=0x7f30e8009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:49:53.697 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:53.695+0000 7f30dbfff700 1 -- 172.21.15.134:0/1353405309 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f30e80136c0 con 0x7f30f4108a10 2024-01-21T11:49:53.697 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:53.695+0000 7f30f9f64700 1 -- 172.21.15.134:0/1353405309 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30f4109520 con 0x7f30f4108a10 2024-01-21T11:49:53.697 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:53.695+0000 7f30dbfff700 1 -- 172.21.15.134:0/1353405309 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f30e80095f0 con 0x7f30f4108a10 2024-01-21T11:49:53.697 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:53.695+0000 7f30dbfff700 1 -- 172.21.15.134:0/1353405309 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f30e8014420 con 0x7f30f4108a10 2024-01-21T11:49:53.697 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:53.695+0000 7f30f9f64700 1 -- 172.21.15.134:0/1353405309 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30f410a070 con 0x7f30f4108a10 2024-01-21T11:49:53.697 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:53.696+0000 7f30dbfff700 1 -- 172.21.15.134:0/1353405309 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f30e8025060 con 0x7f30f4108a10 2024-01-21T11:49:53.697 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:53.696+0000 7f30f9f64700 1 -- 172.21.15.134:0/1353405309 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30f4059100 con 0x7f30f4108a10 2024-01-21T11:49:53.697 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:53.696+0000 7f30dbfff700 1 --2- 172.21.15.134:0/1353405309 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f30dc02ff60 0x7f30dc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:49:53.698 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:53.696+0000 7f30dbfff700 1 -- 172.21.15.134:0/1353405309 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f30e8040e40 con 0x7f30f4108a10 2024-01-21T11:49:53.698 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:53.696+0000 7f30f2ffd700 1 --2- 172.21.15.134:0/1353405309 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f30dc02ff60 0x7f30dc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:49:53.698 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:53.697+0000 7f30f2ffd700 1 --2- 172.21.15.134:0/1353405309 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f30dc02ff60 0x7f30dc032420 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f30e40096a0 tx=0x7f30e4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:49:53.701 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:53.701+0000 7f30dbfff700 1 -- 172.21.15.134:0/1353405309 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f30e801c950 con 0x7f30f4108a10 2024-01-21T11:49:53.907 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:53.906+0000 7f30f9f64700 1 -- 172.21.15.134:0/1353405309 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f30f410a7d0 con 0x7f30f4108a10 2024-01-21T11:49:53.907 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:53.907+0000 7f30dbfff700 1 -- 172.21.15.134:0/1353405309 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f30e801cb60 con 0x7f30f4108a10 2024-01-21T11:49:53.908 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:49:53.908 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:49:53.909 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:53.909+0000 7f30f9f64700 1 -- 172.21.15.134:0/1353405309 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f30dc02ff60 msgr2=0x7f30dc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:49:53.909 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:53.909+0000 7f30f9f64700 1 --2- 172.21.15.134:0/1353405309 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f30dc02ff60 0x7f30dc032420 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f30e40096a0 tx=0x7f30e4006b40).stop 2024-01-21T11:49:53.910 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:53.909+0000 7f30f9f64700 1 -- 172.21.15.134:0/1353405309 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30f4108a10 msgr2=0x7f30f4108e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:49:53.910 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:53.909+0000 7f30f9f64700 1 --2- 172.21.15.134:0/1353405309 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30f4108a10 0x7f30f4108e30 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7f30e8009f80 tx=0x7f30e8009b30).stop 2024-01-21T11:49:53.910 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:53.909+0000 7f30f9f64700 1 -- 172.21.15.134:0/1353405309 shutdown_connections 2024-01-21T11:49:53.910 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:53.909+0000 7f30f9f64700 1 --2- 172.21.15.134:0/1353405309 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f30dc02ff60 0x7f30dc032420 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:49:53.910 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:53.909+0000 7f30f9f64700 1 --2- 172.21.15.134:0/1353405309 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30f4108a10 0x7f30f4108e30 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:49:53.910 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:53.909+0000 7f30f9f64700 1 -- 172.21.15.134:0/1353405309 >> 172.21.15.134:0/1353405309 conn(0x7f30f40f1c70 msgr2=0x7f30f40f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:49:53.910 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:53.909+0000 7f30f9f64700 1 -- 172.21.15.134:0/1353405309 shutdown_connections 2024-01-21T11:49:53.910 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:53.909+0000 7f30f9f64700 1 -- 172.21.15.134:0/1353405309 wait complete. 2024-01-21T11:49:53.910 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:49:53.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:49:53 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:49:51.877678+0000 mgr.smithi029.blhxxk (mgr.14164) 241 : cluster [DBG] pgmap v229: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:49:54.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:49:54 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:49:53.908403+0000 mon.smithi029 (mon.0) 300 : audit [DBG] from='client.? 172.21.15.134:0/1353405309' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:49:55.301 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:49:55.302 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:49:55.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:49:55 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:49:53.878028+0000 mgr.smithi029.blhxxk (mgr.14164) 242 : cluster [DBG] pgmap v230: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:49:56.450 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:56.449+0000 7f6571dbf700 1 -- 172.21.15.134:0/1629713157 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f656c060280 msgr2=0x7f656c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:49:56.450 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:56.449+0000 7f6571dbf700 1 --2- 172.21.15.134:0/1629713157 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f656c060280 0x7f656c0f90f0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7f6554009fb0 tx=0x7f6554009b30).stop 2024-01-21T11:49:56.450 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:56.449+0000 7f6571dbf700 1 -- 172.21.15.134:0/1629713157 shutdown_connections 2024-01-21T11:49:56.450 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:56.449+0000 7f6571dbf700 1 --2- 172.21.15.134:0/1629713157 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f656c060280 0x7f656c0f90f0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:49:56.450 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:56.449+0000 7f6571dbf700 1 -- 172.21.15.134:0/1629713157 >> 172.21.15.134:0/1629713157 conn(0x7f656c0f0b70 msgr2=0x7f656c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:49:56.451 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:56.449+0000 7f6571dbf700 1 -- 172.21.15.134:0/1629713157 shutdown_connections 2024-01-21T11:49:56.451 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:56.449+0000 7f6571dbf700 1 -- 172.21.15.134:0/1629713157 wait complete. 2024-01-21T11:49:56.451 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:56.450+0000 7f6571dbf700 1 Processor -- start 2024-01-21T11:49:56.451 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:56.450+0000 7f6571dbf700 1 -- start start 2024-01-21T11:49:56.451 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:56.450+0000 7f6571dbf700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f656c060280 0x7f656c1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:49:56.451 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:56.450+0000 7f6571dbf700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f656c106bd0 con 0x7f656c060280 2024-01-21T11:49:56.452 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:56.450+0000 7f656b7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f656c060280 0x7f656c1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:49:56.453 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:56.450+0000 7f656b7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f656c060280 0x7f656c1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:44592/0 (socket says 172.21.15.134:44592) 2024-01-21T11:49:56.453 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:56.450+0000 7f656b7fe700 1 -- 172.21.15.134:0/1747308244 learned_addr learned my addr 172.21.15.134:0/1747308244 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:49:56.453 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:56.451+0000 7f656b7fe700 1 -- 172.21.15.134:0/1747308244 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6554005040 con 0x7f656c060280 2024-01-21T11:49:56.453 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:56.451+0000 7f656b7fe700 1 --2- 172.21.15.134:0/1747308244 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f656c060280 0x7f656c1066e0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f6554013510 tx=0x7f655400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:49:56.453 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:56.451+0000 7f6563fff700 1 -- 172.21.15.134:0/1747308244 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6554009320 con 0x7f656c060280 2024-01-21T11:49:56.453 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:56.451+0000 7f6563fff700 1 -- 172.21.15.134:0/1747308244 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6554013f00 con 0x7f656c060280 2024-01-21T11:49:56.453 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:56.451+0000 7f6571dbf700 1 -- 172.21.15.134:0/1747308244 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f656c106dd0 con 0x7f656c060280 2024-01-21T11:49:56.453 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:56.451+0000 7f6563fff700 1 -- 172.21.15.134:0/1747308244 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6554014dc0 con 0x7f656c060280 2024-01-21T11:49:56.453 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:56.451+0000 7f6571dbf700 1 -- 172.21.15.134:0/1747308244 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f656c107260 con 0x7f656c060280 2024-01-21T11:49:56.453 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:56.452+0000 7f6563fff700 1 -- 172.21.15.134:0/1747308244 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f65540095f0 con 0x7f656c060280 2024-01-21T11:49:56.453 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:56.452+0000 7f6571dbf700 1 -- 172.21.15.134:0/1747308244 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f656c045ad0 con 0x7f656c060280 2024-01-21T11:49:56.453 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:56.452+0000 7f6563fff700 1 --2- 172.21.15.134:0/1747308244 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f655802ffb0 0x7f6558032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:49:56.453 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:56.452+0000 7f6563fff700 1 -- 172.21.15.134:0/1747308244 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6554040fe0 con 0x7f656c060280 2024-01-21T11:49:56.454 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:56.452+0000 7f656affd700 1 --2- 172.21.15.134:0/1747308244 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f655802ffb0 0x7f6558032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:49:56.454 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:56.453+0000 7f656affd700 1 --2- 172.21.15.134:0/1747308244 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f655802ffb0 0x7f6558032470 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f655c0096a0 tx=0x7f655c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:49:56.457 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:56.457+0000 7f6563fff700 1 -- 172.21.15.134:0/1747308244 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f655406d2b0 con 0x7f656c060280 2024-01-21T11:49:56.661 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:56.661+0000 7f6571dbf700 1 -- 172.21.15.134:0/1747308244 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f656c108110 con 0x7f656c060280 2024-01-21T11:49:56.661 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:56.661+0000 7f6563fff700 1 -- 172.21.15.134:0/1747308244 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6554014420 con 0x7f656c060280 2024-01-21T11:49:56.662 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:49:56.662 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:49:56.663 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:56.663+0000 7f6571dbf700 1 -- 172.21.15.134:0/1747308244 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f655802ffb0 msgr2=0x7f6558032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:49:56.663 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:56.663+0000 7f6571dbf700 1 --2- 172.21.15.134:0/1747308244 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f655802ffb0 0x7f6558032470 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f655c0096a0 tx=0x7f655c006b40).stop 2024-01-21T11:49:56.664 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:56.663+0000 7f6571dbf700 1 -- 172.21.15.134:0/1747308244 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f656c060280 msgr2=0x7f656c1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:49:56.664 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:56.663+0000 7f6571dbf700 1 --2- 172.21.15.134:0/1747308244 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f656c060280 0x7f656c1066e0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f6554013510 tx=0x7f655400b040).stop 2024-01-21T11:49:56.664 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:56.663+0000 7f6571dbf700 1 -- 172.21.15.134:0/1747308244 shutdown_connections 2024-01-21T11:49:56.664 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:56.663+0000 7f6571dbf700 1 --2- 172.21.15.134:0/1747308244 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f655802ffb0 0x7f6558032470 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:49:56.664 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:56.663+0000 7f6571dbf700 1 --2- 172.21.15.134:0/1747308244 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f656c060280 0x7f656c1066e0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:49:56.664 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:56.663+0000 7f6571dbf700 1 -- 172.21.15.134:0/1747308244 >> 172.21.15.134:0/1747308244 conn(0x7f656c0f0b70 msgr2=0x7f656c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:49:56.664 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:56.663+0000 7f6571dbf700 1 -- 172.21.15.134:0/1747308244 shutdown_connections 2024-01-21T11:49:56.664 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:56.663+0000 7f6571dbf700 1 -- 172.21.15.134:0/1747308244 wait complete. 2024-01-21T11:49:56.664 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:49:57.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:49:57 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:49:55.878367+0000 mgr.smithi029.blhxxk (mgr.14164) 243 : cluster [DBG] pgmap v231: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:49:57.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:49:57 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:49:56.662749+0000 mon.smithi029 (mon.0) 301 : audit [DBG] from='client.? 172.21.15.134:0/1747308244' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:49:58.030 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:49:58.030 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:49:59.198 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:59.196+0000 7f532968a700 1 -- 172.21.15.134:0/2275512762 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53240f6510 msgr2=0x7f53240f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:49:59.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:59.196+0000 7f532968a700 1 --2- 172.21.15.134:0/2275512762 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53240f6510 0x7f53240f6930 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7f5314009fb0 tx=0x7f5314009b30).stop 2024-01-21T11:49:59.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:59.196+0000 7f532968a700 1 -- 172.21.15.134:0/2275512762 shutdown_connections 2024-01-21T11:49:59.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:59.196+0000 7f532968a700 1 --2- 172.21.15.134:0/2275512762 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53240f6510 0x7f53240f6930 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:49:59.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:59.196+0000 7f532968a700 1 -- 172.21.15.134:0/2275512762 >> 172.21.15.134:0/2275512762 conn(0x7f53240f1c70 msgr2=0x7f53240f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:49:59.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:59.197+0000 7f532968a700 1 -- 172.21.15.134:0/2275512762 shutdown_connections 2024-01-21T11:49:59.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:59.197+0000 7f532968a700 1 -- 172.21.15.134:0/2275512762 wait complete. 2024-01-21T11:49:59.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:59.197+0000 7f532968a700 1 Processor -- start 2024-01-21T11:49:59.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:59.197+0000 7f532968a700 1 -- start start 2024-01-21T11:49:59.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:59.197+0000 7f532968a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53240f6510 0x7f532406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:49:59.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:59.197+0000 7f532968a700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5324069e20 con 0x7f53240f6510 2024-01-21T11:49:59.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:59.197+0000 7f5322ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53240f6510 0x7f532406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:49:59.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:59.198+0000 7f5322ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53240f6510 0x7f532406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:44612/0 (socket says 172.21.15.134:44612) 2024-01-21T11:49:59.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:59.198+0000 7f5322ffd700 1 -- 172.21.15.134:0/2911644518 learned_addr learned my addr 172.21.15.134:0/2911644518 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:49:59.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:59.198+0000 7f5322ffd700 1 -- 172.21.15.134:0/2911644518 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5314005040 con 0x7f53240f6510 2024-01-21T11:49:59.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:59.198+0000 7f5322ffd700 1 --2- 172.21.15.134:0/2911644518 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53240f6510 0x7f532406b9a0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7f5314014000 tx=0x7f531400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:49:59.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:59.198+0000 7f530b7fe700 1 -- 172.21.15.134:0/2911644518 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f53140095f0 con 0x7f53240f6510 2024-01-21T11:49:59.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:59.198+0000 7f530b7fe700 1 -- 172.21.15.134:0/2911644518 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5314007e70 con 0x7f53240f6510 2024-01-21T11:49:59.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:59.198+0000 7f532968a700 1 -- 172.21.15.134:0/2911644518 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5324069fc0 con 0x7f53240f6510 2024-01-21T11:49:59.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:59.198+0000 7f530b7fe700 1 -- 172.21.15.134:0/2911644518 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5314014ea0 con 0x7f53240f6510 2024-01-21T11:49:59.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:59.198+0000 7f532968a700 1 -- 172.21.15.134:0/2911644518 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f532406a450 con 0x7f53240f6510 2024-01-21T11:49:59.201 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:59.199+0000 7f530b7fe700 1 -- 172.21.15.134:0/2911644518 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5314025070 con 0x7f53240f6510 2024-01-21T11:49:59.201 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:59.199+0000 7f532968a700 1 -- 172.21.15.134:0/2911644518 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5324059100 con 0x7f53240f6510 2024-01-21T11:49:59.201 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:59.199+0000 7f530b7fe700 1 --2- 172.21.15.134:0/2911644518 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f530c02ff60 0x7f530c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:49:59.202 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:59.199+0000 7f530b7fe700 1 -- 172.21.15.134:0/2911644518 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5314040930 con 0x7f53240f6510 2024-01-21T11:49:59.202 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:59.200+0000 7f53227fc700 1 --2- 172.21.15.134:0/2911644518 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f530c02ff60 0x7f530c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:49:59.202 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:59.200+0000 7f53227fc700 1 --2- 172.21.15.134:0/2911644518 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f530c02ff60 0x7f530c032420 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f53180096a0 tx=0x7f5318006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:49:59.204 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:59.204+0000 7f530b7fe700 1 -- 172.21.15.134:0/2911644518 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f531400f4e0 con 0x7f53240f6510 2024-01-21T11:49:59.411 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:59.411+0000 7f532968a700 1 -- 172.21.15.134:0/2911644518 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f53240fbf60 con 0x7f53240f6510 2024-01-21T11:49:59.411 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:59.411+0000 7f530b7fe700 1 -- 172.21.15.134:0/2911644518 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f531400f300 con 0x7f53240f6510 2024-01-21T11:49:59.411 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:49:59.412 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:49:59.413 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:59.412+0000 7f532968a700 1 -- 172.21.15.134:0/2911644518 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f530c02ff60 msgr2=0x7f530c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:49:59.413 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:59.412+0000 7f532968a700 1 --2- 172.21.15.134:0/2911644518 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f530c02ff60 0x7f530c032420 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f53180096a0 tx=0x7f5318006b40).stop 2024-01-21T11:49:59.413 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:59.412+0000 7f532968a700 1 -- 172.21.15.134:0/2911644518 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53240f6510 msgr2=0x7f532406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:49:59.413 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:59.412+0000 7f532968a700 1 --2- 172.21.15.134:0/2911644518 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53240f6510 0x7f532406b9a0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7f5314014000 tx=0x7f531400b040).stop 2024-01-21T11:49:59.413 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:59.413+0000 7f532968a700 1 -- 172.21.15.134:0/2911644518 shutdown_connections 2024-01-21T11:49:59.413 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:59.413+0000 7f532968a700 1 --2- 172.21.15.134:0/2911644518 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f530c02ff60 0x7f530c032420 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:49:59.413 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:59.413+0000 7f532968a700 1 --2- 172.21.15.134:0/2911644518 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53240f6510 0x7f532406b9a0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:49:59.413 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:59.413+0000 7f532968a700 1 -- 172.21.15.134:0/2911644518 >> 172.21.15.134:0/2911644518 conn(0x7f53240f1c70 msgr2=0x7f53240f27a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:49:59.413 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:59.413+0000 7f532968a700 1 -- 172.21.15.134:0/2911644518 shutdown_connections 2024-01-21T11:49:59.413 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:49:59.413+0000 7f532968a700 1 -- 172.21.15.134:0/2911644518 wait complete. 2024-01-21T11:49:59.414 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:49:59.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:49:59 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:49:57.878675+0000 mgr.smithi029.blhxxk (mgr.14164) 244 : cluster [DBG] pgmap v232: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:49:59.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:49:59 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:49:59.412383+0000 mon.smithi029 (mon.0) 302 : audit [DBG] from='client.? 172.21.15.134:0/2911644518' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:50:00.800 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:50:00.800 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:50:01.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:01 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:49:59.879042+0000 mgr.smithi029.blhxxk (mgr.14164) 245 : cluster [DBG] pgmap v233: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:50:01.992 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:01.990+0000 7f72ca00e700 1 -- 172.21.15.134:0/2660793988 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72c40f51a0 msgr2=0x7f72c40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:50:01.992 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:01.990+0000 7f72ca00e700 1 --2- 172.21.15.134:0/2660793988 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72c40f51a0 0x7f72c40f55c0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7f72c0009fb0 tx=0x7f72c0009b30).stop 2024-01-21T11:50:01.992 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:01.990+0000 7f72ca00e700 1 -- 172.21.15.134:0/2660793988 shutdown_connections 2024-01-21T11:50:01.992 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:01.990+0000 7f72ca00e700 1 --2- 172.21.15.134:0/2660793988 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72c40f51a0 0x7f72c40f55c0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:50:01.992 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:01.990+0000 7f72ca00e700 1 -- 172.21.15.134:0/2660793988 >> 172.21.15.134:0/2660793988 conn(0x7f72c40f09a0 msgr2=0x7f72c40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:50:01.992 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:01.990+0000 7f72ca00e700 1 -- 172.21.15.134:0/2660793988 shutdown_connections 2024-01-21T11:50:01.992 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:01.990+0000 7f72ca00e700 1 -- 172.21.15.134:0/2660793988 wait complete. 2024-01-21T11:50:01.992 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:01.990+0000 7f72ca00e700 1 Processor -- start 2024-01-21T11:50:01.993 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:01.990+0000 7f72ca00e700 1 -- start start 2024-01-21T11:50:01.993 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:01.990+0000 7f72ca00e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72c40f51a0 0x7f72c40fb3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:50:01.993 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:01.990+0000 7f72ca00e700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72c40f9830 con 0x7f72c40f51a0 2024-01-21T11:50:01.993 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:01.991+0000 7f72c900c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72c40f51a0 0x7f72c40fb3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:50:01.993 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:01.991+0000 7f72c900c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72c40f51a0 0x7f72c40fb3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:44642/0 (socket says 172.21.15.134:44642) 2024-01-21T11:50:01.993 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:01.991+0000 7f72c900c700 1 -- 172.21.15.134:0/661661444 learned_addr learned my addr 172.21.15.134:0/661661444 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:50:01.993 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:01.991+0000 7f72c900c700 1 -- 172.21.15.134:0/661661444 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72c0005040 con 0x7f72c40f51a0 2024-01-21T11:50:01.993 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:01.991+0000 7f72c900c700 1 --2- 172.21.15.134:0/661661444 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72c40f51a0 0x7f72c40fb3b0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7f72c0009830 tx=0x7f72c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:50:01.993 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:01.991+0000 7f72b9ffb700 1 -- 172.21.15.134:0/661661444 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f72c0009320 con 0x7f72c40f51a0 2024-01-21T11:50:01.993 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:01.991+0000 7f72b9ffb700 1 -- 172.21.15.134:0/661661444 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f72c00136c0 con 0x7f72c40f51a0 2024-01-21T11:50:01.993 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:01.991+0000 7f72ca00e700 1 -- 172.21.15.134:0/661661444 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72c40f99d0 con 0x7f72c40f51a0 2024-01-21T11:50:01.993 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:01.991+0000 7f72b9ffb700 1 -- 172.21.15.134:0/661661444 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f72c001ee90 con 0x7f72c40f51a0 2024-01-21T11:50:01.993 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:01.991+0000 7f72ca00e700 1 -- 172.21.15.134:0/661661444 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72c40f9e60 con 0x7f72c40f51a0 2024-01-21T11:50:01.994 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:01.992+0000 7f72b9ffb700 1 -- 172.21.15.134:0/661661444 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f72c001e420 con 0x7f72c40f51a0 2024-01-21T11:50:01.994 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:01.992+0000 7f72ca00e700 1 -- 172.21.15.134:0/661661444 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72c4059100 con 0x7f72c40f51a0 2024-01-21T11:50:01.994 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:01.992+0000 7f72b9ffb700 1 --2- 172.21.15.134:0/661661444 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f72b002ff10 0x7f72b00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:50:01.994 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:01.992+0000 7f72b9ffb700 1 -- 172.21.15.134:0/661661444 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f72c0040fc0 con 0x7f72c40f51a0 2024-01-21T11:50:01.994 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:01.993+0000 7f72c880b700 1 --2- 172.21.15.134:0/661661444 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f72b002ff10 0x7f72b00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:50:01.995 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:01.993+0000 7f72c880b700 1 --2- 172.21.15.134:0/661661444 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f72b002ff10 0x7f72b00323d0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f72b40096a0 tx=0x7f72b4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:50:01.997 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:01.997+0000 7f72b9ffb700 1 -- 172.21.15.134:0/661661444 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f72c006d2b0 con 0x7f72c40f51a0 2024-01-21T11:50:02.206 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:02.206+0000 7f72ca00e700 1 -- 172.21.15.134:0/661661444 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f72c4045ad0 con 0x7f72c40f51a0 2024-01-21T11:50:02.206 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:02.206+0000 7f72b9ffb700 1 -- 172.21.15.134:0/661661444 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f72c00095f0 con 0x7f72c40f51a0 2024-01-21T11:50:02.207 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:50:02.207 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:50:02.208 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:02.208+0000 7f72ca00e700 1 -- 172.21.15.134:0/661661444 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f72b002ff10 msgr2=0x7f72b00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:50:02.208 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:02.208+0000 7f72ca00e700 1 --2- 172.21.15.134:0/661661444 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f72b002ff10 0x7f72b00323d0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f72b40096a0 tx=0x7f72b4006b40).stop 2024-01-21T11:50:02.208 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:02.208+0000 7f72ca00e700 1 -- 172.21.15.134:0/661661444 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72c40f51a0 msgr2=0x7f72c40fb3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:50:02.208 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:02.208+0000 7f72ca00e700 1 --2- 172.21.15.134:0/661661444 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72c40f51a0 0x7f72c40fb3b0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7f72c0009830 tx=0x7f72c000b040).stop 2024-01-21T11:50:02.208 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:02.208+0000 7f72ca00e700 1 -- 172.21.15.134:0/661661444 shutdown_connections 2024-01-21T11:50:02.208 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:02.208+0000 7f72ca00e700 1 --2- 172.21.15.134:0/661661444 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f72b002ff10 0x7f72b00323d0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:50:02.208 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:02.208+0000 7f72ca00e700 1 --2- 172.21.15.134:0/661661444 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72c40f51a0 0x7f72c40fb3b0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:50:02.209 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:02.208+0000 7f72ca00e700 1 -- 172.21.15.134:0/661661444 >> 172.21.15.134:0/661661444 conn(0x7f72c40f09a0 msgr2=0x7f72c40f1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:50:02.209 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:02.208+0000 7f72ca00e700 1 -- 172.21.15.134:0/661661444 shutdown_connections 2024-01-21T11:50:02.209 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:02.208+0000 7f72ca00e700 1 -- 172.21.15.134:0/661661444 wait complete. 2024-01-21T11:50:02.209 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:50:02.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:02 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:50:01.945159+0000 mon.smithi029 (mon.0) 303 : audit [INF] from='mgr.14164 172.21.15.29:0/3252518221' entity='mgr.smithi029.blhxxk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi029.blhxxk/mirror_snapshot_schedule"}]: dispatch 2024-01-21T11:50:02.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:02 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:50:01.946449+0000 mon.smithi029 (mon.0) 304 : audit [INF] from='mgr.14164 172.21.15.29:0/3252518221' entity='mgr.smithi029.blhxxk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi029.blhxxk/trash_purge_schedule"}]: dispatch 2024-01-21T11:50:02.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:02 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:50:02.207608+0000 mon.smithi029 (mon.0) 305 : audit [DBG] from='client.? 172.21.15.134:0/661661444' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:50:03.584 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:50:03.585 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:50:03.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:03 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cephadm 2024-01-21T11:50:01.710075+0000 mgr.smithi029.blhxxk (mgr.14164) 246 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi029 ... 2024-01-21T11:50:03.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:03 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: Verifying port 9100 ... 2024-01-21T11:50:03.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:03 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: Non-zero exit code 1 from systemctl start ceph-e4084074-b851-11ee-95b0-87774f69a715@node-exporter.smithi029 2024-01-21T11:50:03.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:03 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: systemctl: stderr Job for ceph-e4084074-b851-11ee-95b0-87774f69a715@node-exporter.smithi029.service failed because a timeout was exceeded. 2024-01-21T11:50:03.920 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:03 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: systemctl: stderr See "systemctl status ceph-e4084074-b851-11ee-95b0-87774f69a715@node-exporter.smithi029.service" and "journalctl -xe" for details. 2024-01-21T11:50:03.920 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:03 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: Traceback (most recent call last): 2024-01-21T11:50:03.920 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:03 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: File "/var/lib/ceph/e4084074-b851-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-21T11:50:03.920 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:03 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: main() 2024-01-21T11:50:03.920 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:03 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: File "/var/lib/ceph/e4084074-b851-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-21T11:50:03.920 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:03 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: r = ctx.func(ctx) 2024-01-21T11:50:03.920 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:03 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: File "/var/lib/ceph/e4084074-b851-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-21T11:50:03.920 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:03 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: return func(ctx) 2024-01-21T11:50:03.920 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:03 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: File "/var/lib/ceph/e4084074-b851-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2024-01-21T11:50:03.920 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:03 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: ports=daemon_ports) 2024-01-21T11:50:03.920 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:03 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: File "/var/lib/ceph/e4084074-b851-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2024-01-21T11:50:03.920 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:03 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: c, osd_fsid=osd_fsid, ports=ports) 2024-01-21T11:50:03.920 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:03 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: File "/var/lib/ceph/e4084074-b851-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2024-01-21T11:50:03.921 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:03 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2024-01-21T11:50:03.921 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:03 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: File "/var/lib/ceph/e4084074-b851-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2024-01-21T11:50:03.921 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:03 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-21T11:50:03.921 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:03 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: RuntimeError: Failed command: systemctl start ceph-e4084074-b851-11ee-95b0-87774f69a715@node-exporter.smithi029 2024-01-21T11:50:03.921 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:03 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: Traceback (most recent call last): 2024-01-21T11:50:03.921 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:03 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2024-01-21T11:50:03.921 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:03 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: yield (conn, connr) 2024-01-21T11:50:03.921 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:03 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2024-01-21T11:50:03.921 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:03 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: code, '\n'.join(err))) 2024-01-21T11:50:03.921 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:03 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi029 ... 2024-01-21T11:50:03.921 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:03 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: Verifying port 9100 ... 2024-01-21T11:50:03.921 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:03 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: Non-zero exit code 1 from systemctl start ceph-e4084074-b851-11ee-95b0-87774f69a715@node-exporter.smithi029 2024-01-21T11:50:03.921 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:03 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: systemctl: stderr Job for ceph-e4084074-b851-11ee-95b0-87774f69a715@node-exporter.smithi029.service failed because a timeout was exceeded. 2024-01-21T11:50:03.921 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:03 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: systemctl: stderr See "systemctl status ceph-e4084074-b851-11ee-95b0-87774f69a715@node-exporter.smithi029.service" and "journalctl -xe" for details. 2024-01-21T11:50:03.921 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:03 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: Traceback (most recent call last): 2024-01-21T11:50:03.921 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:03 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: File "/var/lib/ceph/e4084074-b851-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-21T11:50:03.922 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:03 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: main() 2024-01-21T11:50:03.922 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:03 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: File "/var/lib/ceph/e4084074-b851-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-21T11:50:03.922 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:03 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: r = ctx.func(ctx) 2024-01-21T11:50:03.922 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:03 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: File "/var/lib/ceph/e4084074-b851-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-21T11:50:03.922 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:03 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: return func(ctx) 2024-01-21T11:50:03.922 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:03 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: File "/var/lib/ceph/e4084074-b851-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2024-01-21T11:50:03.922 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:03 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: ports=daemon_ports) 2024-01-21T11:50:03.922 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:03 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: File "/var/lib/ceph/e4084074-b851-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2024-01-21T11:50:03.922 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:03 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: c, osd_fsid=osd_fsid, ports=ports) 2024-01-21T11:50:03.922 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:03 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: File "/var/lib/ceph/e4084074-b851-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2024-01-21T11:50:03.922 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:03 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2024-01-21T11:50:03.922 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:03 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: File "/var/lib/ceph/e4084074-b851-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2024-01-21T11:50:03.922 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:03 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-21T11:50:03.922 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:03 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: RuntimeError: Failed command: systemctl start ceph-e4084074-b851-11ee-95b0-87774f69a715@node-exporter.smithi029 2024-01-21T11:50:03.923 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:03 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cephadm 2024-01-21T11:50:01.715903+0000 mgr.smithi029.blhxxk (mgr.14164) 247 : cephadm [INF] Deploying daemon prometheus.smithi029 on smithi029 2024-01-21T11:50:03.923 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:03 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:50:01.879327+0000 mgr.smithi029.blhxxk (mgr.14164) 248 : cluster [DBG] pgmap v234: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:50:04.832 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:04.831+0000 7ff37ec38700 1 -- 172.21.15.134:0/2079860376 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3780f47c0 msgr2=0x7ff3780f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:50:04.832 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:04.831+0000 7ff37ec38700 1 --2- 172.21.15.134:0/2079860376 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3780f47c0 0x7ff3780f4be0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7ff36c009fb0 tx=0x7ff36c009b30).stop 2024-01-21T11:50:04.832 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:04.831+0000 7ff37ec38700 1 -- 172.21.15.134:0/2079860376 shutdown_connections 2024-01-21T11:50:04.832 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:04.831+0000 7ff37ec38700 1 --2- 172.21.15.134:0/2079860376 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3780f47c0 0x7ff3780f4be0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:50:04.832 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:04.831+0000 7ff37ec38700 1 -- 172.21.15.134:0/2079860376 >> 172.21.15.134:0/2079860376 conn(0x7ff3780f02e0 msgr2=0x7ff3780f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:50:04.833 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:04.831+0000 7ff37ec38700 1 -- 172.21.15.134:0/2079860376 shutdown_connections 2024-01-21T11:50:04.833 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:04.831+0000 7ff37ec38700 1 -- 172.21.15.134:0/2079860376 wait complete. 2024-01-21T11:50:04.833 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:04.832+0000 7ff37ec38700 1 Processor -- start 2024-01-21T11:50:04.833 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:04.832+0000 7ff37ec38700 1 -- start start 2024-01-21T11:50:04.834 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:04.832+0000 7ff37ec38700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3780f47c0 0x7ff378106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:50:04.834 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:04.832+0000 7ff37ec38700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff378106bf0 con 0x7ff3780f47c0 2024-01-21T11:50:04.834 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:04.833+0000 7ff37c9d4700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3780f47c0 0x7ff378106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:50:04.834 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:04.833+0000 7ff37c9d4700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3780f47c0 0x7ff378106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:34872/0 (socket says 172.21.15.134:34872) 2024-01-21T11:50:04.835 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:04.833+0000 7ff37c9d4700 1 -- 172.21.15.134:0/271128037 learned_addr learned my addr 172.21.15.134:0/271128037 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:50:04.835 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:04.833+0000 7ff37c9d4700 1 -- 172.21.15.134:0/271128037 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff36c005040 con 0x7ff3780f47c0 2024-01-21T11:50:04.835 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:04.833+0000 7ff37c9d4700 1 --2- 172.21.15.134:0/271128037 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3780f47c0 0x7ff378106700 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7ff36c013510 tx=0x7ff36c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:50:04.835 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:04.833+0000 7ff3757fa700 1 -- 172.21.15.134:0/271128037 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff36c009320 con 0x7ff3780f47c0 2024-01-21T11:50:04.835 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:04.833+0000 7ff37ec38700 1 -- 172.21.15.134:0/271128037 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff378106df0 con 0x7ff3780f47c0 2024-01-21T11:50:04.835 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:04.833+0000 7ff3757fa700 1 -- 172.21.15.134:0/271128037 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff36c013f00 con 0x7ff3780f47c0 2024-01-21T11:50:04.835 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:04.833+0000 7ff3757fa700 1 -- 172.21.15.134:0/271128037 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff36c014d90 con 0x7ff3780f47c0 2024-01-21T11:50:04.835 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:04.833+0000 7ff37ec38700 1 -- 172.21.15.134:0/271128037 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff378107280 con 0x7ff3780f47c0 2024-01-21T11:50:04.835 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:04.834+0000 7ff3757fa700 1 -- 172.21.15.134:0/271128037 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff36c0095f0 con 0x7ff3780f47c0 2024-01-21T11:50:04.835 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:04.834+0000 7ff37ec38700 1 -- 172.21.15.134:0/271128037 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff378045ad0 con 0x7ff3780f47c0 2024-01-21T11:50:04.835 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:04.834+0000 7ff3757fa700 1 --2- 172.21.15.134:0/271128037 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7ff36802ff60 0x7ff368032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:50:04.835 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:04.834+0000 7ff3757fa700 1 -- 172.21.15.134:0/271128037 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff36c041070 con 0x7ff3780f47c0 2024-01-21T11:50:04.836 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:04.835+0000 7ff377fff700 1 --2- 172.21.15.134:0/271128037 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7ff36802ff60 0x7ff368032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:50:04.836 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:04.835+0000 7ff377fff700 1 --2- 172.21.15.134:0/271128037 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7ff36802ff60 0x7ff368032420 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7ff3640096a0 tx=0x7ff364006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:50:04.839 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:04.839+0000 7ff3757fa700 1 -- 172.21.15.134:0/271128037 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff36c014420 con 0x7ff3780f47c0 2024-01-21T11:50:05.045 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:05.044+0000 7ff37ec38700 1 -- 172.21.15.134:0/271128037 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff378108130 con 0x7ff3780f47c0 2024-01-21T11:50:05.045 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:05.045+0000 7ff3757fa700 1 -- 172.21.15.134:0/271128037 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff36c010230 con 0x7ff3780f47c0 2024-01-21T11:50:05.045 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:50:05.046 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:50:05.047 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:05.046+0000 7ff37ec38700 1 -- 172.21.15.134:0/271128037 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7ff36802ff60 msgr2=0x7ff368032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:50:05.047 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:05.046+0000 7ff37ec38700 1 --2- 172.21.15.134:0/271128037 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7ff36802ff60 0x7ff368032420 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7ff3640096a0 tx=0x7ff364006b40).stop 2024-01-21T11:50:05.047 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:05.046+0000 7ff37ec38700 1 -- 172.21.15.134:0/271128037 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3780f47c0 msgr2=0x7ff378106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:50:05.047 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:05.046+0000 7ff37ec38700 1 --2- 172.21.15.134:0/271128037 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3780f47c0 0x7ff378106700 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7ff36c013510 tx=0x7ff36c00b040).stop 2024-01-21T11:50:05.047 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:05.047+0000 7ff37ec38700 1 -- 172.21.15.134:0/271128037 shutdown_connections 2024-01-21T11:50:05.047 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:05.047+0000 7ff37ec38700 1 --2- 172.21.15.134:0/271128037 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7ff36802ff60 0x7ff368032420 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:50:05.047 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:05.047+0000 7ff37ec38700 1 --2- 172.21.15.134:0/271128037 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3780f47c0 0x7ff378106700 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:50:05.047 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:05.047+0000 7ff37ec38700 1 -- 172.21.15.134:0/271128037 >> 172.21.15.134:0/271128037 conn(0x7ff3780f02e0 msgr2=0x7ff3780f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:50:05.047 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:05.047+0000 7ff37ec38700 1 -- 172.21.15.134:0/271128037 shutdown_connections 2024-01-21T11:50:05.047 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:05.047+0000 7ff37ec38700 1 -- 172.21.15.134:0/271128037 wait complete. 2024-01-21T11:50:05.048 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:50:05.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:50:03.879680+0000 mgr.smithi029.blhxxk (mgr.14164) 249 : cluster [DBG] pgmap v235: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:50:05.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:05 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:50:05.046342+0000 mon.smithi029 (mon.0) 306 : audit [DBG] from='client.? 172.21.15.134:0/271128037' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:50:06.421 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:50:06.421 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:50:07.569 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:07.568+0000 7f8689018700 1 -- 172.21.15.134:0/2913189783 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86840f5410 msgr2=0x7f86840f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:50:07.569 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:07.568+0000 7f8689018700 1 --2- 172.21.15.134:0/2913189783 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86840f5410 0x7f86840f5830 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7f866c009fb0 tx=0x7f866c009b30).stop 2024-01-21T11:50:07.569 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:07.568+0000 7f8689018700 1 -- 172.21.15.134:0/2913189783 shutdown_connections 2024-01-21T11:50:07.569 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:07.568+0000 7f8689018700 1 --2- 172.21.15.134:0/2913189783 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86840f5410 0x7f86840f5830 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:50:07.569 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:07.568+0000 7f8689018700 1 -- 172.21.15.134:0/2913189783 >> 172.21.15.134:0/2913189783 conn(0x7f86840f0b70 msgr2=0x7f86840f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:50:07.569 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:07.568+0000 7f8689018700 1 -- 172.21.15.134:0/2913189783 shutdown_connections 2024-01-21T11:50:07.569 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:07.568+0000 7f8689018700 1 -- 172.21.15.134:0/2913189783 wait complete. 2024-01-21T11:50:07.571 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:07.569+0000 7f8689018700 1 Processor -- start 2024-01-21T11:50:07.571 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:07.569+0000 7f8689018700 1 -- start start 2024-01-21T11:50:07.571 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:07.569+0000 7f8689018700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86840f5410 0x7f8684105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:50:07.571 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:07.569+0000 7f8689018700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8684105b20 con 0x7f86840f5410 2024-01-21T11:50:07.571 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:07.570+0000 7f8682d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86840f5410 0x7f8684105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:50:07.571 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:07.570+0000 7f8682d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86840f5410 0x7f8684105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:34888/0 (socket says 172.21.15.134:34888) 2024-01-21T11:50:07.571 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:07.570+0000 7f8682d9d700 1 -- 172.21.15.134:0/3241215920 learned_addr learned my addr 172.21.15.134:0/3241215920 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:50:07.571 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:07.570+0000 7f8682d9d700 1 -- 172.21.15.134:0/3241215920 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f866c005040 con 0x7f86840f5410 2024-01-21T11:50:07.571 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:07.570+0000 7f8682d9d700 1 --2- 172.21.15.134:0/3241215920 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86840f5410 0x7f8684105630 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7f866c013510 tx=0x7f866c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:50:07.571 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:07.570+0000 7f867b7fe700 1 -- 172.21.15.134:0/3241215920 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f866c009320 con 0x7f86840f5410 2024-01-21T11:50:07.571 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:07.570+0000 7f8689018700 1 -- 172.21.15.134:0/3241215920 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8684105d20 con 0x7f86840f5410 2024-01-21T11:50:07.571 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:07.570+0000 7f867b7fe700 1 -- 172.21.15.134:0/3241215920 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f866c013f00 con 0x7f86840f5410 2024-01-21T11:50:07.571 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:07.570+0000 7f867b7fe700 1 -- 172.21.15.134:0/3241215920 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f866c014d90 con 0x7f86840f5410 2024-01-21T11:50:07.572 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:07.570+0000 7f8689018700 1 -- 172.21.15.134:0/3241215920 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86841061b0 con 0x7f86840f5410 2024-01-21T11:50:07.572 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:07.571+0000 7f867b7fe700 1 -- 172.21.15.134:0/3241215920 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f866c0263a0 con 0x7f86840f5410 2024-01-21T11:50:07.573 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:07.571+0000 7f8689018700 1 -- 172.21.15.134:0/3241215920 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8684045ad0 con 0x7f86840f5410 2024-01-21T11:50:07.573 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:07.571+0000 7f867b7fe700 1 --2- 172.21.15.134:0/3241215920 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f867002ffb0 0x7f8670032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:50:07.573 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:07.571+0000 7f867b7fe700 1 -- 172.21.15.134:0/3241215920 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f866c041380 con 0x7f86840f5410 2024-01-21T11:50:07.573 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:07.572+0000 7f868259c700 1 --2- 172.21.15.134:0/3241215920 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f867002ffb0 0x7f8670032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:50:07.573 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:07.572+0000 7f868259c700 1 --2- 172.21.15.134:0/3241215920 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f867002ffb0 0x7f8670032470 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f86740096a0 tx=0x7f8674006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:50:07.576 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:07.576+0000 7f867b7fe700 1 -- 172.21.15.134:0/3241215920 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f866c01d3d0 con 0x7f86840f5410 2024-01-21T11:50:07.781 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:07.781+0000 7f8689018700 1 -- 172.21.15.134:0/3241215920 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f86840fae60 con 0x7f86840f5410 2024-01-21T11:50:07.782 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:07.782+0000 7f867b7fe700 1 -- 172.21.15.134:0/3241215920 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f866c012330 con 0x7f86840f5410 2024-01-21T11:50:07.782 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:50:07.782 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:50:07.784 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:07.783+0000 7f8689018700 1 -- 172.21.15.134:0/3241215920 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f867002ffb0 msgr2=0x7f8670032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:50:07.784 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:07.783+0000 7f8689018700 1 --2- 172.21.15.134:0/3241215920 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f867002ffb0 0x7f8670032470 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f86740096a0 tx=0x7f8674006b40).stop 2024-01-21T11:50:07.784 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:07.783+0000 7f8689018700 1 -- 172.21.15.134:0/3241215920 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86840f5410 msgr2=0x7f8684105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:50:07.784 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:07.783+0000 7f8689018700 1 --2- 172.21.15.134:0/3241215920 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86840f5410 0x7f8684105630 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7f866c013510 tx=0x7f866c00b040).stop 2024-01-21T11:50:07.784 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:07.783+0000 7f8689018700 1 -- 172.21.15.134:0/3241215920 shutdown_connections 2024-01-21T11:50:07.784 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:07.783+0000 7f8689018700 1 --2- 172.21.15.134:0/3241215920 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f867002ffb0 0x7f8670032470 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:50:07.784 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:07.784+0000 7f8689018700 1 --2- 172.21.15.134:0/3241215920 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86840f5410 0x7f8684105630 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:50:07.784 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:07.784+0000 7f8689018700 1 -- 172.21.15.134:0/3241215920 >> 172.21.15.134:0/3241215920 conn(0x7f86840f0b70 msgr2=0x7f86840f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:50:07.784 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:07.784+0000 7f8689018700 1 -- 172.21.15.134:0/3241215920 shutdown_connections 2024-01-21T11:50:07.784 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:07.784+0000 7f8689018700 1 -- 172.21.15.134:0/3241215920 wait complete. 2024-01-21T11:50:07.785 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:50:07.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:07 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:50:05.880015+0000 mgr.smithi029.blhxxk (mgr.14164) 250 : cluster [DBG] pgmap v236: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:50:08.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:08 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:50:07.783164+0000 mon.smithi029 (mon.0) 307 : audit [DBG] from='client.? 172.21.15.134:0/3241215920' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:50:09.167 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:50:09.167 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:50:09.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:09 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:50:07.880365+0000 mgr.smithi029.blhxxk (mgr.14164) 251 : cluster [DBG] pgmap v237: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:50:10.293 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:10.292+0000 7f531e496700 1 -- 172.21.15.134:0/2432337299 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53180f5410 msgr2=0x7f53180f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:50:10.293 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:10.292+0000 7f531e496700 1 --2- 172.21.15.134:0/2432337299 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53180f5410 0x7f53180f5830 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7f530c009fb0 tx=0x7f530c009b30).stop 2024-01-21T11:50:10.293 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:10.292+0000 7f531e496700 1 -- 172.21.15.134:0/2432337299 shutdown_connections 2024-01-21T11:50:10.293 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:10.292+0000 7f531e496700 1 --2- 172.21.15.134:0/2432337299 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53180f5410 0x7f53180f5830 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:50:10.293 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:10.292+0000 7f531e496700 1 -- 172.21.15.134:0/2432337299 >> 172.21.15.134:0/2432337299 conn(0x7f53180f0b70 msgr2=0x7f53180f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:50:10.293 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:10.292+0000 7f531e496700 1 -- 172.21.15.134:0/2432337299 shutdown_connections 2024-01-21T11:50:10.293 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:10.292+0000 7f531e496700 1 -- 172.21.15.134:0/2432337299 wait complete. 2024-01-21T11:50:10.294 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:10.293+0000 7f531e496700 1 Processor -- start 2024-01-21T11:50:10.294 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:10.293+0000 7f531e496700 1 -- start start 2024-01-21T11:50:10.294 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:10.293+0000 7f531e496700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53180f5410 0x7f5318109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:50:10.294 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:10.293+0000 7f531e496700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5318060220 con 0x7f53180f5410 2024-01-21T11:50:10.295 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:10.293+0000 7f5317fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53180f5410 0x7f5318109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:50:10.295 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:10.293+0000 7f5317fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53180f5410 0x7f5318109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:34896/0 (socket says 172.21.15.134:34896) 2024-01-21T11:50:10.295 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:10.293+0000 7f5317fff700 1 -- 172.21.15.134:0/924766608 learned_addr learned my addr 172.21.15.134:0/924766608 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:50:10.295 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:10.294+0000 7f5317fff700 1 -- 172.21.15.134:0/924766608 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f530c005040 con 0x7f53180f5410 2024-01-21T11:50:10.295 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:10.294+0000 7f5317fff700 1 --2- 172.21.15.134:0/924766608 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53180f5410 0x7f5318109450 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7f530c013510 tx=0x7f530c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:50:10.296 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:10.294+0000 7f5314ff9700 1 -- 172.21.15.134:0/924766608 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f530c009320 con 0x7f53180f5410 2024-01-21T11:50:10.296 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:10.294+0000 7f531e496700 1 -- 172.21.15.134:0/924766608 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5318060420 con 0x7f53180f5410 2024-01-21T11:50:10.296 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:10.294+0000 7f5314ff9700 1 -- 172.21.15.134:0/924766608 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f530c013f00 con 0x7f53180f5410 2024-01-21T11:50:10.296 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:10.294+0000 7f5314ff9700 1 -- 172.21.15.134:0/924766608 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f530c014d90 con 0x7f53180f5410 2024-01-21T11:50:10.296 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:10.294+0000 7f531e496700 1 -- 172.21.15.134:0/924766608 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5318109b00 con 0x7f53180f5410 2024-01-21T11:50:10.296 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:10.295+0000 7f531e496700 1 -- 172.21.15.134:0/924766608 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5318059100 con 0x7f53180f5410 2024-01-21T11:50:10.296 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:10.295+0000 7f5314ff9700 1 -- 172.21.15.134:0/924766608 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f530c0095f0 con 0x7f53180f5410 2024-01-21T11:50:10.296 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:10.295+0000 7f5314ff9700 1 --2- 172.21.15.134:0/924766608 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f530002ff60 0x7f5300032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:50:10.296 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:10.295+0000 7f5314ff9700 1 -- 172.21.15.134:0/924766608 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f530c040fe0 con 0x7f53180f5410 2024-01-21T11:50:10.296 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:10.295+0000 7f53177fe700 1 --2- 172.21.15.134:0/924766608 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f530002ff60 0x7f5300032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:50:10.297 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:10.296+0000 7f53177fe700 1 --2- 172.21.15.134:0/924766608 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f530002ff60 0x7f5300032420 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f53080096a0 tx=0x7f5308006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:50:10.300 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:10.300+0000 7f5314ff9700 1 -- 172.21.15.134:0/924766608 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f530c01d7f0 con 0x7f53180f5410 2024-01-21T11:50:10.505 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:10.505+0000 7f531e496700 1 -- 172.21.15.134:0/924766608 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5318045ad0 con 0x7f53180f5410 2024-01-21T11:50:10.506 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:10.506+0000 7f5314ff9700 1 -- 172.21.15.134:0/924766608 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f530c010230 con 0x7f53180f5410 2024-01-21T11:50:10.506 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:50:10.506 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:50:10.508 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:10.508+0000 7f531e496700 1 -- 172.21.15.134:0/924766608 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f530002ff60 msgr2=0x7f5300032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:50:10.508 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:10.508+0000 7f531e496700 1 --2- 172.21.15.134:0/924766608 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f530002ff60 0x7f5300032420 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f53080096a0 tx=0x7f5308006b40).stop 2024-01-21T11:50:10.508 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:10.508+0000 7f531e496700 1 -- 172.21.15.134:0/924766608 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53180f5410 msgr2=0x7f5318109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:50:10.508 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:10.508+0000 7f531e496700 1 --2- 172.21.15.134:0/924766608 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53180f5410 0x7f5318109450 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7f530c013510 tx=0x7f530c00b040).stop 2024-01-21T11:50:10.508 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:10.508+0000 7f531e496700 1 -- 172.21.15.134:0/924766608 shutdown_connections 2024-01-21T11:50:10.508 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:10.508+0000 7f531e496700 1 --2- 172.21.15.134:0/924766608 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f530002ff60 0x7f5300032420 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:50:10.508 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:10.508+0000 7f531e496700 1 --2- 172.21.15.134:0/924766608 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53180f5410 0x7f5318109450 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:50:10.509 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:10.508+0000 7f531e496700 1 -- 172.21.15.134:0/924766608 >> 172.21.15.134:0/924766608 conn(0x7f53180f0b70 msgr2=0x7f53180f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:50:10.509 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:10.508+0000 7f531e496700 1 -- 172.21.15.134:0/924766608 shutdown_connections 2024-01-21T11:50:10.509 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:10.508+0000 7f531e496700 1 -- 172.21.15.134:0/924766608 wait complete. 2024-01-21T11:50:10.509 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:50:10.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:10 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:50:10.507246+0000 mon.smithi029 (mon.0) 308 : audit [DBG] from='client.? 172.21.15.134:0/924766608' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:50:11.897 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:50:11.897 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:50:11.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:11 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:50:09.880617+0000 mgr.smithi029.blhxxk (mgr.14164) 252 : cluster [DBG] pgmap v238: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:50:13.081 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:13.079+0000 7fa82d251700 1 -- 172.21.15.134:0/2186910056 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8280f6510 msgr2=0x7fa8280f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:50:13.081 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:13.079+0000 7fa82d251700 1 --2- 172.21.15.134:0/2186910056 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8280f6510 0x7fa8280f6930 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7fa810009fb0 tx=0x7fa810009b30).stop 2024-01-21T11:50:13.082 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:13.079+0000 7fa82d251700 1 -- 172.21.15.134:0/2186910056 shutdown_connections 2024-01-21T11:50:13.082 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:13.079+0000 7fa82d251700 1 --2- 172.21.15.134:0/2186910056 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8280f6510 0x7fa8280f6930 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:50:13.082 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:13.079+0000 7fa82d251700 1 -- 172.21.15.134:0/2186910056 >> 172.21.15.134:0/2186910056 conn(0x7fa8280f1c70 msgr2=0x7fa8280f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:50:13.082 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:13.079+0000 7fa82d251700 1 -- 172.21.15.134:0/2186910056 shutdown_connections 2024-01-21T11:50:13.082 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:13.079+0000 7fa82d251700 1 -- 172.21.15.134:0/2186910056 wait complete. 2024-01-21T11:50:13.082 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:13.080+0000 7fa82d251700 1 Processor -- start 2024-01-21T11:50:13.082 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:13.080+0000 7fa82d251700 1 -- start start 2024-01-21T11:50:13.082 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:13.080+0000 7fa82d251700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8280f6510 0x7fa82806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:50:13.082 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:13.080+0000 7fa82d251700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa828069e20 con 0x7fa8280f6510 2024-01-21T11:50:13.082 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:13.080+0000 7fa826d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8280f6510 0x7fa82806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:50:13.082 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:13.080+0000 7fa826d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8280f6510 0x7fa82806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:34910/0 (socket says 172.21.15.134:34910) 2024-01-21T11:50:13.082 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:13.080+0000 7fa826d9d700 1 -- 172.21.15.134:0/4121847769 learned_addr learned my addr 172.21.15.134:0/4121847769 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:50:13.082 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:13.081+0000 7fa826d9d700 1 -- 172.21.15.134:0/4121847769 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa810005040 con 0x7fa8280f6510 2024-01-21T11:50:13.083 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:13.081+0000 7fa826d9d700 1 --2- 172.21.15.134:0/4121847769 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8280f6510 0x7fa82806b9a0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7fa810014000 tx=0x7fa81000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:50:13.083 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:13.081+0000 7fa81f7fe700 1 -- 172.21.15.134:0/4121847769 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa8100095f0 con 0x7fa8280f6510 2024-01-21T11:50:13.083 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:13.081+0000 7fa82d251700 1 -- 172.21.15.134:0/4121847769 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa828069fc0 con 0x7fa8280f6510 2024-01-21T11:50:13.083 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:13.081+0000 7fa81f7fe700 1 -- 172.21.15.134:0/4121847769 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa810007e70 con 0x7fa8280f6510 2024-01-21T11:50:13.083 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:13.081+0000 7fa81f7fe700 1 -- 172.21.15.134:0/4121847769 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa810014ea0 con 0x7fa8280f6510 2024-01-21T11:50:13.083 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:13.081+0000 7fa82d251700 1 -- 172.21.15.134:0/4121847769 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa82806a450 con 0x7fa8280f6510 2024-01-21T11:50:13.084 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:13.082+0000 7fa81f7fe700 1 -- 172.21.15.134:0/4121847769 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa810025070 con 0x7fa8280f6510 2024-01-21T11:50:13.084 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:13.082+0000 7fa82d251700 1 -- 172.21.15.134:0/4121847769 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa828059100 con 0x7fa8280f6510 2024-01-21T11:50:13.084 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:13.082+0000 7fa81f7fe700 1 --2- 172.21.15.134:0/4121847769 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fa81402ff60 0x7fa814032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:50:13.084 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:13.082+0000 7fa81f7fe700 1 -- 172.21.15.134:0/4121847769 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa8100409b0 con 0x7fa8280f6510 2024-01-21T11:50:13.084 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:13.082+0000 7fa82659c700 1 --2- 172.21.15.134:0/4121847769 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fa81402ff60 0x7fa814032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:50:13.084 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:13.083+0000 7fa82659c700 1 --2- 172.21.15.134:0/4121847769 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fa81402ff60 0x7fa814032420 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7fa8180096a0 tx=0x7fa818006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:50:13.087 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:13.087+0000 7fa81f7fe700 1 -- 172.21.15.134:0/4121847769 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa8100079c0 con 0x7fa8280f6510 2024-01-21T11:50:13.294 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:13.294+0000 7fa82d251700 1 -- 172.21.15.134:0/4121847769 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa8280fbf60 con 0x7fa8280f6510 2024-01-21T11:50:13.294 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:13.294+0000 7fa81f7fe700 1 -- 172.21.15.134:0/4121847769 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa81000f300 con 0x7fa8280f6510 2024-01-21T11:50:13.295 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:50:13.295 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:50:13.296 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:13.296+0000 7fa82d251700 1 -- 172.21.15.134:0/4121847769 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fa81402ff60 msgr2=0x7fa814032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:50:13.296 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:13.296+0000 7fa82d251700 1 --2- 172.21.15.134:0/4121847769 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fa81402ff60 0x7fa814032420 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7fa8180096a0 tx=0x7fa818006b40).stop 2024-01-21T11:50:13.297 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:13.296+0000 7fa82d251700 1 -- 172.21.15.134:0/4121847769 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8280f6510 msgr2=0x7fa82806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:50:13.297 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:13.296+0000 7fa82d251700 1 --2- 172.21.15.134:0/4121847769 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8280f6510 0x7fa82806b9a0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7fa810014000 tx=0x7fa81000b040).stop 2024-01-21T11:50:13.297 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:13.296+0000 7fa82d251700 1 -- 172.21.15.134:0/4121847769 shutdown_connections 2024-01-21T11:50:13.297 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:13.296+0000 7fa82d251700 1 --2- 172.21.15.134:0/4121847769 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fa81402ff60 0x7fa814032420 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:50:13.297 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:13.296+0000 7fa82d251700 1 --2- 172.21.15.134:0/4121847769 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8280f6510 0x7fa82806b9a0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:50:13.297 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:13.296+0000 7fa82d251700 1 -- 172.21.15.134:0/4121847769 >> 172.21.15.134:0/4121847769 conn(0x7fa8280f1c70 msgr2=0x7fa8280f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:50:13.297 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:13.296+0000 7fa82d251700 1 -- 172.21.15.134:0/4121847769 shutdown_connections 2024-01-21T11:50:13.297 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:13.296+0000 7fa82d251700 1 -- 172.21.15.134:0/4121847769 wait complete. 2024-01-21T11:50:13.297 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:50:13.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:13 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:50:11.880916+0000 mgr.smithi029.blhxxk (mgr.14164) 253 : cluster [DBG] pgmap v239: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:50:13.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:13 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:50:13.295674+0000 mon.smithi029 (mon.0) 309 : audit [DBG] from='client.? 172.21.15.134:0/4121847769' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:50:14.671 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:50:14.672 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:50:15.894 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:15.893+0000 7fd7293b5700 1 -- 172.21.15.134:0/2330799257 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7240f5410 msgr2=0x7fd7240f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:50:15.894 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:15.893+0000 7fd7293b5700 1 --2- 172.21.15.134:0/2330799257 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7240f5410 0x7fd7240f5830 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7fd714009fb0 tx=0x7fd714009b30).stop 2024-01-21T11:50:15.894 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:15.893+0000 7fd7293b5700 1 -- 172.21.15.134:0/2330799257 shutdown_connections 2024-01-21T11:50:15.894 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:15.893+0000 7fd7293b5700 1 --2- 172.21.15.134:0/2330799257 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7240f5410 0x7fd7240f5830 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:50:15.894 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:15.893+0000 7fd7293b5700 1 -- 172.21.15.134:0/2330799257 >> 172.21.15.134:0/2330799257 conn(0x7fd7240f0b70 msgr2=0x7fd7240f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:50:15.894 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:15.893+0000 7fd7293b5700 1 -- 172.21.15.134:0/2330799257 shutdown_connections 2024-01-21T11:50:15.894 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:15.893+0000 7fd7293b5700 1 -- 172.21.15.134:0/2330799257 wait complete. 2024-01-21T11:50:15.894 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:15.894+0000 7fd7293b5700 1 Processor -- start 2024-01-21T11:50:15.894 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:15.894+0000 7fd7293b5700 1 -- start start 2024-01-21T11:50:15.894 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:15.894+0000 7fd7293b5700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7240f5410 0x7fd7241055d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:50:15.894 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:15.894+0000 7fd7293b5700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd724105ac0 con 0x7fd7240f5410 2024-01-21T11:50:15.895 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:15.895+0000 7fd722ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7240f5410 0x7fd7241055d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:50:15.895 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:15.895+0000 7fd722ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7240f5410 0x7fd7241055d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:57394/0 (socket says 172.21.15.134:57394) 2024-01-21T11:50:15.895 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:15.895+0000 7fd722ffd700 1 -- 172.21.15.134:0/2654948465 learned_addr learned my addr 172.21.15.134:0/2654948465 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:50:15.896 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:15.895+0000 7fd722ffd700 1 -- 172.21.15.134:0/2654948465 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd714005040 con 0x7fd7240f5410 2024-01-21T11:50:15.896 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:15.895+0000 7fd722ffd700 1 --2- 172.21.15.134:0/2654948465 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7240f5410 0x7fd7241055d0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7fd714013510 tx=0x7fd71400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:50:15.896 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:15.895+0000 7fd70b7fe700 1 -- 172.21.15.134:0/2654948465 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd714009320 con 0x7fd7240f5410 2024-01-21T11:50:15.896 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:15.895+0000 7fd7293b5700 1 -- 172.21.15.134:0/2654948465 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd724105cc0 con 0x7fd7240f5410 2024-01-21T11:50:15.897 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:15.895+0000 7fd7293b5700 1 -- 172.21.15.134:0/2654948465 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd724106150 con 0x7fd7240f5410 2024-01-21T11:50:15.897 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:15.895+0000 7fd70b7fe700 1 -- 172.21.15.134:0/2654948465 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd714013f00 con 0x7fd7240f5410 2024-01-21T11:50:15.897 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:15.895+0000 7fd70b7fe700 1 -- 172.21.15.134:0/2654948465 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd714014d60 con 0x7fd7240f5410 2024-01-21T11:50:15.897 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:15.896+0000 7fd70b7fe700 1 -- 172.21.15.134:0/2654948465 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd7140263a0 con 0x7fd7240f5410 2024-01-21T11:50:15.897 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:15.896+0000 7fd7293b5700 1 -- 172.21.15.134:0/2654948465 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd704004fa0 con 0x7fd7240f5410 2024-01-21T11:50:15.897 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:15.896+0000 7fd70b7fe700 1 --2- 172.21.15.134:0/2654948465 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fd70c02ff60 0x7fd70c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:50:15.897 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:15.896+0000 7fd70b7fe700 1 -- 172.21.15.134:0/2654948465 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd714040fe0 con 0x7fd7240f5410 2024-01-21T11:50:15.897 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:15.897+0000 7fd7227fc700 1 --2- 172.21.15.134:0/2654948465 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fd70c02ff60 0x7fd70c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:50:15.897 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:15.897+0000 7fd7227fc700 1 --2- 172.21.15.134:0/2654948465 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fd70c02ff60 0x7fd70c032420 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7fd7180096a0 tx=0x7fd718006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:50:15.900 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:15.901+0000 7fd70b7fe700 1 -- 172.21.15.134:0/2654948465 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd714012920 con 0x7fd7240f5410 2024-01-21T11:50:15.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:15 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:50:13.881274+0000 mgr.smithi029.blhxxk (mgr.14164) 254 : cluster [DBG] pgmap v240: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:50:16.106 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:16.107+0000 7fd7293b5700 1 -- 172.21.15.134:0/2654948465 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd704005670 con 0x7fd7240f5410 2024-01-21T11:50:16.107 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:16.108+0000 7fd70b7fe700 1 -- 172.21.15.134:0/2654948465 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd714012330 con 0x7fd7240f5410 2024-01-21T11:50:16.107 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:50:16.107 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:50:16.108 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:16.109+0000 7fd7293b5700 1 -- 172.21.15.134:0/2654948465 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fd70c02ff60 msgr2=0x7fd70c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:50:16.109 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:16.109+0000 7fd7293b5700 1 --2- 172.21.15.134:0/2654948465 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fd70c02ff60 0x7fd70c032420 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7fd7180096a0 tx=0x7fd718006b40).stop 2024-01-21T11:50:16.109 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:16.109+0000 7fd7293b5700 1 -- 172.21.15.134:0/2654948465 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7240f5410 msgr2=0x7fd7241055d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:50:16.109 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:16.109+0000 7fd7293b5700 1 --2- 172.21.15.134:0/2654948465 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7240f5410 0x7fd7241055d0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7fd714013510 tx=0x7fd71400b040).stop 2024-01-21T11:50:16.109 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:16.109+0000 7fd7293b5700 1 -- 172.21.15.134:0/2654948465 shutdown_connections 2024-01-21T11:50:16.109 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:16.109+0000 7fd7293b5700 1 --2- 172.21.15.134:0/2654948465 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fd70c02ff60 0x7fd70c032420 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:50:16.109 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:16.109+0000 7fd7293b5700 1 --2- 172.21.15.134:0/2654948465 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7240f5410 0x7fd7241055d0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:50:16.109 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:16.109+0000 7fd7293b5700 1 -- 172.21.15.134:0/2654948465 >> 172.21.15.134:0/2654948465 conn(0x7fd7240f0b70 msgr2=0x7fd7240f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:50:16.109 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:16.109+0000 7fd7293b5700 1 -- 172.21.15.134:0/2654948465 shutdown_connections 2024-01-21T11:50:16.109 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:16.109+0000 7fd7293b5700 1 -- 172.21.15.134:0/2654948465 wait complete. 2024-01-21T11:50:16.110 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:50:16.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:16 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:50:16.107819+0000 mon.smithi029 (mon.0) 310 : audit [DBG] from='client.? 172.21.15.134:0/2654948465' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:50:17.499 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:50:17.499 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:50:17.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:17 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:50:15.881654+0000 mgr.smithi029.blhxxk (mgr.14164) 255 : cluster [DBG] pgmap v241: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:50:18.669 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:18.667+0000 7f6d3bc79700 1 -- 172.21.15.134:0/1401136347 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d340f5410 msgr2=0x7f6d340f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:50:18.670 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:18.667+0000 7f6d3bc79700 1 --2- 172.21.15.134:0/1401136347 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d340f5410 0x7f6d340f5830 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7f6d24009fb0 tx=0x7f6d24009b30).stop 2024-01-21T11:50:18.670 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:18.667+0000 7f6d3bc79700 1 -- 172.21.15.134:0/1401136347 shutdown_connections 2024-01-21T11:50:18.670 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:18.667+0000 7f6d3bc79700 1 --2- 172.21.15.134:0/1401136347 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d340f5410 0x7f6d340f5830 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:50:18.670 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:18.667+0000 7f6d3bc79700 1 -- 172.21.15.134:0/1401136347 >> 172.21.15.134:0/1401136347 conn(0x7f6d340f0b70 msgr2=0x7f6d340f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:50:18.670 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:18.667+0000 7f6d3bc79700 1 -- 172.21.15.134:0/1401136347 shutdown_connections 2024-01-21T11:50:18.670 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:18.668+0000 7f6d3bc79700 1 -- 172.21.15.134:0/1401136347 wait complete. 2024-01-21T11:50:18.670 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:18.668+0000 7f6d3bc79700 1 Processor -- start 2024-01-21T11:50:18.670 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:18.668+0000 7f6d3bc79700 1 -- start start 2024-01-21T11:50:18.670 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:18.668+0000 7f6d3bc79700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d340f5410 0x7f6d34106710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:50:18.670 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:18.668+0000 7f6d3bc79700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d34106c00 con 0x7f6d340f5410 2024-01-21T11:50:18.670 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:18.668+0000 7f6d39a15700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d340f5410 0x7f6d34106710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:50:18.670 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:18.668+0000 7f6d39a15700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d340f5410 0x7f6d34106710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:57410/0 (socket says 172.21.15.134:57410) 2024-01-21T11:50:18.670 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:18.668+0000 7f6d39a15700 1 -- 172.21.15.134:0/3441231605 learned_addr learned my addr 172.21.15.134:0/3441231605 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:50:18.671 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:18.669+0000 7f6d39a15700 1 -- 172.21.15.134:0/3441231605 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d24005040 con 0x7f6d340f5410 2024-01-21T11:50:18.671 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:18.669+0000 7f6d39a15700 1 --2- 172.21.15.134:0/3441231605 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d340f5410 0x7f6d34106710 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7f6d24013510 tx=0x7f6d2400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:50:18.671 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:18.669+0000 7f6d2a7fc700 1 -- 172.21.15.134:0/3441231605 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6d24009320 con 0x7f6d340f5410 2024-01-21T11:50:18.671 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:18.669+0000 7f6d3bc79700 1 -- 172.21.15.134:0/3441231605 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d34106e00 con 0x7f6d340f5410 2024-01-21T11:50:18.671 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:18.669+0000 7f6d2a7fc700 1 -- 172.21.15.134:0/3441231605 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6d24013f00 con 0x7f6d340f5410 2024-01-21T11:50:18.671 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:18.669+0000 7f6d2a7fc700 1 -- 172.21.15.134:0/3441231605 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6d24014d90 con 0x7f6d340f5410 2024-01-21T11:50:18.671 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:18.669+0000 7f6d3bc79700 1 -- 172.21.15.134:0/3441231605 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d34107290 con 0x7f6d340f5410 2024-01-21T11:50:18.672 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:18.670+0000 7f6d2a7fc700 1 -- 172.21.15.134:0/3441231605 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6d240095f0 con 0x7f6d340f5410 2024-01-21T11:50:18.672 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:18.670+0000 7f6d3bc79700 1 -- 172.21.15.134:0/3441231605 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d34045ad0 con 0x7f6d340f5410 2024-01-21T11:50:18.672 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:18.670+0000 7f6d2a7fc700 1 --2- 172.21.15.134:0/3441231605 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f6d2002ff60 0x7f6d20032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:50:18.672 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:18.670+0000 7f6d2a7fc700 1 -- 172.21.15.134:0/3441231605 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6d24041000 con 0x7f6d340f5410 2024-01-21T11:50:18.672 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:18.670+0000 7f6d39214700 1 --2- 172.21.15.134:0/3441231605 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f6d2002ff60 0x7f6d20032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:50:18.672 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:18.671+0000 7f6d39214700 1 --2- 172.21.15.134:0/3441231605 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f6d2002ff60 0x7f6d20032420 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f6d300096a0 tx=0x7f6d30006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:50:18.675 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:18.675+0000 7f6d2a7fc700 1 -- 172.21.15.134:0/3441231605 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6d24014420 con 0x7f6d340f5410 2024-01-21T11:50:18.883 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:18.883+0000 7f6d3bc79700 1 -- 172.21.15.134:0/3441231605 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6d340fae60 con 0x7f6d340f5410 2024-01-21T11:50:18.884 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:18.884+0000 7f6d2a7fc700 1 -- 172.21.15.134:0/3441231605 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6d24010230 con 0x7f6d340f5410 2024-01-21T11:50:18.884 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:50:18.884 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:50:18.886 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:18.885+0000 7f6d3bc79700 1 -- 172.21.15.134:0/3441231605 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f6d2002ff60 msgr2=0x7f6d20032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:50:18.886 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:18.885+0000 7f6d3bc79700 1 --2- 172.21.15.134:0/3441231605 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f6d2002ff60 0x7f6d20032420 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f6d300096a0 tx=0x7f6d30006b40).stop 2024-01-21T11:50:18.886 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:18.885+0000 7f6d3bc79700 1 -- 172.21.15.134:0/3441231605 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d340f5410 msgr2=0x7f6d34106710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:50:18.886 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:18.885+0000 7f6d3bc79700 1 --2- 172.21.15.134:0/3441231605 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d340f5410 0x7f6d34106710 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7f6d24013510 tx=0x7f6d2400b040).stop 2024-01-21T11:50:18.886 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:18.885+0000 7f6d3bc79700 1 -- 172.21.15.134:0/3441231605 shutdown_connections 2024-01-21T11:50:18.886 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:18.885+0000 7f6d3bc79700 1 --2- 172.21.15.134:0/3441231605 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f6d2002ff60 0x7f6d20032420 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:50:18.886 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:18.885+0000 7f6d3bc79700 1 --2- 172.21.15.134:0/3441231605 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d340f5410 0x7f6d34106710 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:50:18.887 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:18.885+0000 7f6d3bc79700 1 -- 172.21.15.134:0/3441231605 >> 172.21.15.134:0/3441231605 conn(0x7f6d340f0b70 msgr2=0x7f6d3405fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:50:18.887 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:18.886+0000 7f6d3bc79700 1 -- 172.21.15.134:0/3441231605 shutdown_connections 2024-01-21T11:50:18.887 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:18.886+0000 7f6d3bc79700 1 -- 172.21.15.134:0/3441231605 wait complete. 2024-01-21T11:50:18.887 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:50:19.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:19 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:50:17.881962+0000 mgr.smithi029.blhxxk (mgr.14164) 256 : cluster [DBG] pgmap v242: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:50:19.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:19 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:50:18.885224+0000 mon.smithi029 (mon.0) 311 : audit [DBG] from='client.? 172.21.15.134:0/3441231605' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:50:20.280 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:50:20.280 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:50:21.463 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:21.462+0000 7f6629afe700 1 -- 172.21.15.134:0/481963332 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6624060220 msgr2=0x7f66240f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:50:21.463 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:21.462+0000 7f6629afe700 1 --2- 172.21.15.134:0/481963332 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6624060220 0x7f66240f9150 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7f6618009fb0 tx=0x7f6618009b30).stop 2024-01-21T11:50:21.463 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:21.462+0000 7f6629afe700 1 -- 172.21.15.134:0/481963332 shutdown_connections 2024-01-21T11:50:21.463 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:21.462+0000 7f6629afe700 1 --2- 172.21.15.134:0/481963332 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6624060220 0x7f66240f9150 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:50:21.463 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:21.462+0000 7f6629afe700 1 -- 172.21.15.134:0/481963332 >> 172.21.15.134:0/481963332 conn(0x7f66240f0b70 msgr2=0x7f66240f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:50:21.464 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:21.462+0000 7f6629afe700 1 -- 172.21.15.134:0/481963332 shutdown_connections 2024-01-21T11:50:21.464 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:21.462+0000 7f6629afe700 1 -- 172.21.15.134:0/481963332 wait complete. 2024-01-21T11:50:21.464 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:21.463+0000 7f6629afe700 1 Processor -- start 2024-01-21T11:50:21.464 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:21.463+0000 7f6629afe700 1 -- start start 2024-01-21T11:50:21.464 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:21.463+0000 7f6629afe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6624060220 0x7f66240f6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:50:21.464 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:21.463+0000 7f6629afe700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66240f5050 con 0x7f6624060220 2024-01-21T11:50:21.466 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:21.463+0000 7f66237fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6624060220 0x7f66240f6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:50:21.466 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:21.463+0000 7f66237fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6624060220 0x7f66240f6bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:57428/0 (socket says 172.21.15.134:57428) 2024-01-21T11:50:21.466 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:21.463+0000 7f66237fe700 1 -- 172.21.15.134:0/442378510 learned_addr learned my addr 172.21.15.134:0/442378510 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:50:21.466 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:21.464+0000 7f66237fe700 1 -- 172.21.15.134:0/442378510 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6618005040 con 0x7f6624060220 2024-01-21T11:50:21.466 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:21.464+0000 7f66237fe700 1 --2- 172.21.15.134:0/442378510 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6624060220 0x7f66240f6bd0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7f6618013510 tx=0x7f661800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:50:21.466 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:21.464+0000 7f660bfff700 1 -- 172.21.15.134:0/442378510 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6618009320 con 0x7f6624060220 2024-01-21T11:50:21.466 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:21.464+0000 7f6629afe700 1 -- 172.21.15.134:0/442378510 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66240f51f0 con 0x7f6624060220 2024-01-21T11:50:21.466 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:21.464+0000 7f660bfff700 1 -- 172.21.15.134:0/442378510 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6618013f00 con 0x7f6624060220 2024-01-21T11:50:21.466 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:21.464+0000 7f660bfff700 1 -- 172.21.15.134:0/442378510 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6618014dc0 con 0x7f6624060220 2024-01-21T11:50:21.466 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:21.464+0000 7f6629afe700 1 -- 172.21.15.134:0/442378510 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66240f5680 con 0x7f6624060220 2024-01-21T11:50:21.466 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:21.465+0000 7f660bfff700 1 -- 172.21.15.134:0/442378510 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f66180095f0 con 0x7f6624060220 2024-01-21T11:50:21.466 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:21.465+0000 7f6629afe700 1 -- 172.21.15.134:0/442378510 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6624059100 con 0x7f6624060220 2024-01-21T11:50:21.467 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:21.465+0000 7f660bfff700 1 --2- 172.21.15.134:0/442378510 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f660c02ff60 0x7f660c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:50:21.467 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:21.465+0000 7f660bfff700 1 -- 172.21.15.134:0/442378510 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6618041000 con 0x7f6624060220 2024-01-21T11:50:21.467 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:21.465+0000 7f6622ffd700 1 --2- 172.21.15.134:0/442378510 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f660c02ff60 0x7f660c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:50:21.467 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:21.466+0000 7f6622ffd700 1 --2- 172.21.15.134:0/442378510 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f660c02ff60 0x7f660c032420 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f66140096a0 tx=0x7f6614006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:50:21.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:21.470+0000 7f660bfff700 1 -- 172.21.15.134:0/442378510 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6618014420 con 0x7f6624060220 2024-01-21T11:50:21.678 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:21.677+0000 7f6629afe700 1 -- 172.21.15.134:0/442378510 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f66240fbf50 con 0x7f6624060220 2024-01-21T11:50:21.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:21.678+0000 7f660bfff700 1 -- 172.21.15.134:0/442378510 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6618010230 con 0x7f6624060220 2024-01-21T11:50:21.680 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:50:21.680 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:50:21.682 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:21.680+0000 7f6629afe700 1 -- 172.21.15.134:0/442378510 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f660c02ff60 msgr2=0x7f660c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:50:21.682 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:21.680+0000 7f6629afe700 1 --2- 172.21.15.134:0/442378510 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f660c02ff60 0x7f660c032420 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f66140096a0 tx=0x7f6614006b40).stop 2024-01-21T11:50:21.682 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:21.680+0000 7f6629afe700 1 -- 172.21.15.134:0/442378510 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6624060220 msgr2=0x7f66240f6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:50:21.682 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:21.680+0000 7f6629afe700 1 --2- 172.21.15.134:0/442378510 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6624060220 0x7f66240f6bd0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7f6618013510 tx=0x7f661800b040).stop 2024-01-21T11:50:21.682 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:21.680+0000 7f6629afe700 1 -- 172.21.15.134:0/442378510 shutdown_connections 2024-01-21T11:50:21.683 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:21.680+0000 7f6629afe700 1 --2- 172.21.15.134:0/442378510 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f660c02ff60 0x7f660c032420 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:50:21.683 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:21.680+0000 7f6629afe700 1 --2- 172.21.15.134:0/442378510 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6624060220 0x7f66240f6bd0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:50:21.683 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:21.680+0000 7f6629afe700 1 -- 172.21.15.134:0/442378510 >> 172.21.15.134:0/442378510 conn(0x7f66240f0b70 msgr2=0x7f66240f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:50:21.683 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:21.680+0000 7f6629afe700 1 -- 172.21.15.134:0/442378510 shutdown_connections 2024-01-21T11:50:21.683 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:21.680+0000 7f6629afe700 1 -- 172.21.15.134:0/442378510 wait complete. 2024-01-21T11:50:21.683 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:50:21.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:21 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:50:19.882317+0000 mgr.smithi029.blhxxk (mgr.14164) 257 : cluster [DBG] pgmap v243: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:50:22.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:22 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:50:21.679629+0000 mon.smithi029 (mon.0) 312 : audit [DBG] from='client.? 172.21.15.134:0/442378510' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:50:23.086 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:50:23.087 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:50:23.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:23 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:50:21.882630+0000 mgr.smithi029.blhxxk (mgr.14164) 258 : cluster [DBG] pgmap v244: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:50:24.183 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:24.181+0000 7ff6360f6700 1 -- 172.21.15.134:0/1739849787 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6300f47c0 msgr2=0x7ff6300f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:50:24.183 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:24.181+0000 7ff6360f6700 1 --2- 172.21.15.134:0/1739849787 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6300f47c0 0x7ff6300f4be0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7ff624009fb0 tx=0x7ff624009b30).stop 2024-01-21T11:50:24.183 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:24.181+0000 7ff6360f6700 1 -- 172.21.15.134:0/1739849787 shutdown_connections 2024-01-21T11:50:24.183 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:24.181+0000 7ff6360f6700 1 --2- 172.21.15.134:0/1739849787 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6300f47c0 0x7ff6300f4be0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:50:24.183 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:24.181+0000 7ff6360f6700 1 -- 172.21.15.134:0/1739849787 >> 172.21.15.134:0/1739849787 conn(0x7ff6300f02e0 msgr2=0x7ff6300f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:50:24.183 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:24.181+0000 7ff6360f6700 1 -- 172.21.15.134:0/1739849787 shutdown_connections 2024-01-21T11:50:24.183 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:24.181+0000 7ff6360f6700 1 -- 172.21.15.134:0/1739849787 wait complete. 2024-01-21T11:50:24.183 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:24.182+0000 7ff6360f6700 1 Processor -- start 2024-01-21T11:50:24.183 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:24.182+0000 7ff6360f6700 1 -- start start 2024-01-21T11:50:24.183 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:24.182+0000 7ff6360f6700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6300f47c0 0x7ff630106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:50:24.183 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:24.182+0000 7ff6360f6700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff630106bf0 con 0x7ff6300f47c0 2024-01-21T11:50:24.184 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:24.182+0000 7ff62f7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6300f47c0 0x7ff630106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:50:24.184 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:24.182+0000 7ff62f7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6300f47c0 0x7ff630106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:57436/0 (socket says 172.21.15.134:57436) 2024-01-21T11:50:24.184 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:24.182+0000 7ff62f7fe700 1 -- 172.21.15.134:0/3361749048 learned_addr learned my addr 172.21.15.134:0/3361749048 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:50:24.184 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:24.183+0000 7ff62f7fe700 1 -- 172.21.15.134:0/3361749048 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff624005040 con 0x7ff6300f47c0 2024-01-21T11:50:24.184 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:24.183+0000 7ff62f7fe700 1 --2- 172.21.15.134:0/3361749048 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6300f47c0 0x7ff630106700 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7ff624013510 tx=0x7ff62400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:50:24.185 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:24.183+0000 7ff617fff700 1 -- 172.21.15.134:0/3361749048 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff624009320 con 0x7ff6300f47c0 2024-01-21T11:50:24.185 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:24.183+0000 7ff6360f6700 1 -- 172.21.15.134:0/3361749048 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff630106df0 con 0x7ff6300f47c0 2024-01-21T11:50:24.185 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:24.183+0000 7ff617fff700 1 -- 172.21.15.134:0/3361749048 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff624013f00 con 0x7ff6300f47c0 2024-01-21T11:50:24.185 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:24.183+0000 7ff617fff700 1 -- 172.21.15.134:0/3361749048 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff624014d90 con 0x7ff6300f47c0 2024-01-21T11:50:24.185 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:24.183+0000 7ff6360f6700 1 -- 172.21.15.134:0/3361749048 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff630107280 con 0x7ff6300f47c0 2024-01-21T11:50:24.185 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:24.184+0000 7ff617fff700 1 -- 172.21.15.134:0/3361749048 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff6240095f0 con 0x7ff6300f47c0 2024-01-21T11:50:24.185 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:24.184+0000 7ff6360f6700 1 -- 172.21.15.134:0/3361749048 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff630045ad0 con 0x7ff6300f47c0 2024-01-21T11:50:24.186 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:24.184+0000 7ff617fff700 1 --2- 172.21.15.134:0/3361749048 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7ff61802ff60 0x7ff618032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:50:24.186 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:24.184+0000 7ff617fff700 1 -- 172.21.15.134:0/3361749048 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff6240410a0 con 0x7ff6300f47c0 2024-01-21T11:50:24.186 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:24.184+0000 7ff62effd700 1 --2- 172.21.15.134:0/3361749048 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7ff61802ff60 0x7ff618032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:50:24.186 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:24.185+0000 7ff62effd700 1 --2- 172.21.15.134:0/3361749048 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7ff61802ff60 0x7ff618032420 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7ff6200096a0 tx=0x7ff620006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:50:24.189 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:24.189+0000 7ff617fff700 1 -- 172.21.15.134:0/3361749048 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff62401d7f0 con 0x7ff6300f47c0 2024-01-21T11:50:24.395 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:24.394+0000 7ff6360f6700 1 -- 172.21.15.134:0/3361749048 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff630108130 con 0x7ff6300f47c0 2024-01-21T11:50:24.395 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:24.395+0000 7ff617fff700 1 -- 172.21.15.134:0/3361749048 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff624010230 con 0x7ff6300f47c0 2024-01-21T11:50:24.395 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:50:24.395 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:50:24.397 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:24.396+0000 7ff6360f6700 1 -- 172.21.15.134:0/3361749048 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7ff61802ff60 msgr2=0x7ff618032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:50:24.397 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:24.396+0000 7ff6360f6700 1 --2- 172.21.15.134:0/3361749048 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7ff61802ff60 0x7ff618032420 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7ff6200096a0 tx=0x7ff620006b40).stop 2024-01-21T11:50:24.397 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:24.396+0000 7ff6360f6700 1 -- 172.21.15.134:0/3361749048 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6300f47c0 msgr2=0x7ff630106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:50:24.397 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:24.396+0000 7ff6360f6700 1 --2- 172.21.15.134:0/3361749048 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6300f47c0 0x7ff630106700 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7ff624013510 tx=0x7ff62400b040).stop 2024-01-21T11:50:24.397 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:24.396+0000 7ff6360f6700 1 -- 172.21.15.134:0/3361749048 shutdown_connections 2024-01-21T11:50:24.397 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:24.396+0000 7ff6360f6700 1 --2- 172.21.15.134:0/3361749048 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7ff61802ff60 0x7ff618032420 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:50:24.398 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:24.396+0000 7ff6360f6700 1 --2- 172.21.15.134:0/3361749048 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6300f47c0 0x7ff630106700 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:50:24.398 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:24.396+0000 7ff6360f6700 1 -- 172.21.15.134:0/3361749048 >> 172.21.15.134:0/3361749048 conn(0x7ff6300f02e0 msgr2=0x7ff6300f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:50:24.398 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:24.396+0000 7ff6360f6700 1 -- 172.21.15.134:0/3361749048 shutdown_connections 2024-01-21T11:50:24.398 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:24.396+0000 7ff6360f6700 1 -- 172.21.15.134:0/3361749048 wait complete. 2024-01-21T11:50:24.398 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:50:24.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:24 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:50:24.396064+0000 mon.smithi029 (mon.0) 313 : audit [DBG] from='client.? 172.21.15.134:0/3361749048' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:50:25.762 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:50:25.763 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:50:25.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:25 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:50:23.883016+0000 mgr.smithi029.blhxxk (mgr.14164) 259 : cluster [DBG] pgmap v245: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:50:26.932 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:26.931+0000 7f3d0a5d1700 1 -- 172.21.15.134:0/4221927351 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d04060280 msgr2=0x7f3d040f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:50:26.933 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:26.931+0000 7f3d0a5d1700 1 --2- 172.21.15.134:0/4221927351 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d04060280 0x7f3d040f90f0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7f3cf4009fb0 tx=0x7f3cf4009b30).stop 2024-01-21T11:50:26.933 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:26.931+0000 7f3d0a5d1700 1 -- 172.21.15.134:0/4221927351 shutdown_connections 2024-01-21T11:50:26.933 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:26.931+0000 7f3d0a5d1700 1 --2- 172.21.15.134:0/4221927351 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d04060280 0x7f3d040f90f0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:50:26.933 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:26.931+0000 7f3d0a5d1700 1 -- 172.21.15.134:0/4221927351 >> 172.21.15.134:0/4221927351 conn(0x7f3d040f0b70 msgr2=0x7f3d040f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:50:26.933 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:26.931+0000 7f3d0a5d1700 1 -- 172.21.15.134:0/4221927351 shutdown_connections 2024-01-21T11:50:26.933 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:26.931+0000 7f3d0a5d1700 1 -- 172.21.15.134:0/4221927351 wait complete. 2024-01-21T11:50:26.933 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:26.932+0000 7f3d0a5d1700 1 Processor -- start 2024-01-21T11:50:26.933 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:26.932+0000 7f3d0a5d1700 1 -- start start 2024-01-21T11:50:26.933 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:26.932+0000 7f3d0a5d1700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d04060280 0x7f3d041066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:50:26.933 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:26.932+0000 7f3d0a5d1700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d04106bd0 con 0x7f3d04060280 2024-01-21T11:50:26.933 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:26.932+0000 7f3d03fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d04060280 0x7f3d041066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:50:26.933 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:26.933+0000 7f3d03fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d04060280 0x7f3d041066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:41666/0 (socket says 172.21.15.134:41666) 2024-01-21T11:50:26.934 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:26.933+0000 7f3d03fff700 1 -- 172.21.15.134:0/4935587 learned_addr learned my addr 172.21.15.134:0/4935587 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:50:26.935 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:26.933+0000 7f3d03fff700 1 -- 172.21.15.134:0/4935587 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cf4005040 con 0x7f3d04060280 2024-01-21T11:50:26.935 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:26.933+0000 7f3d03fff700 1 --2- 172.21.15.134:0/4935587 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d04060280 0x7f3d041066e0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f3cf4013510 tx=0x7f3cf400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:50:26.935 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:26.933+0000 7f3d00ff9700 1 -- 172.21.15.134:0/4935587 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3cf4009320 con 0x7f3d04060280 2024-01-21T11:50:26.935 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:26.933+0000 7f3d0a5d1700 1 -- 172.21.15.134:0/4935587 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d04106dd0 con 0x7f3d04060280 2024-01-21T11:50:26.935 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:26.933+0000 7f3d00ff9700 1 -- 172.21.15.134:0/4935587 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3cf4013f00 con 0x7f3d04060280 2024-01-21T11:50:26.935 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:26.933+0000 7f3d00ff9700 1 -- 172.21.15.134:0/4935587 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3cf4014dc0 con 0x7f3d04060280 2024-01-21T11:50:26.935 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:26.933+0000 7f3d0a5d1700 1 -- 172.21.15.134:0/4935587 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d04107260 con 0x7f3d04060280 2024-01-21T11:50:26.935 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:26.934+0000 7f3d00ff9700 1 -- 172.21.15.134:0/4935587 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3cf40095f0 con 0x7f3d04060280 2024-01-21T11:50:26.935 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:26.934+0000 7f3d0a5d1700 1 -- 172.21.15.134:0/4935587 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d04045ad0 con 0x7f3d04060280 2024-01-21T11:50:26.935 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:26.934+0000 7f3d00ff9700 1 --2- 172.21.15.134:0/4935587 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f3cec02ff60 0x7f3cec032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:50:26.936 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:26.934+0000 7f3d00ff9700 1 -- 172.21.15.134:0/4935587 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3cf4041020 con 0x7f3d04060280 2024-01-21T11:50:26.936 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:26.934+0000 7f3d037fe700 1 --2- 172.21.15.134:0/4935587 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f3cec02ff60 0x7f3cec032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:50:26.936 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:26.935+0000 7f3d037fe700 1 --2- 172.21.15.134:0/4935587 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f3cec02ff60 0x7f3cec032420 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f3cf80096a0 tx=0x7f3cf8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:50:26.939 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:26.939+0000 7f3d00ff9700 1 -- 172.21.15.134:0/4935587 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3cf4014420 con 0x7f3d04060280 2024-01-21T11:50:27.147 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:27.146+0000 7f3d0a5d1700 1 -- 172.21.15.134:0/4935587 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3d041080b0 con 0x7f3d04060280 2024-01-21T11:50:27.147 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:27.147+0000 7f3d00ff9700 1 -- 172.21.15.134:0/4935587 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3cf401c920 con 0x7f3d04060280 2024-01-21T11:50:27.148 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:50:27.148 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:50:27.149 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:27.149+0000 7f3d0a5d1700 1 -- 172.21.15.134:0/4935587 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f3cec02ff60 msgr2=0x7f3cec032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:50:27.149 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:27.149+0000 7f3d0a5d1700 1 --2- 172.21.15.134:0/4935587 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f3cec02ff60 0x7f3cec032420 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f3cf80096a0 tx=0x7f3cf8006b40).stop 2024-01-21T11:50:27.149 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:27.149+0000 7f3d0a5d1700 1 -- 172.21.15.134:0/4935587 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d04060280 msgr2=0x7f3d041066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:50:27.149 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:27.149+0000 7f3d0a5d1700 1 --2- 172.21.15.134:0/4935587 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d04060280 0x7f3d041066e0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f3cf4013510 tx=0x7f3cf400b040).stop 2024-01-21T11:50:27.149 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:27.149+0000 7f3d0a5d1700 1 -- 172.21.15.134:0/4935587 shutdown_connections 2024-01-21T11:50:27.150 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:27.149+0000 7f3d0a5d1700 1 --2- 172.21.15.134:0/4935587 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f3cec02ff60 0x7f3cec032420 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:50:27.150 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:27.149+0000 7f3d0a5d1700 1 --2- 172.21.15.134:0/4935587 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d04060280 0x7f3d041066e0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:50:27.150 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:27.149+0000 7f3d0a5d1700 1 -- 172.21.15.134:0/4935587 >> 172.21.15.134:0/4935587 conn(0x7f3d040f0b70 msgr2=0x7f3d040f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:50:27.150 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:27.149+0000 7f3d0a5d1700 1 -- 172.21.15.134:0/4935587 shutdown_connections 2024-01-21T11:50:27.150 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:27.149+0000 7f3d0a5d1700 1 -- 172.21.15.134:0/4935587 wait complete. 2024-01-21T11:50:27.150 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:50:27.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:27 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:50:25.883359+0000 mgr.smithi029.blhxxk (mgr.14164) 260 : cluster [DBG] pgmap v246: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:50:27.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:27 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:50:27.148469+0000 mon.smithi029 (mon.0) 314 : audit [DBG] from='client.? 172.21.15.134:0/4935587' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:50:28.545 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:50:28.546 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:50:29.686 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:29.681+0000 7fc697958700 1 -- 172.21.15.134:0/1212146072 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6900f5410 msgr2=0x7fc6900f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:50:29.686 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:29.681+0000 7fc697958700 1 --2- 172.21.15.134:0/1212146072 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6900f5410 0x7fc6900f5830 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7fc680009fb0 tx=0x7fc680009b30).stop 2024-01-21T11:50:29.686 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:29.681+0000 7fc697958700 1 -- 172.21.15.134:0/1212146072 shutdown_connections 2024-01-21T11:50:29.686 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:29.681+0000 7fc697958700 1 --2- 172.21.15.134:0/1212146072 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6900f5410 0x7fc6900f5830 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:50:29.686 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:29.681+0000 7fc697958700 1 -- 172.21.15.134:0/1212146072 >> 172.21.15.134:0/1212146072 conn(0x7fc6900f0b70 msgr2=0x7fc6900f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:50:29.686 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:29.681+0000 7fc697958700 1 -- 172.21.15.134:0/1212146072 shutdown_connections 2024-01-21T11:50:29.686 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:29.681+0000 7fc697958700 1 -- 172.21.15.134:0/1212146072 wait complete. 2024-01-21T11:50:29.686 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:29.682+0000 7fc697958700 1 Processor -- start 2024-01-21T11:50:29.686 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:29.682+0000 7fc697958700 1 -- start start 2024-01-21T11:50:29.686 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:29.682+0000 7fc697958700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6900f5410 0x7fc690109400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:50:29.687 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:29.682+0000 7fc697958700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc690060240 con 0x7fc6900f5410 2024-01-21T11:50:29.687 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:29.682+0000 7fc6956f4700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6900f5410 0x7fc690109400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:50:29.687 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:29.682+0000 7fc6956f4700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6900f5410 0x7fc690109400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:41684/0 (socket says 172.21.15.134:41684) 2024-01-21T11:50:29.687 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:29.682+0000 7fc6956f4700 1 -- 172.21.15.134:0/3752452823 learned_addr learned my addr 172.21.15.134:0/3752452823 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:50:29.687 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:29.683+0000 7fc6956f4700 1 -- 172.21.15.134:0/3752452823 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc680005040 con 0x7fc6900f5410 2024-01-21T11:50:29.687 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:29.683+0000 7fc6956f4700 1 --2- 172.21.15.134:0/3752452823 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6900f5410 0x7fc690109400 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7fc680013510 tx=0x7fc68000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:50:29.687 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:29.683+0000 7fc685ffb700 1 -- 172.21.15.134:0/3752452823 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc680009320 con 0x7fc6900f5410 2024-01-21T11:50:29.687 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:29.683+0000 7fc697958700 1 -- 172.21.15.134:0/3752452823 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc690060440 con 0x7fc6900f5410 2024-01-21T11:50:29.687 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:29.683+0000 7fc685ffb700 1 -- 172.21.15.134:0/3752452823 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc680013f00 con 0x7fc6900f5410 2024-01-21T11:50:29.687 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:29.683+0000 7fc685ffb700 1 -- 172.21.15.134:0/3752452823 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc680014d90 con 0x7fc6900f5410 2024-01-21T11:50:29.687 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:29.683+0000 7fc697958700 1 -- 172.21.15.134:0/3752452823 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc690109ab0 con 0x7fc6900f5410 2024-01-21T11:50:29.687 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:29.684+0000 7fc697958700 1 -- 172.21.15.134:0/3752452823 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc690045ad0 con 0x7fc6900f5410 2024-01-21T11:50:29.687 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:29.684+0000 7fc685ffb700 1 -- 172.21.15.134:0/3752452823 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc6800095f0 con 0x7fc6900f5410 2024-01-21T11:50:29.687 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:29.684+0000 7fc685ffb700 1 --2- 172.21.15.134:0/3752452823 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fc67c02ff60 0x7fc67c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:50:29.687 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:29.684+0000 7fc685ffb700 1 -- 172.21.15.134:0/3752452823 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc680041000 con 0x7fc6900f5410 2024-01-21T11:50:29.688 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:29.685+0000 7fc694ef3700 1 --2- 172.21.15.134:0/3752452823 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fc67c02ff60 0x7fc67c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:50:29.688 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:29.685+0000 7fc694ef3700 1 --2- 172.21.15.134:0/3752452823 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fc67c02ff60 0x7fc67c032420 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7fc68c0096a0 tx=0x7fc68c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:50:29.689 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:29.689+0000 7fc685ffb700 1 -- 172.21.15.134:0/3752452823 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc68001d7f0 con 0x7fc6900f5410 2024-01-21T11:50:29.895 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:29.895+0000 7fc697958700 1 -- 172.21.15.134:0/3752452823 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc6900fae60 con 0x7fc6900f5410 2024-01-21T11:50:29.896 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:29.895+0000 7fc685ffb700 1 -- 172.21.15.134:0/3752452823 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc680010230 con 0x7fc6900f5410 2024-01-21T11:50:29.896 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:50:29.896 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:50:29.897 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:29.897+0000 7fc697958700 1 -- 172.21.15.134:0/3752452823 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fc67c02ff60 msgr2=0x7fc67c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:50:29.897 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:29.897+0000 7fc697958700 1 --2- 172.21.15.134:0/3752452823 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fc67c02ff60 0x7fc67c032420 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7fc68c0096a0 tx=0x7fc68c006b40).stop 2024-01-21T11:50:29.897 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:29.897+0000 7fc697958700 1 -- 172.21.15.134:0/3752452823 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6900f5410 msgr2=0x7fc690109400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:50:29.897 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:29.897+0000 7fc697958700 1 --2- 172.21.15.134:0/3752452823 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6900f5410 0x7fc690109400 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7fc680013510 tx=0x7fc68000b040).stop 2024-01-21T11:50:29.897 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:29.897+0000 7fc697958700 1 -- 172.21.15.134:0/3752452823 shutdown_connections 2024-01-21T11:50:29.897 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:29.897+0000 7fc697958700 1 --2- 172.21.15.134:0/3752452823 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fc67c02ff60 0x7fc67c032420 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:50:29.898 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:29.897+0000 7fc697958700 1 --2- 172.21.15.134:0/3752452823 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6900f5410 0x7fc690109400 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:50:29.898 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:29.897+0000 7fc697958700 1 -- 172.21.15.134:0/3752452823 >> 172.21.15.134:0/3752452823 conn(0x7fc6900f0b70 msgr2=0x7fc6900f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:50:29.898 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:29.897+0000 7fc697958700 1 -- 172.21.15.134:0/3752452823 shutdown_connections 2024-01-21T11:50:29.898 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:29.897+0000 7fc697958700 1 -- 172.21.15.134:0/3752452823 wait complete. 2024-01-21T11:50:29.898 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:50:29.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:29 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:50:27.883683+0000 mgr.smithi029.blhxxk (mgr.14164) 261 : cluster [DBG] pgmap v247: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:50:30.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:30 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:50:29.896559+0000 mon.smithi029 (mon.0) 315 : audit [DBG] from='client.? 172.21.15.134:0/3752452823' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:50:31.274 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:50:31.274 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:50:31.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:31 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:50:29.884056+0000 mgr.smithi029.blhxxk (mgr.14164) 262 : cluster [DBG] pgmap v248: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:50:32.491 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:32.489+0000 7fa9fb447700 1 -- 172.21.15.134:0/146430606 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9f40f5950 msgr2=0x7fa9f40f5d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:50:32.491 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:32.489+0000 7fa9fb447700 1 --2- 172.21.15.134:0/146430606 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9f40f5950 0x7fa9f40f5d70 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0x7fa9e4009fb0 tx=0x7fa9e4009b30).stop 2024-01-21T11:50:32.491 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:32.489+0000 7fa9fb447700 1 -- 172.21.15.134:0/146430606 shutdown_connections 2024-01-21T11:50:32.492 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:32.489+0000 7fa9fb447700 1 --2- 172.21.15.134:0/146430606 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9f40f5950 0x7fa9f40f5d70 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:50:32.492 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:32.489+0000 7fa9fb447700 1 -- 172.21.15.134:0/146430606 >> 172.21.15.134:0/146430606 conn(0x7fa9f40f0260 msgr2=0x7fa9f40f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:50:32.492 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:32.489+0000 7fa9fb447700 1 -- 172.21.15.134:0/146430606 shutdown_connections 2024-01-21T11:50:32.492 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:32.489+0000 7fa9fb447700 1 -- 172.21.15.134:0/146430606 wait complete. 2024-01-21T11:50:32.492 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:32.490+0000 7fa9fb447700 1 Processor -- start 2024-01-21T11:50:32.492 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:32.490+0000 7fa9fb447700 1 -- start start 2024-01-21T11:50:32.492 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:32.490+0000 7fa9fb447700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9f40f5950 0x7fa9f4106740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:50:32.492 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:32.490+0000 7fa9fb447700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9f4106c30 con 0x7fa9f40f5950 2024-01-21T11:50:32.492 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:32.490+0000 7fa9f91e3700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9f40f5950 0x7fa9f4106740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:50:32.492 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:32.490+0000 7fa9f91e3700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9f40f5950 0x7fa9f4106740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:41704/0 (socket says 172.21.15.134:41704) 2024-01-21T11:50:32.492 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:32.490+0000 7fa9f91e3700 1 -- 172.21.15.134:0/503813387 learned_addr learned my addr 172.21.15.134:0/503813387 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:50:32.493 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:32.491+0000 7fa9f91e3700 1 -- 172.21.15.134:0/503813387 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9e4005040 con 0x7fa9f40f5950 2024-01-21T11:50:32.493 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:32.491+0000 7fa9f91e3700 1 --2- 172.21.15.134:0/503813387 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9f40f5950 0x7fa9f4106740 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7fa9e4013510 tx=0x7fa9e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:50:32.493 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:32.491+0000 7fa9e9ffb700 1 -- 172.21.15.134:0/503813387 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa9e4009320 con 0x7fa9f40f5950 2024-01-21T11:50:32.493 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:32.491+0000 7fa9fb447700 1 -- 172.21.15.134:0/503813387 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9f4106e30 con 0x7fa9f40f5950 2024-01-21T11:50:32.493 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:32.491+0000 7fa9e9ffb700 1 -- 172.21.15.134:0/503813387 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa9e4013f00 con 0x7fa9f40f5950 2024-01-21T11:50:32.493 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:32.491+0000 7fa9e9ffb700 1 -- 172.21.15.134:0/503813387 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa9e4014d90 con 0x7fa9f40f5950 2024-01-21T11:50:32.493 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:32.491+0000 7fa9fb447700 1 -- 172.21.15.134:0/503813387 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9f41072c0 con 0x7fa9f40f5950 2024-01-21T11:50:32.494 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:32.492+0000 7fa9e9ffb700 1 -- 172.21.15.134:0/503813387 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa9e4014420 con 0x7fa9f40f5950 2024-01-21T11:50:32.494 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:32.492+0000 7fa9fb447700 1 -- 172.21.15.134:0/503813387 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9f4045ad0 con 0x7fa9f40f5950 2024-01-21T11:50:32.494 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:32.492+0000 7fa9e9ffb700 1 --2- 172.21.15.134:0/503813387 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fa9e002ff60 0x7fa9e0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:50:32.494 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:32.492+0000 7fa9e9ffb700 1 -- 172.21.15.134:0/503813387 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa9e4041110 con 0x7fa9f40f5950 2024-01-21T11:50:32.494 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:32.493+0000 7fa9f89e2700 1 --2- 172.21.15.134:0/503813387 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fa9e002ff60 0x7fa9e0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:50:32.494 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:32.493+0000 7fa9f89e2700 1 --2- 172.21.15.134:0/503813387 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fa9e002ff60 0x7fa9e0032420 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7fa9f00096a0 tx=0x7fa9f0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:50:32.497 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:32.497+0000 7fa9e9ffb700 1 -- 172.21.15.134:0/503813387 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa9e40095f0 con 0x7fa9f40f5950 2024-01-21T11:50:32.700 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:32.700+0000 7fa9fb447700 1 -- 172.21.15.134:0/503813387 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa9f4108110 con 0x7fa9f40f5950 2024-01-21T11:50:32.701 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:32.700+0000 7fa9e9ffb700 1 -- 172.21.15.134:0/503813387 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa9e401db70 con 0x7fa9f40f5950 2024-01-21T11:50:32.701 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:50:32.701 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:50:32.702 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:32.702+0000 7fa9fb447700 1 -- 172.21.15.134:0/503813387 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fa9e002ff60 msgr2=0x7fa9e0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:50:32.703 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:32.702+0000 7fa9fb447700 1 --2- 172.21.15.134:0/503813387 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fa9e002ff60 0x7fa9e0032420 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7fa9f00096a0 tx=0x7fa9f0006b40).stop 2024-01-21T11:50:32.703 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:32.702+0000 7fa9fb447700 1 -- 172.21.15.134:0/503813387 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9f40f5950 msgr2=0x7fa9f4106740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:50:32.703 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:32.702+0000 7fa9fb447700 1 --2- 172.21.15.134:0/503813387 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9f40f5950 0x7fa9f4106740 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7fa9e4013510 tx=0x7fa9e400b040).stop 2024-01-21T11:50:32.703 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:32.702+0000 7fa9fb447700 1 -- 172.21.15.134:0/503813387 shutdown_connections 2024-01-21T11:50:32.703 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:32.702+0000 7fa9fb447700 1 --2- 172.21.15.134:0/503813387 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fa9e002ff60 0x7fa9e0032420 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:50:32.703 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:32.702+0000 7fa9fb447700 1 --2- 172.21.15.134:0/503813387 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9f40f5950 0x7fa9f4106740 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:50:32.703 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:32.702+0000 7fa9fb447700 1 -- 172.21.15.134:0/503813387 >> 172.21.15.134:0/503813387 conn(0x7fa9f40f0260 msgr2=0x7fa9f40f0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:50:32.703 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:32.702+0000 7fa9fb447700 1 -- 172.21.15.134:0/503813387 shutdown_connections 2024-01-21T11:50:32.703 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:32.702+0000 7fa9fb447700 1 -- 172.21.15.134:0/503813387 wait complete. 2024-01-21T11:50:32.703 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:50:33.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:33 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:50:31.884440+0000 mgr.smithi029.blhxxk (mgr.14164) 263 : cluster [DBG] pgmap v249: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:50:33.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:33 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:50:32.701776+0000 mon.smithi029 (mon.0) 316 : audit [DBG] from='client.? 172.21.15.134:0/503813387' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:50:34.076 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:50:34.076 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:50:35.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:35.322+0000 7f91577f9700 1 -- 172.21.15.134:0/1122863505 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91500f6510 msgr2=0x7f91500f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:50:35.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:35.322+0000 7f91577f9700 1 --2- 172.21.15.134:0/1122863505 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91500f6510 0x7f91500f6930 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7f9140009fb0 tx=0x7f9140009b30).stop 2024-01-21T11:50:35.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:35.322+0000 7f91577f9700 1 -- 172.21.15.134:0/1122863505 shutdown_connections 2024-01-21T11:50:35.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:35.322+0000 7f91577f9700 1 --2- 172.21.15.134:0/1122863505 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91500f6510 0x7f91500f6930 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:50:35.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:35.322+0000 7f91577f9700 1 -- 172.21.15.134:0/1122863505 >> 172.21.15.134:0/1122863505 conn(0x7f91500f1c70 msgr2=0x7f91500f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:50:35.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:35.323+0000 7f91577f9700 1 -- 172.21.15.134:0/1122863505 shutdown_connections 2024-01-21T11:50:35.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:35.323+0000 7f91577f9700 1 -- 172.21.15.134:0/1122863505 wait complete. 2024-01-21T11:50:35.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:35.323+0000 7f91577f9700 1 Processor -- start 2024-01-21T11:50:35.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:35.323+0000 7f91577f9700 1 -- start start 2024-01-21T11:50:35.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:35.323+0000 7f91577f9700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91500f6510 0x7f915006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:50:35.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:35.323+0000 7f91577f9700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9150069e20 con 0x7f91500f6510 2024-01-21T11:50:35.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:35.323+0000 7f9155595700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91500f6510 0x7f915006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:50:35.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:35.324+0000 7f9155595700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91500f6510 0x7f915006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:33310/0 (socket says 172.21.15.134:33310) 2024-01-21T11:50:35.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:35.324+0000 7f9155595700 1 -- 172.21.15.134:0/2024942584 learned_addr learned my addr 172.21.15.134:0/2024942584 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:50:35.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:35.324+0000 7f9155595700 1 -- 172.21.15.134:0/2024942584 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9140005040 con 0x7f91500f6510 2024-01-21T11:50:35.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:35.324+0000 7f9155595700 1 --2- 172.21.15.134:0/2024942584 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91500f6510 0x7f915006b9a0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7f9140013510 tx=0x7f914000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:50:35.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:35.324+0000 7f9145ffb700 1 -- 172.21.15.134:0/2024942584 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9140009320 con 0x7f91500f6510 2024-01-21T11:50:35.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:35.324+0000 7f91577f9700 1 -- 172.21.15.134:0/2024942584 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9150069fc0 con 0x7f91500f6510 2024-01-21T11:50:35.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:35.324+0000 7f9145ffb700 1 -- 172.21.15.134:0/2024942584 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9140013f00 con 0x7f91500f6510 2024-01-21T11:50:35.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:35.324+0000 7f9145ffb700 1 -- 172.21.15.134:0/2024942584 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9140014d90 con 0x7f91500f6510 2024-01-21T11:50:35.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:35.324+0000 7f91577f9700 1 -- 172.21.15.134:0/2024942584 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f915006a450 con 0x7f91500f6510 2024-01-21T11:50:35.329 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:35.325+0000 7f91577f9700 1 -- 172.21.15.134:0/2024942584 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9150059100 con 0x7f91500f6510 2024-01-21T11:50:35.329 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:35.325+0000 7f9145ffb700 1 -- 172.21.15.134:0/2024942584 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f91400095f0 con 0x7f91500f6510 2024-01-21T11:50:35.330 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:35.325+0000 7f9145ffb700 1 --2- 172.21.15.134:0/2024942584 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f913c02ff60 0x7f913c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:50:35.330 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:35.326+0000 7f9145ffb700 1 -- 172.21.15.134:0/2024942584 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9140040fc0 con 0x7f91500f6510 2024-01-21T11:50:35.330 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:35.326+0000 7f9154d94700 1 --2- 172.21.15.134:0/2024942584 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f913c02ff60 0x7f913c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:50:35.330 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:35.326+0000 7f9154d94700 1 --2- 172.21.15.134:0/2024942584 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f913c02ff60 0x7f913c032420 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f914c0096a0 tx=0x7f914c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:50:35.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:35.331+0000 7f9145ffb700 1 -- 172.21.15.134:0/2024942584 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9140014420 con 0x7f91500f6510 2024-01-21T11:50:35.535 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:35.535+0000 7f91577f9700 1 -- 172.21.15.134:0/2024942584 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9150045ad0 con 0x7f91500f6510 2024-01-21T11:50:35.536 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:35.536+0000 7f9145ffb700 1 -- 172.21.15.134:0/2024942584 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9140010230 con 0x7f91500f6510 2024-01-21T11:50:35.536 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:50:35.536 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:50:35.538 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:35.537+0000 7f91577f9700 1 -- 172.21.15.134:0/2024942584 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f913c02ff60 msgr2=0x7f913c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:50:35.538 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:35.537+0000 7f91577f9700 1 --2- 172.21.15.134:0/2024942584 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f913c02ff60 0x7f913c032420 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f914c0096a0 tx=0x7f914c006b40).stop 2024-01-21T11:50:35.538 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:35.537+0000 7f91577f9700 1 -- 172.21.15.134:0/2024942584 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91500f6510 msgr2=0x7f915006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:50:35.538 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:35.537+0000 7f91577f9700 1 --2- 172.21.15.134:0/2024942584 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91500f6510 0x7f915006b9a0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7f9140013510 tx=0x7f914000b040).stop 2024-01-21T11:50:35.538 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:35.537+0000 7f91577f9700 1 -- 172.21.15.134:0/2024942584 shutdown_connections 2024-01-21T11:50:35.538 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:35.537+0000 7f91577f9700 1 --2- 172.21.15.134:0/2024942584 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f913c02ff60 0x7f913c032420 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:50:35.538 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:35.537+0000 7f91577f9700 1 --2- 172.21.15.134:0/2024942584 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91500f6510 0x7f915006b9a0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:50:35.538 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:35.537+0000 7f91577f9700 1 -- 172.21.15.134:0/2024942584 >> 172.21.15.134:0/2024942584 conn(0x7f91500f1c70 msgr2=0x7f91500f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:50:35.538 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:35.537+0000 7f91577f9700 1 -- 172.21.15.134:0/2024942584 shutdown_connections 2024-01-21T11:50:35.538 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:35.537+0000 7f91577f9700 1 -- 172.21.15.134:0/2024942584 wait complete. 2024-01-21T11:50:35.539 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:50:35.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:35 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:50:33.884806+0000 mgr.smithi029.blhxxk (mgr.14164) 264 : cluster [DBG] pgmap v250: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:50:35.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:35 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:50:35.536887+0000 mon.smithi029 (mon.0) 317 : audit [DBG] from='client.? 172.21.15.134:0/2024942584' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:50:36.921 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:50:36.921 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:50:37.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:37 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:50:35.885162+0000 mgr.smithi029.blhxxk (mgr.14164) 265 : cluster [DBG] pgmap v251: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:50:38.156 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:38.156+0000 7f402974e700 1 -- 172.21.15.134:0/2340686295 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40240f51a0 msgr2=0x7f40240f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:50:38.156 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:38.156+0000 7f402974e700 1 --2- 172.21.15.134:0/2340686295 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40240f51a0 0x7f40240f55c0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7f4014009fb0 tx=0x7f4014009b30).stop 2024-01-21T11:50:38.156 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:38.156+0000 7f402974e700 1 -- 172.21.15.134:0/2340686295 shutdown_connections 2024-01-21T11:50:38.156 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:38.156+0000 7f402974e700 1 --2- 172.21.15.134:0/2340686295 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40240f51a0 0x7f40240f55c0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:50:38.157 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:38.156+0000 7f402974e700 1 -- 172.21.15.134:0/2340686295 >> 172.21.15.134:0/2340686295 conn(0x7f40240f09a0 msgr2=0x7f40240f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:50:38.157 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:38.156+0000 7f402974e700 1 -- 172.21.15.134:0/2340686295 shutdown_connections 2024-01-21T11:50:38.157 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:38.156+0000 7f402974e700 1 -- 172.21.15.134:0/2340686295 wait complete. 2024-01-21T11:50:38.157 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:38.156+0000 7f402974e700 1 Processor -- start 2024-01-21T11:50:38.159 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:38.156+0000 7f402974e700 1 -- start start 2024-01-21T11:50:38.159 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:38.156+0000 7f402974e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40240f51a0 0x7f40241064a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:50:38.159 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:38.156+0000 7f402974e700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4024106990 con 0x7f40240f51a0 2024-01-21T11:50:38.159 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:38.157+0000 7f4023fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40240f51a0 0x7f40241064a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:50:38.159 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:38.157+0000 7f4023fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40240f51a0 0x7f40241064a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:33332/0 (socket says 172.21.15.134:33332) 2024-01-21T11:50:38.159 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:38.157+0000 7f4023fff700 1 -- 172.21.15.134:0/1673566332 learned_addr learned my addr 172.21.15.134:0/1673566332 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:50:38.159 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:38.157+0000 7f4023fff700 1 -- 172.21.15.134:0/1673566332 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4014005040 con 0x7f40240f51a0 2024-01-21T11:50:38.160 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:38.157+0000 7f4023fff700 1 --2- 172.21.15.134:0/1673566332 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40240f51a0 0x7f40241064a0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f4014013510 tx=0x7f401400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:50:38.160 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:38.157+0000 7f4020ff9700 1 -- 172.21.15.134:0/1673566332 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4014009320 con 0x7f40240f51a0 2024-01-21T11:50:38.160 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:38.157+0000 7f4020ff9700 1 -- 172.21.15.134:0/1673566332 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4014013f00 con 0x7f40240f51a0 2024-01-21T11:50:38.160 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:38.157+0000 7f402974e700 1 -- 172.21.15.134:0/1673566332 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4024106b90 con 0x7f40240f51a0 2024-01-21T11:50:38.160 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:38.157+0000 7f4020ff9700 1 -- 172.21.15.134:0/1673566332 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4014014dc0 con 0x7f40240f51a0 2024-01-21T11:50:38.160 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:38.157+0000 7f402974e700 1 -- 172.21.15.134:0/1673566332 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4024107020 con 0x7f40240f51a0 2024-01-21T11:50:38.160 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:38.158+0000 7f4020ff9700 1 -- 172.21.15.134:0/1673566332 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f40140095f0 con 0x7f40240f51a0 2024-01-21T11:50:38.160 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:38.158+0000 7f402974e700 1 -- 172.21.15.134:0/1673566332 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4024045ad0 con 0x7f40240f51a0 2024-01-21T11:50:38.160 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:38.158+0000 7f4020ff9700 1 --2- 172.21.15.134:0/1673566332 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f400c02ff60 0x7f400c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:50:38.160 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:38.158+0000 7f4020ff9700 1 -- 172.21.15.134:0/1673566332 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f40140410c0 con 0x7f40240f51a0 2024-01-21T11:50:38.160 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:38.159+0000 7f40237fe700 1 --2- 172.21.15.134:0/1673566332 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f400c02ff60 0x7f400c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:50:38.160 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:38.159+0000 7f40237fe700 1 --2- 172.21.15.134:0/1673566332 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f400c02ff60 0x7f400c032420 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f40180096a0 tx=0x7f4018006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:50:38.163 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:38.163+0000 7f4020ff9700 1 -- 172.21.15.134:0/1673566332 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f401401c790 con 0x7f40240f51a0 2024-01-21T11:50:38.368 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:38.368+0000 7f402974e700 1 -- 172.21.15.134:0/1673566332 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4024107ed0 con 0x7f40240f51a0 2024-01-21T11:50:38.369 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:38.369+0000 7f4020ff9700 1 -- 172.21.15.134:0/1673566332 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4014010230 con 0x7f40240f51a0 2024-01-21T11:50:38.369 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:50:38.369 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:50:38.371 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:38.370+0000 7f402974e700 1 -- 172.21.15.134:0/1673566332 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f400c02ff60 msgr2=0x7f400c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:50:38.371 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:38.370+0000 7f402974e700 1 --2- 172.21.15.134:0/1673566332 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f400c02ff60 0x7f400c032420 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f40180096a0 tx=0x7f4018006b40).stop 2024-01-21T11:50:38.371 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:38.370+0000 7f402974e700 1 -- 172.21.15.134:0/1673566332 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40240f51a0 msgr2=0x7f40241064a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:50:38.371 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:38.370+0000 7f402974e700 1 --2- 172.21.15.134:0/1673566332 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40240f51a0 0x7f40241064a0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f4014013510 tx=0x7f401400b040).stop 2024-01-21T11:50:38.371 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:38.370+0000 7f402974e700 1 -- 172.21.15.134:0/1673566332 shutdown_connections 2024-01-21T11:50:38.371 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:38.370+0000 7f402974e700 1 --2- 172.21.15.134:0/1673566332 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f400c02ff60 0x7f400c032420 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:50:38.371 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:38.370+0000 7f402974e700 1 --2- 172.21.15.134:0/1673566332 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40240f51a0 0x7f40241064a0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:50:38.371 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:38.370+0000 7f402974e700 1 -- 172.21.15.134:0/1673566332 >> 172.21.15.134:0/1673566332 conn(0x7f40240f09a0 msgr2=0x7f402405fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:50:38.371 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:38.370+0000 7f402974e700 1 -- 172.21.15.134:0/1673566332 shutdown_connections 2024-01-21T11:50:38.371 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:38.370+0000 7f402974e700 1 -- 172.21.15.134:0/1673566332 wait complete. 2024-01-21T11:50:38.372 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:50:38.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:38 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:50:38.369778+0000 mon.smithi029 (mon.0) 318 : audit [DBG] from='client.? 172.21.15.134:0/1673566332' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:50:39.743 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:50:39.743 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:50:39.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:39 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:50:37.885472+0000 mgr.smithi029.blhxxk (mgr.14164) 266 : cluster [DBG] pgmap v252: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:50:40.880 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:40.879+0000 7ff52598e700 1 -- 172.21.15.134:0/3741209275 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5200f4740 msgr2=0x7ff5200f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:50:40.881 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:40.879+0000 7ff52598e700 1 --2- 172.21.15.134:0/3741209275 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5200f4740 0x7ff5200f4b60 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0x7ff514009fb0 tx=0x7ff514009b30).stop 2024-01-21T11:50:40.881 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:40.879+0000 7ff52598e700 1 -- 172.21.15.134:0/3741209275 shutdown_connections 2024-01-21T11:50:40.881 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:40.879+0000 7ff52598e700 1 --2- 172.21.15.134:0/3741209275 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5200f4740 0x7ff5200f4b60 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:50:40.881 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:40.879+0000 7ff52598e700 1 -- 172.21.15.134:0/3741209275 >> 172.21.15.134:0/3741209275 conn(0x7ff5200f0260 msgr2=0x7ff5200f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:50:40.881 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:40.879+0000 7ff52598e700 1 -- 172.21.15.134:0/3741209275 shutdown_connections 2024-01-21T11:50:40.881 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:40.879+0000 7ff52598e700 1 -- 172.21.15.134:0/3741209275 wait complete. 2024-01-21T11:50:40.881 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:40.880+0000 7ff52598e700 1 Processor -- start 2024-01-21T11:50:40.881 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:40.880+0000 7ff52598e700 1 -- start start 2024-01-21T11:50:40.881 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:40.880+0000 7ff52598e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5200f4740 0x7ff5201088f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:50:40.881 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:40.880+0000 7ff52598e700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff520108de0 con 0x7ff5200f4740 2024-01-21T11:50:40.881 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:40.880+0000 7ff51effd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5200f4740 0x7ff5201088f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:50:40.882 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:40.880+0000 7ff51effd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5200f4740 0x7ff5201088f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:33342/0 (socket says 172.21.15.134:33342) 2024-01-21T11:50:40.882 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:40.880+0000 7ff51effd700 1 -- 172.21.15.134:0/1268258954 learned_addr learned my addr 172.21.15.134:0/1268258954 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:50:40.882 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:40.881+0000 7ff51effd700 1 -- 172.21.15.134:0/1268258954 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff514005040 con 0x7ff5200f4740 2024-01-21T11:50:40.883 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:40.881+0000 7ff51effd700 1 --2- 172.21.15.134:0/1268258954 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5200f4740 0x7ff5201088f0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7ff514013510 tx=0x7ff51400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:50:40.883 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:40.881+0000 7ff5077fe700 1 -- 172.21.15.134:0/1268258954 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff514009320 con 0x7ff5200f4740 2024-01-21T11:50:40.883 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:40.881+0000 7ff5077fe700 1 -- 172.21.15.134:0/1268258954 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff514013f00 con 0x7ff5200f4740 2024-01-21T11:50:40.884 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:40.881+0000 7ff52598e700 1 -- 172.21.15.134:0/1268258954 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff520108fe0 con 0x7ff5200f4740 2024-01-21T11:50:40.884 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:40.881+0000 7ff5077fe700 1 -- 172.21.15.134:0/1268258954 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff514014d90 con 0x7ff5200f4740 2024-01-21T11:50:40.884 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:40.881+0000 7ff52598e700 1 -- 172.21.15.134:0/1268258954 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff520109470 con 0x7ff5200f4740 2024-01-21T11:50:40.884 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:40.882+0000 7ff5077fe700 1 -- 172.21.15.134:0/1268258954 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff5140095f0 con 0x7ff5200f4740 2024-01-21T11:50:40.884 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:40.882+0000 7ff52598e700 1 -- 172.21.15.134:0/1268258954 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff520045ad0 con 0x7ff5200f4740 2024-01-21T11:50:40.884 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:40.882+0000 7ff5077fe700 1 --2- 172.21.15.134:0/1268258954 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7ff50802ff60 0x7ff508032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:50:40.884 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:40.882+0000 7ff5077fe700 1 -- 172.21.15.134:0/1268258954 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff5140410a0 con 0x7ff5200f4740 2024-01-21T11:50:40.884 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:40.882+0000 7ff51e7fc700 1 --2- 172.21.15.134:0/1268258954 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7ff50802ff60 0x7ff508032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:50:40.884 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:40.883+0000 7ff51e7fc700 1 --2- 172.21.15.134:0/1268258954 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7ff50802ff60 0x7ff508032420 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7ff5100096a0 tx=0x7ff510006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:50:40.887 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:40.887+0000 7ff5077fe700 1 -- 172.21.15.134:0/1268258954 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff51401d7f0 con 0x7ff5200f4740 2024-01-21T11:50:41.094 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:41.093+0000 7ff52598e700 1 -- 172.21.15.134:0/1268258954 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff52010a320 con 0x7ff5200f4740 2024-01-21T11:50:41.094 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:41.094+0000 7ff5077fe700 1 -- 172.21.15.134:0/1268258954 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff514010230 con 0x7ff5200f4740 2024-01-21T11:50:41.094 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:50:41.094 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:50:41.096 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:41.095+0000 7ff52598e700 1 -- 172.21.15.134:0/1268258954 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7ff50802ff60 msgr2=0x7ff508032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:50:41.096 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:41.095+0000 7ff52598e700 1 --2- 172.21.15.134:0/1268258954 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7ff50802ff60 0x7ff508032420 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7ff5100096a0 tx=0x7ff510006b40).stop 2024-01-21T11:50:41.096 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:41.095+0000 7ff52598e700 1 -- 172.21.15.134:0/1268258954 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5200f4740 msgr2=0x7ff5201088f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:50:41.096 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:41.095+0000 7ff52598e700 1 --2- 172.21.15.134:0/1268258954 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5200f4740 0x7ff5201088f0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7ff514013510 tx=0x7ff51400b040).stop 2024-01-21T11:50:41.096 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:41.095+0000 7ff52598e700 1 -- 172.21.15.134:0/1268258954 shutdown_connections 2024-01-21T11:50:41.096 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:41.096+0000 7ff52598e700 1 --2- 172.21.15.134:0/1268258954 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7ff50802ff60 0x7ff508032420 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:50:41.096 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:41.096+0000 7ff52598e700 1 --2- 172.21.15.134:0/1268258954 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5200f4740 0x7ff5201088f0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:50:41.096 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:41.096+0000 7ff52598e700 1 -- 172.21.15.134:0/1268258954 >> 172.21.15.134:0/1268258954 conn(0x7ff5200f0260 msgr2=0x7ff5200f0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:50:41.096 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:41.096+0000 7ff52598e700 1 -- 172.21.15.134:0/1268258954 shutdown_connections 2024-01-21T11:50:41.097 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:41.096+0000 7ff52598e700 1 -- 172.21.15.134:0/1268258954 wait complete. 2024-01-21T11:50:41.097 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:50:41.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:41 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:50:39.885818+0000 mgr.smithi029.blhxxk (mgr.14164) 267 : cluster [DBG] pgmap v253: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:50:41.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:41 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:50:41.095343+0000 mon.smithi029 (mon.0) 319 : audit [DBG] from='client.? 172.21.15.134:0/1268258954' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:50:42.474 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:50:42.474 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:50:43.614 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:43.613+0000 7f8aaaef8700 1 -- 172.21.15.134:0/76350129 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8aa4060220 msgr2=0x7f8aa40f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:50:43.614 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:43.613+0000 7f8aaaef8700 1 --2- 172.21.15.134:0/76350129 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8aa4060220 0x7f8aa40f9150 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7f8a98009fb0 tx=0x7f8a98009b30).stop 2024-01-21T11:50:43.614 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:43.613+0000 7f8aaaef8700 1 -- 172.21.15.134:0/76350129 shutdown_connections 2024-01-21T11:50:43.614 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:43.613+0000 7f8aaaef8700 1 --2- 172.21.15.134:0/76350129 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8aa4060220 0x7f8aa40f9150 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:50:43.614 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:43.613+0000 7f8aaaef8700 1 -- 172.21.15.134:0/76350129 >> 172.21.15.134:0/76350129 conn(0x7f8aa40f0b70 msgr2=0x7f8aa40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:50:43.614 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:43.613+0000 7f8aaaef8700 1 -- 172.21.15.134:0/76350129 shutdown_connections 2024-01-21T11:50:43.615 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:43.613+0000 7f8aaaef8700 1 -- 172.21.15.134:0/76350129 wait complete. 2024-01-21T11:50:43.615 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:43.613+0000 7f8aaaef8700 1 Processor -- start 2024-01-21T11:50:43.615 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:43.614+0000 7f8aaaef8700 1 -- start start 2024-01-21T11:50:43.615 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:43.614+0000 7f8aaaef8700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8aa4060220 0x7f8aa41066a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:50:43.615 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:43.614+0000 7f8aaaef8700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aa4106b90 con 0x7f8aa4060220 2024-01-21T11:50:43.617 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:43.614+0000 7f8aa8c94700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8aa4060220 0x7f8aa41066a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:50:43.617 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:43.614+0000 7f8aa8c94700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8aa4060220 0x7f8aa41066a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:33372/0 (socket says 172.21.15.134:33372) 2024-01-21T11:50:43.617 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:43.614+0000 7f8aa8c94700 1 -- 172.21.15.134:0/2658280401 learned_addr learned my addr 172.21.15.134:0/2658280401 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:50:43.617 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:43.614+0000 7f8aa8c94700 1 -- 172.21.15.134:0/2658280401 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a98005040 con 0x7f8aa4060220 2024-01-21T11:50:43.617 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:43.615+0000 7f8aa8c94700 1 --2- 172.21.15.134:0/2658280401 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8aa4060220 0x7f8aa41066a0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7f8a98013510 tx=0x7f8a9800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:50:43.617 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:43.615+0000 7f8aa17fa700 1 -- 172.21.15.134:0/2658280401 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8a98009320 con 0x7f8aa4060220 2024-01-21T11:50:43.617 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:43.615+0000 7f8aaaef8700 1 -- 172.21.15.134:0/2658280401 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8aa4106d90 con 0x7f8aa4060220 2024-01-21T11:50:43.617 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:43.615+0000 7f8aa17fa700 1 -- 172.21.15.134:0/2658280401 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8a98013f00 con 0x7f8aa4060220 2024-01-21T11:50:43.617 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:43.615+0000 7f8aa17fa700 1 -- 172.21.15.134:0/2658280401 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8a98014dc0 con 0x7f8aa4060220 2024-01-21T11:50:43.617 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:43.615+0000 7f8aaaef8700 1 -- 172.21.15.134:0/2658280401 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8aa4107220 con 0x7f8aa4060220 2024-01-21T11:50:43.617 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:43.616+0000 7f8aa17fa700 1 -- 172.21.15.134:0/2658280401 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8a9801d3d0 con 0x7f8aa4060220 2024-01-21T11:50:43.617 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:43.616+0000 7f8aa17fa700 1 --2- 172.21.15.134:0/2658280401 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f8a9402ff60 0x7f8a94032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:50:43.618 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:43.616+0000 7f8aaaef8700 1 -- 172.21.15.134:0/2658280401 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8aa4045ad0 con 0x7f8aa4060220 2024-01-21T11:50:43.618 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:43.616+0000 7f8aa17fa700 1 -- 172.21.15.134:0/2658280401 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8a98041300 con 0x7f8aa4060220 2024-01-21T11:50:43.618 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:43.616+0000 7f8aa3fff700 1 --2- 172.21.15.134:0/2658280401 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f8a9402ff60 0x7f8a94032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:50:43.618 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:43.617+0000 7f8aa3fff700 1 --2- 172.21.15.134:0/2658280401 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f8a9402ff60 0x7f8a94032420 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f8a900096a0 tx=0x7f8a90006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:50:43.621 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:43.621+0000 7f8aa17fa700 1 -- 172.21.15.134:0/2658280401 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8a98012900 con 0x7f8aa4060220 2024-01-21T11:50:43.827 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:43.827+0000 7f8aaaef8700 1 -- 172.21.15.134:0/2658280401 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8aa41080d0 con 0x7f8aa4060220 2024-01-21T11:50:43.827 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:43.827+0000 7f8aa17fa700 1 -- 172.21.15.134:0/2658280401 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8a9802b030 con 0x7f8aa4060220 2024-01-21T11:50:43.828 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:50:43.828 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:50:43.829 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:43.829+0000 7f8aaaef8700 1 -- 172.21.15.134:0/2658280401 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f8a9402ff60 msgr2=0x7f8a94032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:50:43.829 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:43.829+0000 7f8aaaef8700 1 --2- 172.21.15.134:0/2658280401 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f8a9402ff60 0x7f8a94032420 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f8a900096a0 tx=0x7f8a90006b40).stop 2024-01-21T11:50:43.829 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:43.829+0000 7f8aaaef8700 1 -- 172.21.15.134:0/2658280401 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8aa4060220 msgr2=0x7f8aa41066a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:50:43.829 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:43.829+0000 7f8aaaef8700 1 --2- 172.21.15.134:0/2658280401 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8aa4060220 0x7f8aa41066a0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7f8a98013510 tx=0x7f8a9800b040).stop 2024-01-21T11:50:43.830 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:43.829+0000 7f8aaaef8700 1 -- 172.21.15.134:0/2658280401 shutdown_connections 2024-01-21T11:50:43.830 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:43.829+0000 7f8aaaef8700 1 --2- 172.21.15.134:0/2658280401 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f8a9402ff60 0x7f8a94032420 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:50:43.830 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:43.829+0000 7f8aaaef8700 1 --2- 172.21.15.134:0/2658280401 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8aa4060220 0x7f8aa41066a0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:50:43.830 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:43.829+0000 7f8aaaef8700 1 -- 172.21.15.134:0/2658280401 >> 172.21.15.134:0/2658280401 conn(0x7f8aa40f0b70 msgr2=0x7f8aa40f16a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:50:43.830 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:43.829+0000 7f8aaaef8700 1 -- 172.21.15.134:0/2658280401 shutdown_connections 2024-01-21T11:50:43.830 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:43.829+0000 7f8aaaef8700 1 -- 172.21.15.134:0/2658280401 wait complete. 2024-01-21T11:50:43.830 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:50:43.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:43 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:50:41.886186+0000 mgr.smithi029.blhxxk (mgr.14164) 268 : cluster [DBG] pgmap v254: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:50:44.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:44 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:50:43.828536+0000 mon.smithi029 (mon.0) 320 : audit [DBG] from='client.? 172.21.15.134:0/2658280401' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:50:45.215 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:50:45.215 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:50:45.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:45 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:50:43.886484+0000 mgr.smithi029.blhxxk (mgr.14164) 269 : cluster [DBG] pgmap v255: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:50:46.339 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:46.338+0000 7fa8a337a700 1 -- 172.21.15.134:0/2232172318 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa89c0f6510 msgr2=0x7fa89c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:50:46.339 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:46.338+0000 7fa8a337a700 1 --2- 172.21.15.134:0/2232172318 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa89c0f6510 0x7fa89c0f6930 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7fa898009fb0 tx=0x7fa898009b30).stop 2024-01-21T11:50:46.339 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:46.338+0000 7fa8a337a700 1 -- 172.21.15.134:0/2232172318 shutdown_connections 2024-01-21T11:50:46.339 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:46.338+0000 7fa8a337a700 1 --2- 172.21.15.134:0/2232172318 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa89c0f6510 0x7fa89c0f6930 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:50:46.339 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:46.338+0000 7fa8a337a700 1 -- 172.21.15.134:0/2232172318 >> 172.21.15.134:0/2232172318 conn(0x7fa89c0f1c70 msgr2=0x7fa89c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:50:46.340 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:46.338+0000 7fa8a337a700 1 -- 172.21.15.134:0/2232172318 shutdown_connections 2024-01-21T11:50:46.340 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:46.338+0000 7fa8a337a700 1 -- 172.21.15.134:0/2232172318 wait complete. 2024-01-21T11:50:46.340 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:46.339+0000 7fa8a337a700 1 Processor -- start 2024-01-21T11:50:46.340 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:46.339+0000 7fa8a337a700 1 -- start start 2024-01-21T11:50:46.340 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:46.339+0000 7fa8a337a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa89c0f6510 0x7fa89c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:50:46.340 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:46.339+0000 7fa8a337a700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa89c069e20 con 0x7fa89c0f6510 2024-01-21T11:50:46.340 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:46.339+0000 7fa8a1116700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa89c0f6510 0x7fa89c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:50:46.340 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:46.339+0000 7fa8a1116700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa89c0f6510 0x7fa89c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:50722/0 (socket says 172.21.15.134:50722) 2024-01-21T11:50:46.340 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:46.339+0000 7fa8a1116700 1 -- 172.21.15.134:0/2680031490 learned_addr learned my addr 172.21.15.134:0/2680031490 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:50:46.342 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:46.339+0000 7fa8a1116700 1 -- 172.21.15.134:0/2680031490 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa898005040 con 0x7fa89c0f6510 2024-01-21T11:50:46.342 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:46.340+0000 7fa8a1116700 1 --2- 172.21.15.134:0/2680031490 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa89c0f6510 0x7fa89c06b9a0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7fa898013510 tx=0x7fa89800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:50:46.342 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:46.340+0000 7fa891ffb700 1 -- 172.21.15.134:0/2680031490 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa898009320 con 0x7fa89c0f6510 2024-01-21T11:50:46.342 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:46.340+0000 7fa8a337a700 1 -- 172.21.15.134:0/2680031490 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa89c069fc0 con 0x7fa89c0f6510 2024-01-21T11:50:46.342 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:46.340+0000 7fa891ffb700 1 -- 172.21.15.134:0/2680031490 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa898013f00 con 0x7fa89c0f6510 2024-01-21T11:50:46.342 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:46.340+0000 7fa891ffb700 1 -- 172.21.15.134:0/2680031490 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa898014d90 con 0x7fa89c0f6510 2024-01-21T11:50:46.342 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:46.340+0000 7fa8a337a700 1 -- 172.21.15.134:0/2680031490 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa89c06a450 con 0x7fa89c0f6510 2024-01-21T11:50:46.342 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:46.341+0000 7fa8a337a700 1 -- 172.21.15.134:0/2680031490 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa89c059100 con 0x7fa89c0f6510 2024-01-21T11:50:46.342 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:46.341+0000 7fa891ffb700 1 -- 172.21.15.134:0/2680031490 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa8980095f0 con 0x7fa89c0f6510 2024-01-21T11:50:46.342 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:46.341+0000 7fa891ffb700 1 --2- 172.21.15.134:0/2680031490 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fa88802ff60 0x7fa888032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:50:46.342 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:46.341+0000 7fa891ffb700 1 -- 172.21.15.134:0/2680031490 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa898040fc0 con 0x7fa89c0f6510 2024-01-21T11:50:46.343 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:46.341+0000 7fa8a0915700 1 --2- 172.21.15.134:0/2680031490 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fa88802ff60 0x7fa888032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:50:46.343 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:46.342+0000 7fa8a0915700 1 --2- 172.21.15.134:0/2680031490 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fa88802ff60 0x7fa888032420 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7fa88c0096a0 tx=0x7fa88c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:50:46.346 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:46.346+0000 7fa891ffb700 1 -- 172.21.15.134:0/2680031490 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa898014420 con 0x7fa89c0f6510 2024-01-21T11:50:46.552 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:46.552+0000 7fa8a337a700 1 -- 172.21.15.134:0/2680031490 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa89c045ad0 con 0x7fa89c0f6510 2024-01-21T11:50:46.553 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:46.553+0000 7fa891ffb700 1 -- 172.21.15.134:0/2680031490 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa898010230 con 0x7fa89c0f6510 2024-01-21T11:50:46.553 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:50:46.553 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:50:46.555 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:46.554+0000 7fa8a337a700 1 -- 172.21.15.134:0/2680031490 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fa88802ff60 msgr2=0x7fa888032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:50:46.555 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:46.554+0000 7fa8a337a700 1 --2- 172.21.15.134:0/2680031490 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fa88802ff60 0x7fa888032420 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7fa88c0096a0 tx=0x7fa88c006b40).stop 2024-01-21T11:50:46.555 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:46.554+0000 7fa8a337a700 1 -- 172.21.15.134:0/2680031490 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa89c0f6510 msgr2=0x7fa89c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:50:46.555 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:46.554+0000 7fa8a337a700 1 --2- 172.21.15.134:0/2680031490 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa89c0f6510 0x7fa89c06b9a0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7fa898013510 tx=0x7fa89800b040).stop 2024-01-21T11:50:46.555 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:46.554+0000 7fa8a337a700 1 -- 172.21.15.134:0/2680031490 shutdown_connections 2024-01-21T11:50:46.555 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:46.554+0000 7fa8a337a700 1 --2- 172.21.15.134:0/2680031490 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fa88802ff60 0x7fa888032420 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:50:46.555 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:46.554+0000 7fa8a337a700 1 --2- 172.21.15.134:0/2680031490 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa89c0f6510 0x7fa89c06b9a0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:50:46.555 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:46.554+0000 7fa8a337a700 1 -- 172.21.15.134:0/2680031490 >> 172.21.15.134:0/2680031490 conn(0x7fa89c0f1c70 msgr2=0x7fa89c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:50:46.555 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:46.554+0000 7fa8a337a700 1 -- 172.21.15.134:0/2680031490 shutdown_connections 2024-01-21T11:50:46.555 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:46.554+0000 7fa8a337a700 1 -- 172.21.15.134:0/2680031490 wait complete. 2024-01-21T11:50:46.556 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:50:46.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:46 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:50:46.553974+0000 mon.smithi029 (mon.0) 321 : audit [DBG] from='client.? 172.21.15.134:0/2680031490' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:50:47.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:47 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:50:45.886797+0000 mgr.smithi029.blhxxk (mgr.14164) 270 : cluster [DBG] pgmap v256: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:50:47.928 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:50:47.928 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:50:48.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:48 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:50:47.887137+0000 mgr.smithi029.blhxxk (mgr.14164) 271 : cluster [DBG] pgmap v257: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:50:49.069 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:49.067+0000 7f40f31e6700 1 -- 172.21.15.134:0/2762295027 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40ec0f6510 msgr2=0x7f40ec0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:50:49.069 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:49.067+0000 7f40f31e6700 1 --2- 172.21.15.134:0/2762295027 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40ec0f6510 0x7f40ec0f6930 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7f40e0009fb0 tx=0x7f40e0009b30).stop 2024-01-21T11:50:49.070 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:49.067+0000 7f40f31e6700 1 -- 172.21.15.134:0/2762295027 shutdown_connections 2024-01-21T11:50:49.070 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:49.067+0000 7f40f31e6700 1 --2- 172.21.15.134:0/2762295027 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40ec0f6510 0x7f40ec0f6930 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:50:49.070 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:49.067+0000 7f40f31e6700 1 -- 172.21.15.134:0/2762295027 >> 172.21.15.134:0/2762295027 conn(0x7f40ec0f1c70 msgr2=0x7f40ec0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:50:49.070 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:49.067+0000 7f40f31e6700 1 -- 172.21.15.134:0/2762295027 shutdown_connections 2024-01-21T11:50:49.070 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:49.067+0000 7f40f31e6700 1 -- 172.21.15.134:0/2762295027 wait complete. 2024-01-21T11:50:49.070 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:49.068+0000 7f40f31e6700 1 Processor -- start 2024-01-21T11:50:49.070 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:49.068+0000 7f40f31e6700 1 -- start start 2024-01-21T11:50:49.070 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:49.068+0000 7f40f31e6700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40ec0f6510 0x7f40ec106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:50:49.070 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:49.068+0000 7f40f31e6700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40ec106c20 con 0x7f40ec0f6510 2024-01-21T11:50:49.070 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:49.068+0000 7f40f0f82700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40ec0f6510 0x7f40ec106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:50:49.070 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:49.068+0000 7f40f0f82700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40ec0f6510 0x7f40ec106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:50736/0 (socket says 172.21.15.134:50736) 2024-01-21T11:50:49.070 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:49.068+0000 7f40f0f82700 1 -- 172.21.15.134:0/3307110174 learned_addr learned my addr 172.21.15.134:0/3307110174 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:50:49.070 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:49.068+0000 7f40f0f82700 1 -- 172.21.15.134:0/3307110174 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40e0005040 con 0x7f40ec0f6510 2024-01-21T11:50:49.070 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:49.069+0000 7f40f0f82700 1 --2- 172.21.15.134:0/3307110174 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40ec0f6510 0x7f40ec106730 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f40e0013510 tx=0x7f40e000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:50:49.071 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:49.069+0000 7f40e97fa700 1 -- 172.21.15.134:0/3307110174 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f40e0009320 con 0x7f40ec0f6510 2024-01-21T11:50:49.071 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:49.069+0000 7f40e97fa700 1 -- 172.21.15.134:0/3307110174 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f40e0013f00 con 0x7f40ec0f6510 2024-01-21T11:50:49.071 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:49.069+0000 7f40f31e6700 1 -- 172.21.15.134:0/3307110174 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40ec106e20 con 0x7f40ec0f6510 2024-01-21T11:50:49.071 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:49.069+0000 7f40e97fa700 1 -- 172.21.15.134:0/3307110174 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f40e0014d90 con 0x7f40ec0f6510 2024-01-21T11:50:49.071 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:49.069+0000 7f40f31e6700 1 -- 172.21.15.134:0/3307110174 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40ec1072b0 con 0x7f40ec0f6510 2024-01-21T11:50:49.072 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:49.070+0000 7f40e97fa700 1 -- 172.21.15.134:0/3307110174 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f40e00095f0 con 0x7f40ec0f6510 2024-01-21T11:50:49.072 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:49.070+0000 7f40f31e6700 1 -- 172.21.15.134:0/3307110174 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40ec045ad0 con 0x7f40ec0f6510 2024-01-21T11:50:49.072 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:49.070+0000 7f40e97fa700 1 --2- 172.21.15.134:0/3307110174 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f40dc02ff60 0x7f40dc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:50:49.072 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:49.070+0000 7f40e97fa700 1 -- 172.21.15.134:0/3307110174 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f40e0041000 con 0x7f40ec0f6510 2024-01-21T11:50:49.072 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:49.070+0000 7f40ebfff700 1 --2- 172.21.15.134:0/3307110174 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f40dc02ff60 0x7f40dc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:50:49.072 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:49.070+0000 7f40ebfff700 1 --2- 172.21.15.134:0/3307110174 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f40dc02ff60 0x7f40dc032420 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f40d80096a0 tx=0x7f40d8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:50:49.074 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:49.074+0000 7f40e97fa700 1 -- 172.21.15.134:0/3307110174 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f40e001d7f0 con 0x7f40ec0f6510 2024-01-21T11:50:49.280 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:49.280+0000 7f40f31e6700 1 -- 172.21.15.134:0/3307110174 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f40ec0fbf60 con 0x7f40ec0f6510 2024-01-21T11:50:49.280 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:49.280+0000 7f40e97fa700 1 -- 172.21.15.134:0/3307110174 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f40e0010230 con 0x7f40ec0f6510 2024-01-21T11:50:49.281 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:50:49.281 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:50:49.282 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:49.282+0000 7f40f31e6700 1 -- 172.21.15.134:0/3307110174 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f40dc02ff60 msgr2=0x7f40dc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:50:49.282 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:49.282+0000 7f40f31e6700 1 --2- 172.21.15.134:0/3307110174 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f40dc02ff60 0x7f40dc032420 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f40d80096a0 tx=0x7f40d8006b40).stop 2024-01-21T11:50:49.282 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:49.282+0000 7f40f31e6700 1 -- 172.21.15.134:0/3307110174 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40ec0f6510 msgr2=0x7f40ec106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:50:49.283 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:49.282+0000 7f40f31e6700 1 --2- 172.21.15.134:0/3307110174 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40ec0f6510 0x7f40ec106730 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f40e0013510 tx=0x7f40e000b040).stop 2024-01-21T11:50:49.283 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:49.282+0000 7f40f31e6700 1 -- 172.21.15.134:0/3307110174 shutdown_connections 2024-01-21T11:50:49.283 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:49.282+0000 7f40f31e6700 1 --2- 172.21.15.134:0/3307110174 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f40dc02ff60 0x7f40dc032420 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:50:49.283 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:49.282+0000 7f40f31e6700 1 --2- 172.21.15.134:0/3307110174 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40ec0f6510 0x7f40ec106730 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:50:49.283 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:49.282+0000 7f40f31e6700 1 -- 172.21.15.134:0/3307110174 >> 172.21.15.134:0/3307110174 conn(0x7f40ec0f1c70 msgr2=0x7f40ec0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:50:49.283 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:49.282+0000 7f40f31e6700 1 -- 172.21.15.134:0/3307110174 shutdown_connections 2024-01-21T11:50:49.283 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:49.282+0000 7f40f31e6700 1 -- 172.21.15.134:0/3307110174 wait complete. 2024-01-21T11:50:49.283 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:50:49.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:49 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:50:49.281736+0000 mon.smithi029 (mon.0) 322 : audit [DBG] from='client.? 172.21.15.134:0/3307110174' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:50:50.663 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:50:50.663 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:50:51.857 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:51.856+0000 7ff790f9c700 1 -- 172.21.15.134:0/2084519602 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff78c0f62a0 msgr2=0x7ff78c0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:50:51.857 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:51.856+0000 7ff790f9c700 1 --2- 172.21.15.134:0/2084519602 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff78c0f62a0 0x7ff78c0f66c0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7ff77c009fb0 tx=0x7ff77c009b30).stop 2024-01-21T11:50:51.857 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:51.856+0000 7ff790f9c700 1 -- 172.21.15.134:0/2084519602 shutdown_connections 2024-01-21T11:50:51.858 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:51.856+0000 7ff790f9c700 1 --2- 172.21.15.134:0/2084519602 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff78c0f62a0 0x7ff78c0f66c0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:50:51.858 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:51.856+0000 7ff790f9c700 1 -- 172.21.15.134:0/2084519602 >> 172.21.15.134:0/2084519602 conn(0x7ff78c0f1aa0 msgr2=0x7ff78c0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:50:51.858 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:51.856+0000 7ff790f9c700 1 -- 172.21.15.134:0/2084519602 shutdown_connections 2024-01-21T11:50:51.858 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:51.856+0000 7ff790f9c700 1 -- 172.21.15.134:0/2084519602 wait complete. 2024-01-21T11:50:51.858 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:51.856+0000 7ff790f9c700 1 Processor -- start 2024-01-21T11:50:51.858 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:51.857+0000 7ff790f9c700 1 -- start start 2024-01-21T11:50:51.858 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:51.857+0000 7ff790f9c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff78c0f62a0 0x7ff78c1064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:50:51.858 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:51.857+0000 7ff790f9c700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff78c1069a0 con 0x7ff78c0f62a0 2024-01-21T11:50:51.858 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:51.857+0000 7ff78b7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff78c0f62a0 0x7ff78c1064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:50:51.858 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:51.857+0000 7ff78b7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff78c0f62a0 0x7ff78c1064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:50756/0 (socket says 172.21.15.134:50756) 2024-01-21T11:50:51.858 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:51.857+0000 7ff78b7fe700 1 -- 172.21.15.134:0/3486482697 learned_addr learned my addr 172.21.15.134:0/3486482697 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:50:51.860 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:51.857+0000 7ff78b7fe700 1 -- 172.21.15.134:0/3486482697 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff77c005040 con 0x7ff78c0f62a0 2024-01-21T11:50:51.860 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:51.858+0000 7ff78b7fe700 1 --2- 172.21.15.134:0/3486482697 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff78c0f62a0 0x7ff78c1064b0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7ff77c013510 tx=0x7ff77c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:50:51.860 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:51.858+0000 7ff773fff700 1 -- 172.21.15.134:0/3486482697 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff77c009320 con 0x7ff78c0f62a0 2024-01-21T11:50:51.860 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:51.858+0000 7ff773fff700 1 -- 172.21.15.134:0/3486482697 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff77c013f00 con 0x7ff78c0f62a0 2024-01-21T11:50:51.860 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:51.858+0000 7ff790f9c700 1 -- 172.21.15.134:0/3486482697 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff78c106ba0 con 0x7ff78c0f62a0 2024-01-21T11:50:51.860 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:51.858+0000 7ff773fff700 1 -- 172.21.15.134:0/3486482697 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff77c014dc0 con 0x7ff78c0f62a0 2024-01-21T11:50:51.860 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:51.858+0000 7ff790f9c700 1 -- 172.21.15.134:0/3486482697 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff78c107030 con 0x7ff78c0f62a0 2024-01-21T11:50:51.860 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:51.859+0000 7ff773fff700 1 -- 172.21.15.134:0/3486482697 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff77c0095f0 con 0x7ff78c0f62a0 2024-01-21T11:50:51.860 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:51.859+0000 7ff790f9c700 1 -- 172.21.15.134:0/3486482697 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff78c045ad0 con 0x7ff78c0f62a0 2024-01-21T11:50:51.861 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:51.859+0000 7ff773fff700 1 --2- 172.21.15.134:0/3486482697 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7ff77402ff60 0x7ff774032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:50:51.861 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:51.859+0000 7ff773fff700 1 -- 172.21.15.134:0/3486482697 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff77c041100 con 0x7ff78c0f62a0 2024-01-21T11:50:51.861 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:51.859+0000 7ff78affd700 1 --2- 172.21.15.134:0/3486482697 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7ff77402ff60 0x7ff774032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:50:51.861 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:51.859+0000 7ff78affd700 1 --2- 172.21.15.134:0/3486482697 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7ff77402ff60 0x7ff774032420 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7ff7800096a0 tx=0x7ff780006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:50:51.863 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:51.864+0000 7ff773fff700 1 -- 172.21.15.134:0/3486482697 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff77c014420 con 0x7ff78c0f62a0 2024-01-21T11:50:51.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:51 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:50:49.887465+0000 mgr.smithi029.blhxxk (mgr.14164) 272 : cluster [DBG] pgmap v258: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:50:52.067 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:52.067+0000 7ff790f9c700 1 -- 172.21.15.134:0/3486482697 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff78c107ee0 con 0x7ff78c0f62a0 2024-01-21T11:50:52.068 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:52.068+0000 7ff773fff700 1 -- 172.21.15.134:0/3486482697 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff77c010230 con 0x7ff78c0f62a0 2024-01-21T11:50:52.068 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:50:52.068 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:50:52.069 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:52.069+0000 7ff790f9c700 1 -- 172.21.15.134:0/3486482697 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7ff77402ff60 msgr2=0x7ff774032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:50:52.070 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:52.069+0000 7ff790f9c700 1 --2- 172.21.15.134:0/3486482697 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7ff77402ff60 0x7ff774032420 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7ff7800096a0 tx=0x7ff780006b40).stop 2024-01-21T11:50:52.070 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:52.069+0000 7ff790f9c700 1 -- 172.21.15.134:0/3486482697 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff78c0f62a0 msgr2=0x7ff78c1064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:50:52.070 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:52.069+0000 7ff790f9c700 1 --2- 172.21.15.134:0/3486482697 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff78c0f62a0 0x7ff78c1064b0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7ff77c013510 tx=0x7ff77c00b040).stop 2024-01-21T11:50:52.070 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:52.069+0000 7ff790f9c700 1 -- 172.21.15.134:0/3486482697 shutdown_connections 2024-01-21T11:50:52.070 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:52.069+0000 7ff790f9c700 1 --2- 172.21.15.134:0/3486482697 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7ff77402ff60 0x7ff774032420 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:50:52.070 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:52.069+0000 7ff790f9c700 1 --2- 172.21.15.134:0/3486482697 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff78c0f62a0 0x7ff78c1064b0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:50:52.070 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:52.069+0000 7ff790f9c700 1 -- 172.21.15.134:0/3486482697 >> 172.21.15.134:0/3486482697 conn(0x7ff78c0f1aa0 msgr2=0x7ff78c0f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:50:52.070 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:52.069+0000 7ff790f9c700 1 -- 172.21.15.134:0/3486482697 shutdown_connections 2024-01-21T11:50:52.070 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:52.069+0000 7ff790f9c700 1 -- 172.21.15.134:0/3486482697 wait complete. 2024-01-21T11:50:52.070 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:50:52.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:52 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:50:52.068792+0000 mon.smithi029 (mon.0) 323 : audit [DBG] from='client.? 172.21.15.134:0/3486482697' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:50:53.456 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:50:53.456 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:50:53.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:53 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:50:51.887710+0000 mgr.smithi029.blhxxk (mgr.14164) 273 : cluster [DBG] pgmap v259: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:50:54.621 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:54.619+0000 7f80afa26700 1 -- 172.21.15.134:0/931636999 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80a80f5410 msgr2=0x7f80a80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:50:54.621 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:54.619+0000 7f80afa26700 1 --2- 172.21.15.134:0/931636999 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80a80f5410 0x7f80a80f5830 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7f80a4009fb0 tx=0x7f80a4009b30).stop 2024-01-21T11:50:54.621 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:54.619+0000 7f80afa26700 1 -- 172.21.15.134:0/931636999 shutdown_connections 2024-01-21T11:50:54.621 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:54.619+0000 7f80afa26700 1 --2- 172.21.15.134:0/931636999 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80a80f5410 0x7f80a80f5830 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:50:54.621 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:54.619+0000 7f80afa26700 1 -- 172.21.15.134:0/931636999 >> 172.21.15.134:0/931636999 conn(0x7f80a80f0b70 msgr2=0x7f80a80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:50:54.621 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:54.619+0000 7f80afa26700 1 -- 172.21.15.134:0/931636999 shutdown_connections 2024-01-21T11:50:54.621 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:54.619+0000 7f80afa26700 1 -- 172.21.15.134:0/931636999 wait complete. 2024-01-21T11:50:54.622 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:54.619+0000 7f80afa26700 1 Processor -- start 2024-01-21T11:50:54.622 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:54.620+0000 7f80afa26700 1 -- start start 2024-01-21T11:50:54.622 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:54.620+0000 7f80afa26700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80a80f5410 0x7f80a8109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:50:54.622 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:54.620+0000 7f80afa26700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80a8060240 con 0x7f80a80f5410 2024-01-21T11:50:54.622 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:54.620+0000 7f80ad7c2700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80a80f5410 0x7f80a8109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:50:54.622 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:54.620+0000 7f80ad7c2700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80a80f5410 0x7f80a8109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:40328/0 (socket says 172.21.15.134:40328) 2024-01-21T11:50:54.622 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:54.620+0000 7f80ad7c2700 1 -- 172.21.15.134:0/2094525985 learned_addr learned my addr 172.21.15.134:0/2094525985 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:50:54.622 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:54.620+0000 7f80ad7c2700 1 -- 172.21.15.134:0/2094525985 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80a4005040 con 0x7f80a80f5410 2024-01-21T11:50:54.622 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:54.620+0000 7f80ad7c2700 1 --2- 172.21.15.134:0/2094525985 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80a80f5410 0x7f80a8109430 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7f80a4013510 tx=0x7f80a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:50:54.622 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:54.620+0000 7f809dffb700 1 -- 172.21.15.134:0/2094525985 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f80a4009320 con 0x7f80a80f5410 2024-01-21T11:50:54.622 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:54.621+0000 7f80afa26700 1 -- 172.21.15.134:0/2094525985 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80a8060440 con 0x7f80a80f5410 2024-01-21T11:50:54.622 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:54.621+0000 7f809dffb700 1 -- 172.21.15.134:0/2094525985 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f80a4013f00 con 0x7f80a80f5410 2024-01-21T11:50:54.622 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:54.621+0000 7f809dffb700 1 -- 172.21.15.134:0/2094525985 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f80a4014d90 con 0x7f80a80f5410 2024-01-21T11:50:54.623 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:54.621+0000 7f80afa26700 1 -- 172.21.15.134:0/2094525985 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80a8109ae0 con 0x7f80a80f5410 2024-01-21T11:50:54.623 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:54.622+0000 7f809dffb700 1 -- 172.21.15.134:0/2094525985 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f80a4014420 con 0x7f80a80f5410 2024-01-21T11:50:54.624 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:54.622+0000 7f809dffb700 1 --2- 172.21.15.134:0/2094525985 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f809402ffb0 0x7f8094032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:50:54.624 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:54.622+0000 7f80afa26700 1 -- 172.21.15.134:0/2094525985 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80a8045ad0 con 0x7f80a80f5410 2024-01-21T11:50:54.624 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:54.622+0000 7f809dffb700 1 -- 172.21.15.134:0/2094525985 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f80a4040ff0 con 0x7f80a80f5410 2024-01-21T11:50:54.624 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:54.622+0000 7f80acfc1700 1 --2- 172.21.15.134:0/2094525985 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f809402ffb0 0x7f8094032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:50:54.624 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:54.622+0000 7f80acfc1700 1 --2- 172.21.15.134:0/2094525985 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f809402ffb0 0x7f8094032470 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f80980096a0 tx=0x7f8098006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:50:54.626 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:54.626+0000 7f809dffb700 1 -- 172.21.15.134:0/2094525985 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f80a40095f0 con 0x7f80a80f5410 2024-01-21T11:50:54.833 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:54.832+0000 7f80afa26700 1 -- 172.21.15.134:0/2094525985 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f80a80faf50 con 0x7f80a80f5410 2024-01-21T11:50:54.833 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:54.833+0000 7f809dffb700 1 -- 172.21.15.134:0/2094525985 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f80a401db70 con 0x7f80a80f5410 2024-01-21T11:50:54.834 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:50:54.834 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:50:54.835 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:54.835+0000 7f80afa26700 1 -- 172.21.15.134:0/2094525985 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f809402ffb0 msgr2=0x7f8094032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:50:54.835 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:54.835+0000 7f80afa26700 1 --2- 172.21.15.134:0/2094525985 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f809402ffb0 0x7f8094032470 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f80980096a0 tx=0x7f8098006b40).stop 2024-01-21T11:50:54.835 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:54.835+0000 7f80afa26700 1 -- 172.21.15.134:0/2094525985 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80a80f5410 msgr2=0x7f80a8109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:50:54.835 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:54.835+0000 7f80afa26700 1 --2- 172.21.15.134:0/2094525985 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80a80f5410 0x7f80a8109430 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7f80a4013510 tx=0x7f80a400b040).stop 2024-01-21T11:50:54.835 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:54.835+0000 7f80afa26700 1 -- 172.21.15.134:0/2094525985 shutdown_connections 2024-01-21T11:50:54.835 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:54.835+0000 7f80afa26700 1 --2- 172.21.15.134:0/2094525985 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f809402ffb0 0x7f8094032470 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:50:54.835 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:54.835+0000 7f80afa26700 1 --2- 172.21.15.134:0/2094525985 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80a80f5410 0x7f80a8109430 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:50:54.836 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:54.835+0000 7f80afa26700 1 -- 172.21.15.134:0/2094525985 >> 172.21.15.134:0/2094525985 conn(0x7f80a80f0b70 msgr2=0x7f80a80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:50:54.836 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:54.835+0000 7f80afa26700 1 -- 172.21.15.134:0/2094525985 shutdown_connections 2024-01-21T11:50:54.836 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:54.835+0000 7f80afa26700 1 -- 172.21.15.134:0/2094525985 wait complete. 2024-01-21T11:50:54.836 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:50:55.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:55 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:50:53.888054+0000 mgr.smithi029.blhxxk (mgr.14164) 274 : cluster [DBG] pgmap v260: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:50:55.919 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:55 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:50:54.834446+0000 mon.smithi029 (mon.0) 324 : audit [DBG] from='client.? 172.21.15.134:0/2094525985' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:50:56.163 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:50:56.163 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:50:57.306 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:57.306+0000 7fe965fa3700 1 -- 172.21.15.134:0/226040482 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9600f5410 msgr2=0x7fe9600f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:50:57.306 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:57.306+0000 7fe965fa3700 1 --2- 172.21.15.134:0/226040482 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9600f5410 0x7fe9600f5830 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7fe950009fb0 tx=0x7fe950009b30).stop 2024-01-21T11:50:57.306 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:57.306+0000 7fe965fa3700 1 -- 172.21.15.134:0/226040482 shutdown_connections 2024-01-21T11:50:57.306 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:57.306+0000 7fe965fa3700 1 --2- 172.21.15.134:0/226040482 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9600f5410 0x7fe9600f5830 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:50:57.306 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:57.306+0000 7fe965fa3700 1 -- 172.21.15.134:0/226040482 >> 172.21.15.134:0/226040482 conn(0x7fe9600f0b70 msgr2=0x7fe9600f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:50:57.306 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:57.306+0000 7fe965fa3700 1 -- 172.21.15.134:0/226040482 shutdown_connections 2024-01-21T11:50:57.306 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:57.306+0000 7fe965fa3700 1 -- 172.21.15.134:0/226040482 wait complete. 2024-01-21T11:50:57.307 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:57.306+0000 7fe965fa3700 1 Processor -- start 2024-01-21T11:50:57.307 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:57.306+0000 7fe965fa3700 1 -- start start 2024-01-21T11:50:57.307 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:57.306+0000 7fe965fa3700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9600f5410 0x7fe960109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:50:57.307 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:57.306+0000 7fe965fa3700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe960060220 con 0x7fe9600f5410 2024-01-21T11:50:57.309 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:57.306+0000 7fe95f7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9600f5410 0x7fe960109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:50:57.309 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:57.306+0000 7fe95f7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9600f5410 0x7fe960109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:40350/0 (socket says 172.21.15.134:40350) 2024-01-21T11:50:57.309 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:57.306+0000 7fe95f7fe700 1 -- 172.21.15.134:0/1400709314 learned_addr learned my addr 172.21.15.134:0/1400709314 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:50:57.309 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:57.307+0000 7fe95f7fe700 1 -- 172.21.15.134:0/1400709314 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe950005040 con 0x7fe9600f5410 2024-01-21T11:50:57.309 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:57.307+0000 7fe95f7fe700 1 --2- 172.21.15.134:0/1400709314 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9600f5410 0x7fe960109450 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7fe950013510 tx=0x7fe95000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:50:57.309 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:57.307+0000 7fe957fff700 1 -- 172.21.15.134:0/1400709314 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe950009320 con 0x7fe9600f5410 2024-01-21T11:50:57.309 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:57.307+0000 7fe965fa3700 1 -- 172.21.15.134:0/1400709314 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe960060420 con 0x7fe9600f5410 2024-01-21T11:50:57.309 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:57.307+0000 7fe957fff700 1 -- 172.21.15.134:0/1400709314 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe950013f00 con 0x7fe9600f5410 2024-01-21T11:50:57.309 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:57.307+0000 7fe957fff700 1 -- 172.21.15.134:0/1400709314 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe950014d90 con 0x7fe9600f5410 2024-01-21T11:50:57.309 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:57.307+0000 7fe965fa3700 1 -- 172.21.15.134:0/1400709314 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe960109b00 con 0x7fe9600f5410 2024-01-21T11:50:57.309 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:57.308+0000 7fe965fa3700 1 -- 172.21.15.134:0/1400709314 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe960059100 con 0x7fe9600f5410 2024-01-21T11:50:57.310 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:57.308+0000 7fe957fff700 1 -- 172.21.15.134:0/1400709314 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe9500095f0 con 0x7fe9600f5410 2024-01-21T11:50:57.310 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:57.308+0000 7fe957fff700 1 --2- 172.21.15.134:0/1400709314 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fe94c02ff60 0x7fe94c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:50:57.310 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:57.308+0000 7fe957fff700 1 -- 172.21.15.134:0/1400709314 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe950040fc0 con 0x7fe9600f5410 2024-01-21T11:50:57.310 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:57.308+0000 7fe95effd700 1 --2- 172.21.15.134:0/1400709314 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fe94c02ff60 0x7fe94c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:50:57.310 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:57.309+0000 7fe95effd700 1 --2- 172.21.15.134:0/1400709314 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fe94c02ff60 0x7fe94c032420 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7fe9480096a0 tx=0x7fe948006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:50:57.313 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:57.313+0000 7fe957fff700 1 -- 172.21.15.134:0/1400709314 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe950014420 con 0x7fe9600f5410 2024-01-21T11:50:57.518 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:57.519+0000 7fe965fa3700 1 -- 172.21.15.134:0/1400709314 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe960045ad0 con 0x7fe9600f5410 2024-01-21T11:50:57.519 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:57.520+0000 7fe957fff700 1 -- 172.21.15.134:0/1400709314 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe950010230 con 0x7fe9600f5410 2024-01-21T11:50:57.519 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:50:57.519 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:50:57.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:57.521+0000 7fe965fa3700 1 -- 172.21.15.134:0/1400709314 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fe94c02ff60 msgr2=0x7fe94c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:50:57.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:57.521+0000 7fe965fa3700 1 --2- 172.21.15.134:0/1400709314 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fe94c02ff60 0x7fe94c032420 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7fe9480096a0 tx=0x7fe948006b40).stop 2024-01-21T11:50:57.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:57.521+0000 7fe965fa3700 1 -- 172.21.15.134:0/1400709314 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9600f5410 msgr2=0x7fe960109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:50:57.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:57.522+0000 7fe965fa3700 1 --2- 172.21.15.134:0/1400709314 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9600f5410 0x7fe960109450 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7fe950013510 tx=0x7fe95000b040).stop 2024-01-21T11:50:57.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:57.522+0000 7fe965fa3700 1 -- 172.21.15.134:0/1400709314 shutdown_connections 2024-01-21T11:50:57.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:57.522+0000 7fe965fa3700 1 --2- 172.21.15.134:0/1400709314 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7fe94c02ff60 0x7fe94c032420 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:50:57.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:57.522+0000 7fe965fa3700 1 --2- 172.21.15.134:0/1400709314 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9600f5410 0x7fe960109450 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:50:57.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:57.522+0000 7fe965fa3700 1 -- 172.21.15.134:0/1400709314 >> 172.21.15.134:0/1400709314 conn(0x7fe9600f0b70 msgr2=0x7fe9600f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:50:57.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:57.522+0000 7fe965fa3700 1 -- 172.21.15.134:0/1400709314 shutdown_connections 2024-01-21T11:50:57.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:50:57.522+0000 7fe965fa3700 1 -- 172.21.15.134:0/1400709314 wait complete. 2024-01-21T11:50:57.522 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:50:57.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:57 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:50:55.888433+0000 mgr.smithi029.blhxxk (mgr.14164) 275 : cluster [DBG] pgmap v261: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:50:57.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:57 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:50:57.520131+0000 mon.smithi029 (mon.0) 325 : audit [DBG] from='client.? 172.21.15.134:0/1400709314' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:50:58.960 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T11:50:58.960 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4084074-b851-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T11:50:59.918 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:50:59 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: cluster 2024-01-21T11:50:57.888809+0000 mgr.smithi029.blhxxk (mgr.14164) 276 : cluster [DBG] pgmap v262: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T11:51:00.159 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:51:00.158+0000 7f12a8909700 1 -- 172.21.15.134:0/2530831424 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12a40f4740 msgr2=0x7f12a40f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:51:00.160 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:51:00.158+0000 7f12a8909700 1 --2- 172.21.15.134:0/2530831424 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12a40f4740 0x7f12a40f4b60 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 rx=0x7f1294009fb0 tx=0x7f1294009b30).stop 2024-01-21T11:51:00.160 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:51:00.158+0000 7f12a8909700 1 -- 172.21.15.134:0/2530831424 shutdown_connections 2024-01-21T11:51:00.160 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:51:00.158+0000 7f12a8909700 1 --2- 172.21.15.134:0/2530831424 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12a40f4740 0x7f12a40f4b60 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:51:00.160 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:51:00.158+0000 7f12a8909700 1 -- 172.21.15.134:0/2530831424 >> 172.21.15.134:0/2530831424 conn(0x7f12a40f0260 msgr2=0x7f12a40f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:51:00.160 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:51:00.158+0000 7f12a8909700 1 -- 172.21.15.134:0/2530831424 shutdown_connections 2024-01-21T11:51:00.160 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:51:00.158+0000 7f12a8909700 1 -- 172.21.15.134:0/2530831424 wait complete. 2024-01-21T11:51:00.160 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:51:00.158+0000 7f12a8909700 1 Processor -- start 2024-01-21T11:51:00.160 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:51:00.158+0000 7f12a8909700 1 -- start start 2024-01-21T11:51:00.160 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:51:00.158+0000 7f12a8909700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12a40f4740 0x7f12a41066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:51:00.160 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:51:00.158+0000 7f12a8909700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12a4106bc0 con 0x7f12a40f4740 2024-01-21T11:51:00.160 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:51:00.159+0000 7f12a259c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12a40f4740 0x7f12a41066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:51:00.160 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:51:00.159+0000 7f12a259c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12a40f4740 0x7f12a41066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.134:40376/0 (socket says 172.21.15.134:40376) 2024-01-21T11:51:00.160 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:51:00.159+0000 7f12a259c700 1 -- 172.21.15.134:0/517921964 learned_addr learned my addr 172.21.15.134:0/517921964 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-21T11:51:00.161 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:51:00.159+0000 7f12a259c700 1 -- 172.21.15.134:0/517921964 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1294005040 con 0x7f12a40f4740 2024-01-21T11:51:00.161 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:51:00.159+0000 7f12a259c700 1 --2- 172.21.15.134:0/517921964 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12a40f4740 0x7f12a41066d0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 rx=0x7f1294013510 tx=0x7f129400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:51:00.161 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:51:00.159+0000 7f1292ffd700 1 -- 172.21.15.134:0/517921964 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1294009320 con 0x7f12a40f4740 2024-01-21T11:51:00.162 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:51:00.159+0000 7f12a8909700 1 -- 172.21.15.134:0/517921964 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12a4106dc0 con 0x7f12a40f4740 2024-01-21T11:51:00.162 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:51:00.159+0000 7f1292ffd700 1 -- 172.21.15.134:0/517921964 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1294013f00 con 0x7f12a40f4740 2024-01-21T11:51:00.162 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:51:00.159+0000 7f1292ffd700 1 -- 172.21.15.134:0/517921964 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1294014d90 con 0x7f12a40f4740 2024-01-21T11:51:00.162 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:51:00.159+0000 7f12a8909700 1 -- 172.21.15.134:0/517921964 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12a4107250 con 0x7f12a40f4740 2024-01-21T11:51:00.162 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:51:00.160+0000 7f1292ffd700 1 -- 172.21.15.134:0/517921964 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1294014420 con 0x7f12a40f4740 2024-01-21T11:51:00.162 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:51:00.160+0000 7f1292ffd700 1 --2- 172.21.15.134:0/517921964 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f128c02ffb0 0x7f128c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T11:51:00.162 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:51:00.160+0000 7f1292ffd700 1 -- 172.21.15.134:0/517921964 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1294041090 con 0x7f12a40f4740 2024-01-21T11:51:00.162 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:51:00.161+0000 7f12a8909700 1 -- 172.21.15.134:0/517921964 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12a4059100 con 0x7f12a40f4740 2024-01-21T11:51:00.162 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:51:00.161+0000 7f12a1d9b700 1 --2- 172.21.15.134:0/517921964 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f128c02ffb0 0x7f128c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T11:51:00.162 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:51:00.161+0000 7f12a1d9b700 1 --2- 172.21.15.134:0/517921964 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f128c02ffb0 0x7f128c032470 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f12980096a0 tx=0x7f1298006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T11:51:00.165 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:51:00.165+0000 7f1292ffd700 1 -- 172.21.15.134:0/517921964 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f12940095f0 con 0x7f12a40f4740 2024-01-21T11:51:00.370 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:51:00.369+0000 7f12a8909700 1 -- 172.21.15.134:0/517921964 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f12a4045ad0 con 0x7f12a40f4740 2024-01-21T11:51:00.370 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:51:00.370+0000 7f1292ffd700 1 -- 172.21.15.134:0/517921964 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f129401db70 con 0x7f12a40f4740 2024-01-21T11:51:00.371 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-21T11:51:00.371 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"e4084074-b851-11ee-95b0-87774f69a715","modified":"2024-01-21T11:41:00.354236Z","created":"2024-01-21T11:41:00.354236Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi029","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T11:51:00.372 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:51:00.372+0000 7f12a8909700 1 -- 172.21.15.134:0/517921964 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f128c02ffb0 msgr2=0x7f128c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:51:00.372 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:51:00.372+0000 7f12a8909700 1 --2- 172.21.15.134:0/517921964 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f128c02ffb0 0x7f128c032470 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f12980096a0 tx=0x7f1298006b40).stop 2024-01-21T11:51:00.373 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:51:00.372+0000 7f12a8909700 1 -- 172.21.15.134:0/517921964 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12a40f4740 msgr2=0x7f12a41066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T11:51:00.373 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:51:00.372+0000 7f12a8909700 1 --2- 172.21.15.134:0/517921964 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12a40f4740 0x7f12a41066d0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 rx=0x7f1294013510 tx=0x7f129400b040).stop 2024-01-21T11:51:00.373 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:51:00.372+0000 7f12a8909700 1 -- 172.21.15.134:0/517921964 shutdown_connections 2024-01-21T11:51:00.373 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:51:00.372+0000 7f12a8909700 1 --2- 172.21.15.134:0/517921964 >> [v2:172.21.15.29:6800/3399348778,v1:172.21.15.29:6801/3399348778] conn(0x7f128c02ffb0 0x7f128c032470 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:51:00.373 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:51:00.372+0000 7f12a8909700 1 --2- 172.21.15.134:0/517921964 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12a40f4740 0x7f12a41066d0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T11:51:00.373 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:51:00.372+0000 7f12a8909700 1 -- 172.21.15.134:0/517921964 >> 172.21.15.134:0/517921964 conn(0x7f12a40f0260 msgr2=0x7f12a40f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T11:51:00.373 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:51:00.372+0000 7f12a8909700 1 -- 172.21.15.134:0/517921964 shutdown_connections 2024-01-21T11:51:00.373 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-21T11:51:00.372+0000 7f12a8909700 1 -- 172.21.15.134:0/517921964 wait complete. 2024-01-21T11:51:00.373 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-01-21T11:51:00.778 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds 2024-01-21T11:51:00.779 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-01-21T11:51:00.779 DEBUG:teuthology.orchestra.run.smithi029:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-01-21T11:51:00.804 DEBUG:teuthology.orchestra.run.smithi134:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-01-21T11:51:00.822 INFO:tasks.cephadm:Stopping all daemons... 2024-01-21T11:51:00.823 INFO:tasks.cephadm.mon.smithi029:Stopping mon.smithi029... 2024-01-21T11:51:00.823 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ceph-e4084074-b851-11ee-95b0-87774f69a715@mon.smithi029 2024-01-21T11:51:00.877 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:51:00 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: audit 2024-01-21T11:51:00.371445+0000 mon.smithi029 (mon.0) 326 : audit [DBG] from='client.? 172.21.15.134:0/517921964' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T11:51:01.169 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:51:00 smithi029 systemd[1]: Stopping Ceph mon.smithi029 for e4084074-b851-11ee-95b0-87774f69a715... 2024-01-21T11:51:01.169 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:51:00 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:51:00.957+0000 7fbbea08f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi029 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-01-21T11:51:01.169 INFO:journalctl@ceph.mon.smithi029.smithi029.stdout:Jan 21 11:51:00 smithi029 ceph-e4084074-b851-11ee-95b0-87774f69a715-mon.smithi029[38814]: debug 2024-01-21T11:51:00.957+0000 7fbbea08f700 -1 mon.smithi029@0(leader) e1 *** Got Signal Terminated *** 2024-01-21T11:51:01.388 DEBUG:teuthology.orchestra.run.smithi029:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e4084074-b851-11ee-95b0-87774f69a715@mon.smithi029.service' 2024-01-21T11:51:01.509 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-21T11:51:01.509 INFO:tasks.cephadm.mon.smithi029:Stopped mon.smithi029 2024-01-21T11:51:01.509 INFO:tasks.cephadm.mon.smithi134:Stopping mon.smithi134... 2024-01-21T11:51:01.509 DEBUG:teuthology.orchestra.run.smithi134:> sudo systemctl stop ceph-e4084074-b851-11ee-95b0-87774f69a715@mon.smithi134 2024-01-21T11:51:01.544 INFO:teuthology.orchestra.run.smithi134.stderr:Failed to stop ceph-e4084074-b851-11ee-95b0-87774f69a715@mon.smithi134.service: Unit ceph-e4084074-b851-11ee-95b0-87774f69a715@mon.smithi134.service not loaded. 2024-01-21T11:51:01.546 DEBUG:teuthology.orchestra.run:got remote process result: 5 2024-01-21T11:51:01.546 ERROR:tasks.cephadm:Failed to stop "ceph.mon.smithi134" Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi134 with status 5: 'sudo systemctl stop ceph-e4084074-b851-11ee-95b0-87774f69a715@mon.smithi134' 2024-01-21T11:51:01.547 INFO:tasks.cephadm:Archiving crash dumps... 2024-01-21T11:51:01.549 DEBUG:teuthology.misc:Transferring archived files from smithi029:/var/lib/ceph/e4084074-b851-11ee-95b0-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-01-19_23:17:02-rados-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7524471/remote/ubuntu@smithi029.front.sepia.ceph.com/crash 2024-01-21T11:51:01.550 DEBUG:teuthology.orchestra.run.smithi029:> sudo tar c -f - -C /var/lib/ceph/e4084074-b851-11ee-95b0-87774f69a715/crash -- . 2024-01-21T11:51:01.596 DEBUG:teuthology.misc:Transferring archived files from smithi134:/var/lib/ceph/e4084074-b851-11ee-95b0-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-01-19_23:17:02-rados-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7524471/remote/ubuntu@smithi134.front.sepia.ceph.com/crash 2024-01-21T11:51:01.597 DEBUG:teuthology.orchestra.run.smithi134:> sudo tar c -f - -C /var/lib/ceph/e4084074-b851-11ee-95b0-87774f69a715/crash -- . 2024-01-21T11:51:01.645 INFO:teuthology.orchestra.run.smithi134.stderr:tar: /var/lib/ceph/e4084074-b851-11ee-95b0-87774f69a715/crash: Cannot open: No such file or directory 2024-01-21T11:51:01.645 INFO:teuthology.orchestra.run.smithi134.stderr:tar: Error is not recoverable: exiting now 2024-01-21T11:51:01.646 INFO:tasks.cephadm:Checking cluster log for badness... 2024-01-21T11:51:01.647 DEBUG:teuthology.orchestra.run.smithi029:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/e4084074-b851-11ee-95b0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-01-21T11:51:01.679 INFO:teuthology.orchestra.run.smithi029.stderr:grep: /var/log/ceph/e4084074-b851-11ee-95b0-87774f69a715/ceph.log: No such file or directory 2024-01-21T11:51:01.681 INFO:tasks.cephadm:Compressing logs... 2024-01-21T11:51:01.682 DEBUG:teuthology.orchestra.run.smithi029:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-01-21T11:51:01.724 DEBUG:teuthology.orchestra.run.smithi134:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-01-21T11:51:01.751 INFO:teuthology.orchestra.run.smithi029.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-01-21T11:51:01.751 INFO:teuthology.orchestra.run.smithi029.stderr:: No such file or directory 2024-01-21T11:51:01.751 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/e4084074-b851-11ee-95b0-87774f69a715/ceph-volume.log 2024-01-21T11:51:01.752 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/e4084074-b851-11ee-95b0-87774f69a715/ceph-mon.smithi029.log 2024-01-21T11:51:01.752 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/e4084074-b851-11ee-95b0-87774f69a715/ceph-mgr.smithi029.blhxxk.log 2024-01-21T11:51:01.752 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/e4084074-b851-11ee-95b0-87774f69a715/ceph-volume.log: /var/log/ceph/e4084074-b851-11ee-95b0-87774f69a715/ceph-mon.smithi029.log: /var/log/ceph/e4084074-b851-11ee-95b0-87774f69a715/ceph-mgr.smithi029.blhxxk.log: 88.6% -- replaced with /var/log/ceph/e4084074-b851-11ee-95b0-87774f69a715/ceph-volume.log.gz 2024-01-21T11:51:01.753 INFO:teuthology.orchestra.run.smithi134.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-01-21T11:51:01.753 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-01-21T11:51:01.754 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cephadm.log: 96.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-01-21T11:51:01.756 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-01-21T11:51:01.756 INFO:teuthology.orchestra.run.smithi134.stderr:real 0m0.014s 2024-01-21T11:51:01.756 INFO:teuthology.orchestra.run.smithi134.stderr:user 0m0.009s 2024-01-21T11:51:01.756 INFO:teuthology.orchestra.run.smithi134.stderr:sys 0m0.014s 2024-01-21T11:51:01.756 INFO:teuthology.orchestra.run.smithi029.stderr: 91.8% -- replaced with /var/log/ceph/cephadm.log.gz 2024-01-21T11:51:01.831 INFO:teuthology.orchestra.run.smithi029.stderr:gzip: /var/log/ceph/e4084074-b851-11ee-95b0-87774f69a715/ceph-mgr.smithi029.blhxxk.log: file size changed while zipping 2024-01-21T11:51:01.832 INFO:teuthology.orchestra.run.smithi029.stderr: 92.7% -- replaced with /var/log/ceph/e4084074-b851-11ee-95b0-87774f69a715/ceph-mgr.smithi029.blhxxk.log.gz 2024-01-21T11:51:01.902 INFO:teuthology.orchestra.run.smithi029.stderr: 90.5% -- replaced with /var/log/ceph/e4084074-b851-11ee-95b0-87774f69a715/ceph-mon.smithi029.log.gz 2024-01-21T11:51:01.904 INFO:teuthology.orchestra.run.smithi029.stderr: 2024-01-21T11:51:01.905 INFO:teuthology.orchestra.run.smithi029.stderr:real 0m0.164s 2024-01-21T11:51:01.905 INFO:teuthology.orchestra.run.smithi029.stderr:user 0m0.246s 2024-01-21T11:51:01.905 INFO:teuthology.orchestra.run.smithi029.stderr:sys 0m0.014s 2024-01-21T11:51:01.905 INFO:tasks.cephadm:Archiving logs... 2024-01-21T11:51:01.905 DEBUG:teuthology.misc:Transferring archived files from smithi029:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-01-19_23:17:02-rados-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7524471/remote/ubuntu@smithi029.front.sepia.ceph.com/log 2024-01-21T11:51:01.906 DEBUG:teuthology.orchestra.run.smithi029:> sudo tar c -f - -C /var/log/ceph -- . 2024-01-21T11:51:02.023 DEBUG:teuthology.misc:Transferring archived files from smithi134:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-01-19_23:17:02-rados-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7524471/remote/ubuntu@smithi134.front.sepia.ceph.com/log 2024-01-21T11:51:02.024 DEBUG:teuthology.orchestra.run.smithi134:> sudo tar c -f - -C /var/log/ceph -- . 2024-01-21T11:51:02.056 INFO:tasks.cephadm:Removing cluster... 2024-01-21T11:51:02.056 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e4084074-b851-11ee-95b0-87774f69a715 --force 2024-01-21T11:52:29.855 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e4084074-b851-11ee-95b0-87774f69a715 --force 2024-01-21T11:52:30.158 INFO:tasks.cephadm:Removing cephadm ... 2024-01-21T11:52:30.159 DEBUG:teuthology.orchestra.run.smithi029:> rm -rf /home/ubuntu/cephtest/cephadm 2024-01-21T11:52:30.178 DEBUG:teuthology.orchestra.run.smithi134:> rm -rf /home/ubuntu/cephtest/cephadm 2024-01-21T11:52:30.197 INFO:tasks.cephadm:Teardown complete 2024-01-21T11:52:30.197 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi134 with status 5: 'sudo systemctl stop ceph-e4084074-b851-11ee-95b0-87774f69a715@mon.smithi134' 2024-01-21T11:52:30.438 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=6980118e12e044099c316153475885e4 Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi134 with status 5: 'sudo systemctl stop ceph-e4084074-b851-11ee-95b0-87774f69a715@mon.smithi134' 2024-01-21T11:52:30.443 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-01-21T11:52:30.463 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-01-21T11:52:30.472 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-01-21T11:52:30.472 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-01-21T11:52:30.487 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-01-21T11:52:30.562 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-01-21T11:52:30.572 INFO:teuthology.task.clock:Checking final clock skew... 2024-01-21T11:52:30.573 DEBUG:teuthology.orchestra.run.smithi029:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-21T11:52:30.576 DEBUG:teuthology.orchestra.run.smithi134:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-21T11:52:30.592 INFO:teuthology.orchestra.run.smithi029.stderr:bash: ntpq: command not found 2024-01-21T11:52:30.593 INFO:teuthology.orchestra.run.smithi134.stderr:bash: ntpq: command not found 2024-01-21T11:52:30.596 INFO:teuthology.orchestra.run.smithi029.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-21T11:52:30.596 INFO:teuthology.orchestra.run.smithi029.stdout:=============================================================================== 2024-01-21T11:52:30.596 INFO:teuthology.orchestra.run.smithi029.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 38 -60us[ -60us] +/- 96ms 2024-01-21T11:52:30.597 INFO:teuthology.orchestra.run.smithi029.stdout:^* hv02.front.sepia.ceph.com 3 6 377 38 -821us[ -799us] +/- 60ms 2024-01-21T11:52:30.597 INFO:teuthology.orchestra.run.smithi029.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 37 +1068us[+1068us] +/- 79ms 2024-01-21T11:52:30.597 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T11:52:30.598 INFO:teuthology.orchestra.run.smithi134.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-21T11:52:30.598 INFO:teuthology.orchestra.run.smithi134.stdout:=============================================================================== 2024-01-21T11:52:30.599 INFO:teuthology.orchestra.run.smithi134.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 101 -26us[ -31us] +/- 95ms 2024-01-21T11:52:30.599 INFO:teuthology.orchestra.run.smithi134.stdout:^* hv02.front.sepia.ceph.com 3 6 377 37 -785us[ -790us] +/- 60ms 2024-01-21T11:52:30.599 INFO:teuthology.orchestra.run.smithi134.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 34 +1098us[+1098us] +/- 80ms 2024-01-21T11:52:30.599 INFO:teuthology.orchestra.run.smithi134.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T11:52:30.599 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-01-21T11:52:30.608 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-01-21T11:52:30.609 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-01-21T11:52:30.617 DEBUG:teuthology.orchestra.run.smithi029:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-01-21T11:52:30.704 DEBUG:teuthology.orchestra.run.smithi134:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-01-21T11:52:30.768 DEBUG:teuthology.orchestra.run.smithi029:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-21T11:52:30.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-21T11:52:30.800 DEBUG:teuthology.orchestra.run.smithi134:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-21T11:52:30.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-21T11:52:30.829 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-01-21T11:52:30.839 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-01-21T11:52:30.849 INFO:teuthology.task.internal:Duration was 1624.774534 seconds 2024-01-21T11:52:30.849 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-01-21T11:52:30.858 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-01-21T11:52:30.859 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-01-21T11:52:30.861 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-01-21T11:52:30.907 INFO:teuthology.orchestra.run.smithi029.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-21T11:52:30.915 INFO:teuthology.orchestra.run.smithi134.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-21T11:52:31.310 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-01-21T11:52:31.310 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi029.front.sepia.ceph.com 2024-01-21T11:52:31.310 DEBUG:teuthology.orchestra.run.smithi029:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-01-21T11:52:31.335 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi134.front.sepia.ceph.com 2024-01-21T11:52:31.335 DEBUG:teuthology.orchestra.run.smithi134:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-01-21T11:52:31.357 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-01-21T11:52:31.358 DEBUG:teuthology.orchestra.run.smithi029:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-01-21T11:52:31.378 DEBUG:teuthology.orchestra.run.smithi134:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-01-21T11:52:31.446 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-01-21T11:52:31.446 DEBUG:teuthology.orchestra.run.smithi029:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-01-21T11:52:31.448 DEBUG:teuthology.orchestra.run.smithi134:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-01-21T11:52:31.556 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-01-21T11:52:31.567 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-01-21T11:52:31.567 DEBUG:teuthology.orchestra.run.smithi029:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-01-21T11:52:31.595 DEBUG:teuthology.orchestra.run.smithi134:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-01-21T11:52:31.622 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-01-21T11:52:31.633 DEBUG:teuthology.orchestra.run.smithi029:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-01-21T11:52:31.637 DEBUG:teuthology.orchestra.run.smithi134:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-01-21T11:52:31.662 INFO:teuthology.orchestra.run.smithi029.stdout:kernel.core_pattern = core 2024-01-21T11:52:31.689 INFO:teuthology.orchestra.run.smithi134.stdout:kernel.core_pattern = core 2024-01-21T11:52:31.731 DEBUG:teuthology.orchestra.run.smithi029:> test -e /home/ubuntu/cephtest/archive/coredump 2024-01-21T11:52:31.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-21T11:52:31.749 DEBUG:teuthology.orchestra.run.smithi134:> test -e /home/ubuntu/cephtest/archive/coredump 2024-01-21T11:52:31.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-21T11:52:31.789 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-01-21T11:52:31.800 INFO:teuthology.task.internal:Transferring archived files... 2024-01-21T11:52:31.803 DEBUG:teuthology.misc:Transferring archived files from smithi029:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-01-19_23:17:02-rados-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7524471/remote/smithi029 2024-01-21T11:52:31.804 DEBUG:teuthology.orchestra.run.smithi029:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-01-21T11:52:31.844 DEBUG:teuthology.misc:Transferring archived files from smithi134:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-01-19_23:17:02-rados-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7524471/remote/smithi134 2024-01-21T11:52:31.845 DEBUG:teuthology.orchestra.run.smithi134:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-01-21T11:52:31.883 INFO:teuthology.task.internal:Removing archive directory... 2024-01-21T11:52:31.884 DEBUG:teuthology.orchestra.run.smithi029:> rm -rf -- /home/ubuntu/cephtest/archive 2024-01-21T11:52:31.886 DEBUG:teuthology.orchestra.run.smithi134:> rm -rf -- /home/ubuntu/cephtest/archive 2024-01-21T11:52:31.935 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-01-21T11:52:31.947 INFO:teuthology.task.internal:Not uploading archives. 2024-01-21T11:52:31.947 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-01-21T11:52:31.958 INFO:teuthology.task.internal:Tidying up after the test... 2024-01-21T11:52:31.958 DEBUG:teuthology.orchestra.run.smithi029:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-01-21T11:52:31.961 DEBUG:teuthology.orchestra.run.smithi134:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-01-21T11:52:31.976 INFO:teuthology.orchestra.run.smithi029.stdout: 525360 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 21 11:52 /home/ubuntu/cephtest 2024-01-21T11:52:31.992 INFO:teuthology.orchestra.run.smithi134.stdout: 393224 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 21 11:52 /home/ubuntu/cephtest 2024-01-21T11:52:31.994 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-01-21T11:52:32.004 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-01-21T11:52:32.027 INFO:teuthology.nuke:Checking targets against current locks 2024-01-21T11:52:32.055 DEBUG:teuthology.nuke:shortname: smithi029 2024-01-21T11:52:32.056 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-21T11:52:32.080 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi029.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-19_23:17:02-rados-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7524471', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-21 11:09:23.441867', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAD9H+axVWmNAwZmUhyVHLDor2Zf4cNXo2bmuMxs2KDYnRaC8FwwIcOBWZqZHujp45PlN2c76oZOawacymQMAss='} 2024-01-21T11:52:32.083 DEBUG:teuthology.nuke:shortname: smithi134 2024-01-21T11:52:32.083 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-21T11:52:32.105 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi134.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-19_23:17:02-rados-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7524471', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-21 11:09:23.439875', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCiE3WmRvczd1PZH1a5QZEFf7ivJikogqRtkcplD6WARPP77b52Dqv/lgqxx2Urh+amBO3Qi1exgJcImtO7Vhnw='} 2024-01-21T11:52:32.138 INFO:teuthology.orchestra.console.smithi029:Power off 2024-01-21T11:52:32.139 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-21T11:52:32.165 DEBUG:teuthology.orchestra.console.smithi029:power off output: Chassis Power Control: Down/Off 2024-01-21T11:52:32.166 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-21T11:52:32.181 INFO:teuthology.orchestra.console.smithi134:Power off 2024-01-21T11:52:32.181 DEBUG:teuthology.orchestra.console.smithi134:pexpect command: ipmitool -H smithi134.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-21T11:52:32.191 DEBUG:teuthology.orchestra.console.smithi029:check power output: Chassis Power is on 2024-01-21T11:52:32.215 DEBUG:teuthology.orchestra.console.smithi134:power off output: Chassis Power Control: Down/Off 2024-01-21T11:52:32.216 DEBUG:teuthology.orchestra.console.smithi134:pexpect command: ipmitool -H smithi134.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-21T11:52:32.241 DEBUG:teuthology.orchestra.console.smithi134:check power output: Chassis Power is on 2024-01-21T11:52:36.192 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-21T11:52:36.242 DEBUG:teuthology.orchestra.console.smithi134:pexpect command: ipmitool -H smithi134.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-21T11:52:36.303 DEBUG:teuthology.orchestra.console.smithi029:check power output: Chassis Power is on 2024-01-21T11:52:36.351 DEBUG:teuthology.orchestra.console.smithi134:check power output: Chassis Power is on 2024-01-21T11:52:40.308 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-21T11:52:40.352 DEBUG:teuthology.orchestra.console.smithi134:pexpect command: ipmitool -H smithi134.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-21T11:52:40.418 DEBUG:teuthology.orchestra.console.smithi029:check power output: Chassis Power is off 2024-01-21T11:52:40.462 DEBUG:teuthology.orchestra.console.smithi134:check power output: Chassis Power is on 2024-01-21T11:52:40.519 INFO:teuthology.orchestra.console.smithi029:Power off completed 2024-01-21T11:52:44.467 DEBUG:teuthology.orchestra.console.smithi134:pexpect command: ipmitool -H smithi134.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-21T11:52:44.576 DEBUG:teuthology.orchestra.console.smithi134:check power output: Chassis Power is off 2024-01-21T11:52:44.676 INFO:teuthology.orchestra.console.smithi134:Power off completed 2024-01-21T11:52:44.781 INFO:teuthology.run:Summary data: description: rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} duration: 1624.7745342254639 failure_reason: 'Command failed on smithi134 with status 5: ''sudo systemctl stop ceph-e4084074-b851-11ee-95b0-87774f69a715@mon.smithi134''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=6980118e12e044099c316153475885e4 status: fail success: false 2024-01-21T11:52:44.782 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-21T11:52:44.866 INFO:teuthology.run:FAIL