2023-11-15T00:42:01.613 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2 2023-11-15T00:42:01.614 DEBUG:teuthology.run:Teuthology command: teuthology --name yuriw-2023-11-14_20:31:57-fs-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi --archive /home/teuthworker/archive/yuriw-2023-11-14_20:31:57-fs-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458065 --owner scheduled_yuriw@teuthology --description fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} --verbose -- /home/teuthworker/archive/yuriw-2023-11-14_20:31:57-fs-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458065/orig.config.yaml 2023-11-15T00:42:01.664 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-15T00:42:01.751 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-11-14_20:31:57-fs-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458065 branch: wip-yuri4-testing-2023-11-13-0820-pacific description: fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} email: yweinste@redhat.com first_in_suite: false job_id: '7458065' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/pacific v16.2.4 ' name: yuriw-2023-11-14_20:31:57-fs-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri4-testing-2023-11-13-0820-pacific ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: 620f25c1794f62207424fc9319427943fb8d1fae ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 620f25c1794f62207424fc9319427943fb8d1fae selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri4-testing-2023-11-13-0820-pacific sha1: 620f25c1794f62207424fc9319427943fb8d1fae owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 5345 sha1: 620f25c1794f62207424fc9319427943fb8d1fae sleep_before_teardown: 0 subset: 111/12000 suite: fs suite_branch: wip-yuri4-testing-2023-11-13-0820-pacific suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 620f25c1794f62207424fc9319427943fb8d1fae targets: smithi055.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFCxUHNBYaiwQjldCEjzagkMP60k+HLdxkSQGUDsAFu627DaiMoEbdnt9Z2pS3+2DNyC7wQptSUKwW9Jh71uKSk= smithi137.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFlLiM0EFGm1UTU66Mp+CYSdELpNyOuOJ19ajyibr4CZtpACk50se3zF6/qrlzSGY3wrTgLwgoGnO6gOn23R/zE= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - pexec: clients: - sudo modprobe -r ceph - sudo modprobe ceph disable_send_metrics=on - install: exclude_packages: - ceph-volume tag: v16.2.4 - print: '**** done install task...' - cephadm: avoid_pacific_features: true cephadm_branch: v16.2.4 cephadm_git_url: https://github.com/ceph/ceph image: docker.io/ceph/ceph:v16.2.4 roleless: true - print: '**** done starting v16.2.4' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 2 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay true - cephadm.shell: host.a: - ceph fs set cephfs inline_data false - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - kclient: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 6899cd26fceddb2fec83dc1a1349394b28c8998e timestamp: 2023-11-14_20:31:57 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3590714 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2023-11-15T00:42:01.752 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa; will attempt to use it 2023-11-15T00:42:01.752 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks 2023-11-15T00:42:01.753 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-11-15T00:42:01.755 INFO:teuthology.task.internal:Checking packages... 2023-11-15T00:42:01.780 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '620f25c1794f62207424fc9319427943fb8d1fae' 2023-11-15T00:42:01.780 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-11-15T00:42:01.781 INFO:teuthology.packaging:ref: None 2023-11-15T00:42:01.781 INFO:teuthology.packaging:tag: None 2023-11-15T00:42:01.781 INFO:teuthology.packaging:branch: wip-yuri4-testing-2023-11-13-0820-pacific 2023-11-15T00:42:01.782 INFO:teuthology.packaging:sha1: 620f25c1794f62207424fc9319427943fb8d1fae 2023-11-15T00:42:01.782 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri4-testing-2023-11-13-0820-pacific 2023-11-15T00:42:02.000 INFO:teuthology.task.internal:Found packages for ceph version 16.2.14-170.g620f25c1 2023-11-15T00:42:02.001 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-11-15T00:42:02.007 INFO:teuthology.task.internal:no buildpackages task found 2023-11-15T00:42:02.007 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-11-15T00:42:02.064 INFO:teuthology.task.internal:Saving configuration 2023-11-15T00:42:02.083 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-11-15T00:42:02.106 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-15T00:42:02.133 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi055.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-14_20:31:57-fs-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458065', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-15 00:37:03.376529', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFCxUHNBYaiwQjldCEjzagkMP60k+HLdxkSQGUDsAFu627DaiMoEbdnt9Z2pS3+2DNyC7wQptSUKwW9Jh71uKSk='} 2023-11-15T00:42:02.155 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi137.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-14_20:31:57-fs-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458065', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-15 00:37:03.374892', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFlLiM0EFGm1UTU66Mp+CYSdELpNyOuOJ19ajyibr4CZtpACk50se3zF6/qrlzSGY3wrTgLwgoGnO6gOn23R/zE='} 2023-11-15T00:42:02.155 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-11-15T00:42:02.161 INFO:teuthology.task.internal:roles: ubuntu@smithi055.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2023-11-15T00:42:02.162 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2023-11-15T00:42:02.162 INFO:teuthology.run_tasks:Running task console_log... 2023-11-15T00:42:02.289 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f068e7cdb80>, signals=[15]) 2023-11-15T00:42:02.291 INFO:teuthology.run_tasks:Running task internal.connect... 2023-11-15T00:42:02.297 INFO:teuthology.task.internal:Opening connections... 2023-11-15T00:42:02.298 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi055.front.sepia.ceph.com 2023-11-15T00:42:02.300 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi055.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-15T00:42:02.370 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi137.front.sepia.ceph.com 2023-11-15T00:42:02.371 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-15T00:42:02.439 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-11-15T00:42:02.445 DEBUG:teuthology.orchestra.run.smithi055:> uname -m 2023-11-15T00:42:02.468 INFO:teuthology.orchestra.run.smithi055.stdout:x86_64 2023-11-15T00:42:02.468 DEBUG:teuthology.orchestra.run.smithi055:> cat /etc/os-release 2023-11-15T00:42:02.530 INFO:teuthology.orchestra.run.smithi055.stdout:NAME="CentOS Stream" 2023-11-15T00:42:02.531 INFO:teuthology.orchestra.run.smithi055.stdout:VERSION="8" 2023-11-15T00:42:02.531 INFO:teuthology.orchestra.run.smithi055.stdout:ID="centos" 2023-11-15T00:42:02.531 INFO:teuthology.orchestra.run.smithi055.stdout:ID_LIKE="rhel fedora" 2023-11-15T00:42:02.531 INFO:teuthology.orchestra.run.smithi055.stdout:VERSION_ID="8" 2023-11-15T00:42:02.531 INFO:teuthology.orchestra.run.smithi055.stdout:PLATFORM_ID="platform:el8" 2023-11-15T00:42:02.532 INFO:teuthology.orchestra.run.smithi055.stdout:PRETTY_NAME="CentOS Stream 8" 2023-11-15T00:42:02.532 INFO:teuthology.orchestra.run.smithi055.stdout:ANSI_COLOR="0;31" 2023-11-15T00:42:02.532 INFO:teuthology.orchestra.run.smithi055.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-11-15T00:42:02.532 INFO:teuthology.orchestra.run.smithi055.stdout:HOME_URL="https://centos.org/" 2023-11-15T00:42:02.532 INFO:teuthology.orchestra.run.smithi055.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-11-15T00:42:02.533 INFO:teuthology.orchestra.run.smithi055.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-11-15T00:42:02.533 INFO:teuthology.orchestra.run.smithi055.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-11-15T00:42:02.533 INFO:teuthology.lock.ops:Updating smithi055.front.sepia.ceph.com on lock server 2023-11-15T00:42:02.555 DEBUG:teuthology.orchestra.run.smithi137:> uname -m 2023-11-15T00:42:02.578 INFO:teuthology.orchestra.run.smithi137.stdout:x86_64 2023-11-15T00:42:02.579 DEBUG:teuthology.orchestra.run.smithi137:> cat /etc/os-release 2023-11-15T00:42:02.639 INFO:teuthology.orchestra.run.smithi137.stdout:NAME="CentOS Stream" 2023-11-15T00:42:02.639 INFO:teuthology.orchestra.run.smithi137.stdout:VERSION="8" 2023-11-15T00:42:02.639 INFO:teuthology.orchestra.run.smithi137.stdout:ID="centos" 2023-11-15T00:42:02.640 INFO:teuthology.orchestra.run.smithi137.stdout:ID_LIKE="rhel fedora" 2023-11-15T00:42:02.640 INFO:teuthology.orchestra.run.smithi137.stdout:VERSION_ID="8" 2023-11-15T00:42:02.640 INFO:teuthology.orchestra.run.smithi137.stdout:PLATFORM_ID="platform:el8" 2023-11-15T00:42:02.640 INFO:teuthology.orchestra.run.smithi137.stdout:PRETTY_NAME="CentOS Stream 8" 2023-11-15T00:42:02.640 INFO:teuthology.orchestra.run.smithi137.stdout:ANSI_COLOR="0;31" 2023-11-15T00:42:02.641 INFO:teuthology.orchestra.run.smithi137.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-11-15T00:42:02.641 INFO:teuthology.orchestra.run.smithi137.stdout:HOME_URL="https://centos.org/" 2023-11-15T00:42:02.641 INFO:teuthology.orchestra.run.smithi137.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-11-15T00:42:02.641 INFO:teuthology.orchestra.run.smithi137.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-11-15T00:42:02.641 INFO:teuthology.orchestra.run.smithi137.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-11-15T00:42:02.642 INFO:teuthology.lock.ops:Updating smithi137.front.sepia.ceph.com on lock server 2023-11-15T00:42:02.663 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-11-15T00:42:02.672 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-11-15T00:42:02.738 INFO:teuthology.task.internal:Checking for old test directory... 2023-11-15T00:42:02.739 DEBUG:teuthology.orchestra.run.smithi055:> test '!' -e /home/ubuntu/cephtest 2023-11-15T00:42:02.742 DEBUG:teuthology.orchestra.run.smithi137:> test '!' -e /home/ubuntu/cephtest 2023-11-15T00:42:02.763 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-11-15T00:42:02.798 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-11-15T00:42:02.799 DEBUG:teuthology.orchestra.run.smithi055:> test -z $(ls -A /var/lib/ceph) 2023-11-15T00:42:02.803 DEBUG:teuthology.orchestra.run.smithi137:> test -z $(ls -A /var/lib/ceph) 2023-11-15T00:42:02.836 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-11-15T00:42:02.890 INFO:teuthology.run_tasks:Running task kernel... 2023-11-15T00:42:02.916 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-11-15T00:42:02.917 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-11-15T00:42:02.918 DEBUG:teuthology.orchestra.run.smithi055:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-15T00:42:02.919 DEBUG:teuthology.orchestra.run.smithi137:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-15T00:42:02.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-15T00:42:02.941 DEBUG:teuthology.orchestra.run.smithi137:> uname -r 2023-11-15T00:42:02.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-15T00:42:02.943 DEBUG:teuthology.orchestra.run.smithi055:> uname -r 2023-11-15T00:42:03.002 INFO:teuthology.orchestra.run.smithi137.stdout:4.18.0-489.el8.x86_64 2023-11-15T00:42:03.003 INFO:teuthology.task.kernel:Running kernel on smithi137: 4.18.0-489.el8.x86_64 2023-11-15T00:42:03.004 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum install -y kernel 2023-11-15T00:42:03.005 INFO:teuthology.orchestra.run.smithi055.stdout:4.18.0-489.el8.x86_64 2023-11-15T00:42:03.006 INFO:teuthology.task.kernel:Running kernel on smithi055: 4.18.0-489.el8.x86_64 2023-11-15T00:42:03.006 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum install -y kernel 2023-11-15T00:50:28.870 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS Stream 8 - AppStream 69 kB/s | 34 MB 08:23 2023-11-15T00:50:37.745 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS Stream 8 - BaseOS 23 MB/s | 53 MB 00:02 2023-11-15T00:50:46.302 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS Stream 8 - Extras 44 kB/s | 18 kB 00:00 2023-11-15T00:50:46.703 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS Stream 8 - Extras common packages 34 kB/s | 6.9 kB 00:00 2023-11-15T00:50:47.030 INFO:teuthology.orchestra.run.smithi055.stdout:Copr repo for python3-asyncssh owned by ceph 22 kB/s | 3.5 kB 00:00 2023-11-15T00:50:48.366 INFO:teuthology.orchestra.run.smithi055.stdout:Extra Packages for Enterprise Linux 14 MB/s | 16 MB 00:01 2023-11-15T00:50:52.397 INFO:teuthology.orchestra.run.smithi055.stdout:lab-extras 399 kB/s | 24 kB 00:00 2023-11-15T00:50:56.222 INFO:teuthology.orchestra.run.smithi055.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-11-15T00:50:56.222 INFO:teuthology.orchestra.run.smithi055.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-11-15T00:50:56.508 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2023-11-15T00:50:56.509 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-15T00:50:56.509 INFO:teuthology.orchestra.run.smithi055.stdout: Package Architecture Version Repository Size 2023-11-15T00:50:56.509 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-15T00:50:56.509 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2023-11-15T00:50:56.510 INFO:teuthology.orchestra.run.smithi055.stdout: kernel x86_64 4.18.0-521.el8 baseos 10 M 2023-11-15T00:50:56.510 INFO:teuthology.orchestra.run.smithi055.stdout:Installing dependencies: 2023-11-15T00:50:56.510 INFO:teuthology.orchestra.run.smithi055.stdout: kernel-core x86_64 4.18.0-521.el8 baseos 43 M 2023-11-15T00:50:56.510 INFO:teuthology.orchestra.run.smithi055.stdout: kernel-modules x86_64 4.18.0-521.el8 baseos 36 M 2023-11-15T00:50:56.510 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-15T00:50:56.511 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2023-11-15T00:50:56.511 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-15T00:50:56.511 INFO:teuthology.orchestra.run.smithi055.stdout:Install 3 Packages 2023-11-15T00:50:56.511 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-15T00:50:56.512 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 89 M 2023-11-15T00:50:56.512 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 96 M 2023-11-15T00:50:56.512 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2023-11-15T00:50:57.740 INFO:teuthology.orchestra.run.smithi055.stdout:(1/3): kernel-4.18.0-521.el8.x86_64.rpm 8.4 MB/s | 10 MB 00:01 2023-11-15T00:51:00.348 INFO:teuthology.orchestra.run.smithi055.stdout:(2/3): kernel-core-4.18.0-521.el8.x86_64.rpm 11 MB/s | 43 MB 00:03 2023-11-15T00:51:00.757 INFO:teuthology.orchestra.run.smithi055.stdout:(3/3): kernel-modules-4.18.0-521.el8.x86_64.rpm 8.6 MB/s | 36 MB 00:04 2023-11-15T00:51:00.757 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2023-11-15T00:51:00.758 INFO:teuthology.orchestra.run.smithi055.stdout:Total 21 MB/s | 89 MB 00:04 2023-11-15T00:51:01.461 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2023-11-15T00:51:01.910 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2023-11-15T00:51:01.910 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2023-11-15T00:51:03.208 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2023-11-15T00:51:03.235 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2023-11-15T00:51:06.112 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2023-11-15T00:51:06.667 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : kernel-core-4.18.0-521.el8.x86_64 1/3 2023-11-15T00:51:10.435 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: kernel-core-4.18.0-521.el8.x86_64 1/3 2023-11-15T00:51:10.808 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : kernel-modules-4.18.0-521.el8.x86_64 2/3 2023-11-15T00:51:14.483 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: kernel-modules-4.18.0-521.el8.x86_64 2/3 2023-11-15T00:51:14.774 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : kernel-4.18.0-521.el8.x86_64 3/3 2023-11-15T00:51:38.092 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: kernel-core-4.18.0-521.el8.x86_64 3/3 2023-11-15T00:51:38.116 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: kernel-modules-4.18.0-521.el8.x86_64 3/3 2023-11-15T00:51:38.482 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: kernel-4.18.0-521.el8.x86_64 3/3 2023-11-15T00:51:38.482 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : kernel-4.18.0-521.el8.x86_64 1/3 2023-11-15T00:51:38.482 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : kernel-core-4.18.0-521.el8.x86_64 2/3 2023-11-15T00:51:39.310 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : kernel-modules-4.18.0-521.el8.x86_64 3/3 2023-11-15T00:51:39.311 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-15T00:51:39.311 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2023-11-15T00:51:39.311 INFO:teuthology.orchestra.run.smithi055.stdout: kernel-4.18.0-521.el8.x86_64 kernel-core-4.18.0-521.el8.x86_64 2023-11-15T00:51:39.311 INFO:teuthology.orchestra.run.smithi055.stdout: kernel-modules-4.18.0-521.el8.x86_64 2023-11-15T00:51:39.312 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-15T00:51:39.312 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2023-11-15T00:51:39.456 DEBUG:teuthology.orchestra.run.smithi055:> rpm -q kernel | sort -rV | head -n 1 2023-11-15T00:51:39.724 INFO:teuthology.orchestra.run.smithi055.stdout:kernel-4.18.0-521.el8.x86_64 2023-11-15T00:51:39.724 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-521.el8.x86_64 2023-11-15T00:51:39.725 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-521.el8.x86_64 2023-11-15T00:51:39.725 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-11-15T00:51:39.725 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-11-15T00:51:39.725 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2023-11-15T00:51:39.726 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi055.front.sepia.ceph.com, path=None, version=distro) 2023-11-15T00:51:39.726 DEBUG:teuthology.orchestra.run.smithi055:> rpm -q kernel | sort -rV | head -n 1 2023-11-15T00:51:39.945 INFO:teuthology.orchestra.run.smithi055.stdout:kernel-4.18.0-521.el8.x86_64 2023-11-15T00:51:39.946 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-521.el8.x86_64 2023-11-15T00:51:39.946 DEBUG:teuthology.orchestra.run.smithi055:> sudo rpm -qi grub2-tools 2023-11-15T00:51:39.982 INFO:teuthology.orchestra.run.smithi055.stdout:Name : grub2-tools 2023-11-15T00:51:39.983 INFO:teuthology.orchestra.run.smithi055.stdout:Epoch : 1 2023-11-15T00:51:39.983 INFO:teuthology.orchestra.run.smithi055.stdout:Version : 2.02 2023-11-15T00:51:39.983 INFO:teuthology.orchestra.run.smithi055.stdout:Release : 129.el8 2023-11-15T00:51:39.983 INFO:teuthology.orchestra.run.smithi055.stdout:Architecture: x86_64 2023-11-15T00:51:39.983 INFO:teuthology.orchestra.run.smithi055.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-11-15T00:51:39.984 INFO:teuthology.orchestra.run.smithi055.stdout:Group : System Environment/Base 2023-11-15T00:51:39.984 INFO:teuthology.orchestra.run.smithi055.stdout:Size : 9265586 2023-11-15T00:51:39.984 INFO:teuthology.orchestra.run.smithi055.stdout:License : GPLv3+ 2023-11-15T00:51:39.984 INFO:teuthology.orchestra.run.smithi055.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-11-15T00:51:39.984 INFO:teuthology.orchestra.run.smithi055.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-11-15T00:51:39.985 INFO:teuthology.orchestra.run.smithi055.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-11-15T00:51:39.985 INFO:teuthology.orchestra.run.smithi055.stdout:Build Host : kbuilder.bsys.centos.org 2023-11-15T00:51:39.985 INFO:teuthology.orchestra.run.smithi055.stdout:Relocations : (not relocatable) 2023-11-15T00:51:39.985 INFO:teuthology.orchestra.run.smithi055.stdout:Packager : CentOS BuildSystem 2023-11-15T00:51:39.985 INFO:teuthology.orchestra.run.smithi055.stdout:Vendor : CentOS 2023-11-15T00:51:39.986 INFO:teuthology.orchestra.run.smithi055.stdout:URL : http://www.gnu.org/software/grub/ 2023-11-15T00:51:39.986 INFO:teuthology.orchestra.run.smithi055.stdout:Summary : Support tools for GRUB. 2023-11-15T00:51:39.986 INFO:teuthology.orchestra.run.smithi055.stdout:Description : 2023-11-15T00:51:39.986 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-15T00:51:39.987 INFO:teuthology.orchestra.run.smithi055.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-11-15T00:51:39.987 INFO:teuthology.orchestra.run.smithi055.stdout:customizable bootloader with modular architecture. It supports a rich 2023-11-15T00:51:39.987 INFO:teuthology.orchestra.run.smithi055.stdout:variety of kernel formats, file systems, computer architectures and 2023-11-15T00:51:39.987 INFO:teuthology.orchestra.run.smithi055.stdout:hardware devices. 2023-11-15T00:51:39.987 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-15T00:51:39.988 INFO:teuthology.orchestra.run.smithi055.stdout:This subpackage provides tools for support of all platforms. 2023-11-15T00:51:39.988 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-11-15T00:51:39.989 INFO:teuthology.task.kernel:Updating grub on smithi055 to boot 4.18.0-521.el8.x86_64 2023-11-15T00:51:39.989 DEBUG:teuthology.orchestra.run.smithi055:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-11-15T00:51:41.379 INFO:teuthology.orchestra.run.smithi055.stderr:Generating grub configuration file ... 2023-11-15T00:51:43.434 INFO:teuthology.orchestra.run.smithi055.stderr:done 2023-11-15T00:51:43.435 DEBUG:teuthology.orchestra.run.smithi055:> mktemp 2023-11-15T00:51:43.467 INFO:teuthology.orchestra.run.smithi055.stdout:/tmp/tmp.sT0V2Bsp3z 2023-11-15T00:51:43.467 DEBUG:teuthology.orchestra.run.smithi055:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.sT0V2Bsp3z 2023-11-15T00:51:43.540 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod 0666 /tmp/tmp.sT0V2Bsp3z 2023-11-15T00:51:43.686 DEBUG:teuthology.orchestra.remote:smithi055:/tmp/tmp.sT0V2Bsp3z is 6KB 2023-11-15T00:51:43.735 DEBUG:teuthology.orchestra.run.smithi055:> rm -fr /tmp/tmp.sT0V2Bsp3z 2023-11-15T00:51:43.753 DEBUG:teuthology.orchestra.run.smithi055:> sudo /bin/ls /boot/loader/entries || true 2023-11-15T00:51:43.819 INFO:teuthology.orchestra.run.smithi055.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-11-15T00:51:43.820 INFO:teuthology.orchestra.run.smithi055.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-11-15T00:51:43.821 INFO:teuthology.orchestra.run.smithi055.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-11-15T00:51:43.821 INFO:teuthology.orchestra.run.smithi055.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-521.el8.x86_64.conf 2023-11-15T00:51:43.823 DEBUG:teuthology.orchestra.run.smithi055:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-521.el8.x86_64 2023-11-15T00:51:43.995 DEBUG:teuthology.orchestra.run.smithi055:> sudo shutdown -r now 2023-11-15T00:52:13.998 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2023-11-15T00:52:13.998 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi055.front.sepia.ceph.com' 2023-11-15T00:52:13.999 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi055.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-15T00:53:14.001 DEBUG:teuthology.orchestra.remote:timed out 2023-11-15T00:53:23.003 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi055.front.sepia.ceph.com' 2023-11-15T00:53:23.004 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi055.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-15T00:53:24.305 DEBUG:teuthology.orchestra.run.smithi055:> true 2023-11-15T00:53:24.841 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi055.front.sepia.ceph.com' 2023-11-15T00:53:24.842 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "4.18.0-521.el8.x86_64"... 2023-11-15T00:53:24.842 DEBUG:teuthology.orchestra.run.smithi055:> uname -r 2023-11-15T00:53:24.864 INFO:teuthology.orchestra.run.smithi055.stdout:4.18.0-521.el8.x86_64 2023-11-15T00:53:24.865 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-521.el8.x86_64 vs 4.18.0-521.el8.x86_64 2023-11-15T00:53:24.865 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-11-15T00:53:24.865 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-11-15T00:53:25.867 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2023-11-15T00:53:25.867 DEBUG:teuthology.orchestra.run.smithi055:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-11-15T00:53:25.939 INFO:teuthology.orchestra.run.smithi055.stdout:ttyS1 2023-11-15T00:53:25.956 DEBUG:teuthology.parallel:result is None 2023-11-15T01:04:03.459 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 8 - AppStream 26 kB/s | 34 MB 21:57 2023-11-15T01:04:12.000 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 8 - BaseOS 22 MB/s | 53 MB 00:02 2023-11-15T01:04:19.966 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 8 - Extras 40 kB/s | 18 kB 00:00 2023-11-15T01:04:20.500 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 8 - Extras common packages 20 kB/s | 6.9 kB 00:00 2023-11-15T01:04:20.818 INFO:teuthology.orchestra.run.smithi137.stdout:Copr repo for python3-asyncssh owned by ceph 21 kB/s | 3.5 kB 00:00 2023-11-15T01:04:22.096 INFO:teuthology.orchestra.run.smithi137.stdout:Extra Packages for Enterprise Linux 14 MB/s | 16 MB 00:01 2023-11-15T01:04:25.878 INFO:teuthology.orchestra.run.smithi137.stdout:lab-extras 436 kB/s | 24 kB 00:00 2023-11-15T01:04:29.502 INFO:teuthology.orchestra.run.smithi137.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-11-15T01:04:29.502 INFO:teuthology.orchestra.run.smithi137.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-11-15T01:04:29.766 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2023-11-15T01:04:29.766 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T01:04:29.766 INFO:teuthology.orchestra.run.smithi137.stdout: Package Architecture Version Repository Size 2023-11-15T01:04:29.767 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T01:04:29.767 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2023-11-15T01:04:29.768 INFO:teuthology.orchestra.run.smithi137.stdout: kernel x86_64 4.18.0-521.el8 baseos 10 M 2023-11-15T01:04:29.768 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2023-11-15T01:04:29.768 INFO:teuthology.orchestra.run.smithi137.stdout: kernel-core x86_64 4.18.0-521.el8 baseos 43 M 2023-11-15T01:04:29.768 INFO:teuthology.orchestra.run.smithi137.stdout: kernel-modules x86_64 4.18.0-521.el8 baseos 36 M 2023-11-15T01:04:29.769 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:04:29.769 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2023-11-15T01:04:29.769 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T01:04:29.769 INFO:teuthology.orchestra.run.smithi137.stdout:Install 3 Packages 2023-11-15T01:04:29.770 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:04:29.770 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 89 M 2023-11-15T01:04:29.771 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 96 M 2023-11-15T01:04:29.771 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2023-11-15T01:04:36.195 INFO:teuthology.orchestra.run.smithi137.stdout:(1/3): kernel-4.18.0-521.el8.x86_64.rpm 1.6 MB/s | 10 MB 00:06 2023-11-15T01:04:42.145 INFO:teuthology.orchestra.run.smithi137.stdout:(2/3): kernel-modules-4.18.0-521.el8.x86_64.rpm 2.9 MB/s | 36 MB 00:12 2023-11-15T01:04:47.552 INFO:teuthology.orchestra.run.smithi137.stdout:(3/3): kernel-core-4.18.0-521.el8.x86_64.rpm 2.4 MB/s | 43 MB 00:17 2023-11-15T01:04:47.553 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2023-11-15T01:04:47.554 INFO:teuthology.orchestra.run.smithi137.stdout:Total 5.0 MB/s | 89 MB 00:17 2023-11-15T01:04:48.233 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2023-11-15T01:04:48.663 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2023-11-15T01:04:48.663 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2023-11-15T01:04:49.878 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2023-11-15T01:04:49.908 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2023-11-15T01:04:52.681 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2023-11-15T01:04:53.129 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : kernel-core-4.18.0-521.el8.x86_64 1/3 2023-11-15T01:04:56.665 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: kernel-core-4.18.0-521.el8.x86_64 1/3 2023-11-15T01:04:57.262 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : kernel-modules-4.18.0-521.el8.x86_64 2/3 2023-11-15T01:05:00.693 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: kernel-modules-4.18.0-521.el8.x86_64 2/3 2023-11-15T01:05:00.949 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : kernel-4.18.0-521.el8.x86_64 3/3 2023-11-15T01:05:22.787 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: kernel-core-4.18.0-521.el8.x86_64 3/3 2023-11-15T01:05:22.811 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: kernel-modules-4.18.0-521.el8.x86_64 3/3 2023-11-15T01:05:23.157 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: kernel-4.18.0-521.el8.x86_64 3/3 2023-11-15T01:05:23.157 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : kernel-4.18.0-521.el8.x86_64 1/3 2023-11-15T01:05:23.157 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : kernel-core-4.18.0-521.el8.x86_64 2/3 2023-11-15T01:05:24.124 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : kernel-modules-4.18.0-521.el8.x86_64 3/3 2023-11-15T01:05:24.124 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:05:24.124 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2023-11-15T01:05:24.125 INFO:teuthology.orchestra.run.smithi137.stdout: kernel-4.18.0-521.el8.x86_64 kernel-core-4.18.0-521.el8.x86_64 2023-11-15T01:05:24.125 INFO:teuthology.orchestra.run.smithi137.stdout: kernel-modules-4.18.0-521.el8.x86_64 2023-11-15T01:05:24.125 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:05:24.125 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2023-11-15T01:05:24.289 DEBUG:teuthology.orchestra.run.smithi137:> rpm -q kernel | sort -rV | head -n 1 2023-11-15T01:05:24.505 INFO:teuthology.orchestra.run.smithi137.stdout:kernel-4.18.0-521.el8.x86_64 2023-11-15T01:05:24.506 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-521.el8.x86_64 2023-11-15T01:05:24.506 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-521.el8.x86_64 2023-11-15T01:05:24.506 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-11-15T01:05:24.506 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-11-15T01:05:24.507 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2023-11-15T01:05:24.507 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi137.front.sepia.ceph.com, path=None, version=distro) 2023-11-15T01:05:24.507 DEBUG:teuthology.orchestra.run.smithi137:> rpm -q kernel | sort -rV | head -n 1 2023-11-15T01:05:24.716 INFO:teuthology.orchestra.run.smithi137.stdout:kernel-4.18.0-521.el8.x86_64 2023-11-15T01:05:24.717 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-521.el8.x86_64 2023-11-15T01:05:24.717 DEBUG:teuthology.orchestra.run.smithi137:> sudo rpm -qi grub2-tools 2023-11-15T01:05:24.751 INFO:teuthology.orchestra.run.smithi137.stdout:Name : grub2-tools 2023-11-15T01:05:24.751 INFO:teuthology.orchestra.run.smithi137.stdout:Epoch : 1 2023-11-15T01:05:24.752 INFO:teuthology.orchestra.run.smithi137.stdout:Version : 2.02 2023-11-15T01:05:24.752 INFO:teuthology.orchestra.run.smithi137.stdout:Release : 129.el8 2023-11-15T01:05:24.752 INFO:teuthology.orchestra.run.smithi137.stdout:Architecture: x86_64 2023-11-15T01:05:24.752 INFO:teuthology.orchestra.run.smithi137.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-11-15T01:05:24.753 INFO:teuthology.orchestra.run.smithi137.stdout:Group : System Environment/Base 2023-11-15T01:05:24.753 INFO:teuthology.orchestra.run.smithi137.stdout:Size : 9265586 2023-11-15T01:05:24.753 INFO:teuthology.orchestra.run.smithi137.stdout:License : GPLv3+ 2023-11-15T01:05:24.753 INFO:teuthology.orchestra.run.smithi137.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-11-15T01:05:24.753 INFO:teuthology.orchestra.run.smithi137.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-11-15T01:05:24.754 INFO:teuthology.orchestra.run.smithi137.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-11-15T01:05:24.754 INFO:teuthology.orchestra.run.smithi137.stdout:Build Host : kbuilder.bsys.centos.org 2023-11-15T01:05:24.754 INFO:teuthology.orchestra.run.smithi137.stdout:Relocations : (not relocatable) 2023-11-15T01:05:24.754 INFO:teuthology.orchestra.run.smithi137.stdout:Packager : CentOS BuildSystem 2023-11-15T01:05:24.755 INFO:teuthology.orchestra.run.smithi137.stdout:Vendor : CentOS 2023-11-15T01:05:24.755 INFO:teuthology.orchestra.run.smithi137.stdout:URL : http://www.gnu.org/software/grub/ 2023-11-15T01:05:24.755 INFO:teuthology.orchestra.run.smithi137.stdout:Summary : Support tools for GRUB. 2023-11-15T01:05:24.755 INFO:teuthology.orchestra.run.smithi137.stdout:Description : 2023-11-15T01:05:24.755 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:05:24.756 INFO:teuthology.orchestra.run.smithi137.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-11-15T01:05:24.756 INFO:teuthology.orchestra.run.smithi137.stdout:customizable bootloader with modular architecture. It supports a rich 2023-11-15T01:05:24.756 INFO:teuthology.orchestra.run.smithi137.stdout:variety of kernel formats, file systems, computer architectures and 2023-11-15T01:05:24.756 INFO:teuthology.orchestra.run.smithi137.stdout:hardware devices. 2023-11-15T01:05:24.757 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:05:24.757 INFO:teuthology.orchestra.run.smithi137.stdout:This subpackage provides tools for support of all platforms. 2023-11-15T01:05:24.758 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-11-15T01:05:24.758 INFO:teuthology.task.kernel:Updating grub on smithi137 to boot 4.18.0-521.el8.x86_64 2023-11-15T01:05:24.758 DEBUG:teuthology.orchestra.run.smithi137:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-11-15T01:05:26.165 INFO:teuthology.orchestra.run.smithi137.stderr:Generating grub configuration file ... 2023-11-15T01:05:28.189 INFO:teuthology.orchestra.run.smithi137.stderr:done 2023-11-15T01:05:28.190 DEBUG:teuthology.orchestra.run.smithi137:> mktemp 2023-11-15T01:05:28.217 INFO:teuthology.orchestra.run.smithi137.stdout:/tmp/tmp.duVkeSiPEF 2023-11-15T01:05:28.218 DEBUG:teuthology.orchestra.run.smithi137:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.duVkeSiPEF 2023-11-15T01:05:28.291 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 0666 /tmp/tmp.duVkeSiPEF 2023-11-15T01:05:28.439 DEBUG:teuthology.orchestra.remote:smithi137:/tmp/tmp.duVkeSiPEF is 6KB 2023-11-15T01:05:28.461 DEBUG:teuthology.orchestra.run.smithi137:> rm -fr /tmp/tmp.duVkeSiPEF 2023-11-15T01:05:28.478 DEBUG:teuthology.orchestra.run.smithi137:> sudo /bin/ls /boot/loader/entries || true 2023-11-15T01:05:28.556 INFO:teuthology.orchestra.run.smithi137.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-11-15T01:05:28.556 INFO:teuthology.orchestra.run.smithi137.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-11-15T01:05:28.557 INFO:teuthology.orchestra.run.smithi137.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-11-15T01:05:28.557 INFO:teuthology.orchestra.run.smithi137.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-521.el8.x86_64.conf 2023-11-15T01:05:28.558 DEBUG:teuthology.orchestra.run.smithi137:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-521.el8.x86_64 2023-11-15T01:05:28.720 DEBUG:teuthology.orchestra.run.smithi137:> sudo shutdown -r now 2023-11-15T01:05:58.723 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2023-11-15T01:05:58.724 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2023-11-15T01:05:58.725 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-15T01:06:17.085 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.137 2023-11-15T01:06:26.086 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2023-11-15T01:06:26.087 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-15T01:07:26.090 DEBUG:teuthology.orchestra.remote:timed out 2023-11-15T01:07:38.093 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2023-11-15T01:07:38.094 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-15T01:07:38.472 DEBUG:teuthology.orchestra.run.smithi137:> true 2023-11-15T01:07:38.783 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi137.front.sepia.ceph.com' 2023-11-15T01:07:38.783 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "4.18.0-521.el8.x86_64"... 2023-11-15T01:07:38.784 DEBUG:teuthology.orchestra.run.smithi137:> uname -r 2023-11-15T01:07:38.803 INFO:teuthology.orchestra.run.smithi137.stdout:4.18.0-521.el8.x86_64 2023-11-15T01:07:38.804 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-521.el8.x86_64 vs 4.18.0-521.el8.x86_64 2023-11-15T01:07:38.804 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-11-15T01:07:38.804 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-11-15T01:07:39.805 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2023-11-15T01:07:39.806 DEBUG:teuthology.orchestra.run.smithi137:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-11-15T01:07:39.886 INFO:teuthology.orchestra.run.smithi137.stdout:ttyS1 2023-11-15T01:07:39.901 DEBUG:teuthology.parallel:result is None 2023-11-15T01:07:39.902 INFO:teuthology.run_tasks:Running task internal.base... 2023-11-15T01:07:39.909 INFO:teuthology.task.internal:Creating test directory... 2023-11-15T01:07:39.910 DEBUG:teuthology.orchestra.run.smithi055:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-15T01:07:39.914 DEBUG:teuthology.orchestra.run.smithi137:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-15T01:07:39.935 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-11-15T01:07:39.942 INFO:teuthology.run_tasks:Running task internal.archive... 2023-11-15T01:07:39.982 INFO:teuthology.task.internal:Creating archive directory... 2023-11-15T01:07:39.982 DEBUG:teuthology.orchestra.run.smithi055:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-15T01:07:39.985 DEBUG:teuthology.orchestra.run.smithi137:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-15T01:07:40.034 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-11-15T01:07:40.041 INFO:teuthology.task.internal:Enabling coredump saving... 2023-11-15T01:07:40.041 DEBUG:teuthology.orchestra.run.smithi055:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-15T01:07:40.065 DEBUG:teuthology.orchestra.run.smithi137:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-15T01:07:40.112 INFO:teuthology.orchestra.run.smithi055.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-15T01:07:40.123 INFO:teuthology.orchestra.run.smithi137.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-15T01:07:40.124 INFO:teuthology.orchestra.run.smithi055.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-15T01:07:40.133 INFO:teuthology.orchestra.run.smithi137.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-15T01:07:40.135 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-11-15T01:07:40.142 INFO:teuthology.task.internal:Configuring sudo... 2023-11-15T01:07:40.143 DEBUG:teuthology.orchestra.run.smithi055:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-15T01:07:40.145 DEBUG:teuthology.orchestra.run.smithi137:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-15T01:07:40.177 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-11-15T01:07:40.185 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-11-15T01:07:40.186 DEBUG:teuthology.orchestra.run.smithi055:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-15T01:07:40.220 DEBUG:teuthology.orchestra.run.smithi137:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-15T01:07:40.239 DEBUG:teuthology.orchestra.run.smithi055:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-15T01:07:40.333 DEBUG:teuthology.orchestra.run.smithi055:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-15T01:07:40.393 DEBUG:teuthology.orchestra.run.smithi055:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-15T01:07:40.436 DEBUG:teuthology.orchestra.run.smithi055:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-15T01:07:40.492 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2023-11-15T01:07:40.492 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-15T01:07:40.566 DEBUG:teuthology.orchestra.run.smithi137:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-15T01:07:40.612 DEBUG:teuthology.orchestra.run.smithi137:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-15T01:07:40.670 DEBUG:teuthology.orchestra.run.smithi137:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-15T01:07:40.756 DEBUG:teuthology.orchestra.run.smithi137:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-15T01:07:40.798 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2023-11-15T01:07:40.799 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-15T01:07:40.867 DEBUG:teuthology.orchestra.run.smithi055:> sudo service rsyslog restart 2023-11-15T01:07:40.870 DEBUG:teuthology.orchestra.run.smithi137:> sudo service rsyslog restart 2023-11-15T01:07:40.937 INFO:teuthology.orchestra.run.smithi055.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-15T01:07:40.973 INFO:teuthology.orchestra.run.smithi137.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-15T01:07:41.340 INFO:teuthology.run_tasks:Running task internal.timer... 2023-11-15T01:07:41.348 INFO:teuthology.task.internal:Starting timer... 2023-11-15T01:07:41.348 INFO:teuthology.run_tasks:Running task pcp... 2023-11-15T01:07:41.385 INFO:teuthology.run_tasks:Running task selinux... 2023-11-15T01:07:41.435 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-11-15T01:07:41.436 DEBUG:teuthology.orchestra.run.smithi055:> sudo service auditd rotate 2023-11-15T01:07:41.536 INFO:teuthology.orchestra.run.smithi055.stdout:Rotating logs: 2023-11-15T01:07:41.538 DEBUG:teuthology.orchestra.run.smithi137:> sudo service auditd rotate 2023-11-15T01:07:41.613 INFO:teuthology.orchestra.run.smithi137.stdout:Rotating logs: 2023-11-15T01:07:41.614 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-11-15T01:07:41.615 DEBUG:teuthology.orchestra.run.smithi055:> /usr/sbin/getenforce 2023-11-15T01:07:41.643 INFO:teuthology.orchestra.run.smithi055.stdout:Permissive 2023-11-15T01:07:41.644 DEBUG:teuthology.orchestra.run.smithi137:> /usr/sbin/getenforce 2023-11-15T01:07:41.699 INFO:teuthology.orchestra.run.smithi137.stdout:Permissive 2023-11-15T01:07:41.699 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi055.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi137.front.sepia.ceph.com': 'permissive'} 2023-11-15T01:07:41.700 DEBUG:teuthology.orchestra.run.smithi055:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-15T01:07:41.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-15T01:07:41.737 DEBUG:teuthology.orchestra.run.smithi137:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-15T01:07:41.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-15T01:07:41.764 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-11-15T01:07:41.765 DEBUG:teuthology.orchestra.run.smithi055:> sudo /usr/sbin/setenforce permissive 2023-11-15T01:07:41.812 DEBUG:teuthology.orchestra.run.smithi137:> sudo /usr/sbin/setenforce permissive 2023-11-15T01:07:41.840 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-11-15T01:07:41.850 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-11-15T01:07:41.850 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-11-15T01:07:41.873 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-11-15T01:07:41.875 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi055.front.sepia.ceph.com,smithi137.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-11-15T01:44:15.455 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi055.front.sepia.ceph.com'), Remote(name='ubuntu@smithi137.front.sepia.ceph.com')] 2023-11-15T01:44:15.457 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi055.front.sepia.ceph.com' 2023-11-15T01:44:15.458 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi055.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-15T01:44:15.533 DEBUG:teuthology.orchestra.run.smithi055:> true 2023-11-15T01:44:15.609 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi055.front.sepia.ceph.com' 2023-11-15T01:44:15.609 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2023-11-15T01:44:15.610 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-15T01:44:15.678 DEBUG:teuthology.orchestra.run.smithi137:> true 2023-11-15T01:44:15.746 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi137.front.sepia.ceph.com' 2023-11-15T01:44:15.747 INFO:teuthology.run_tasks:Running task clock... 2023-11-15T01:44:15.757 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-11-15T01:44:15.758 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-15T01:44:15.758 DEBUG:teuthology.orchestra.run.smithi055:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-15T01:44:15.761 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-15T01:44:15.762 DEBUG:teuthology.orchestra.run.smithi137:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-15T01:44:15.798 INFO:teuthology.orchestra.run.smithi055.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-11-15T01:44:15.815 INFO:teuthology.orchestra.run.smithi055.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-11-15T01:44:15.817 INFO:teuthology.orchestra.run.smithi137.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-11-15T01:44:15.831 INFO:teuthology.orchestra.run.smithi137.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-11-15T01:44:15.876 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: ntpd: command not found 2023-11-15T01:44:15.889 INFO:teuthology.orchestra.run.smithi055.stdout:506 Cannot talk to daemon 2023-11-15T01:44:15.897 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: ntpd: command not found 2023-11-15T01:44:15.905 INFO:teuthology.orchestra.run.smithi055.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-11-15T01:44:15.908 INFO:teuthology.orchestra.run.smithi137.stdout:506 Cannot talk to daemon 2023-11-15T01:44:15.921 INFO:teuthology.orchestra.run.smithi055.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-11-15T01:44:15.921 INFO:teuthology.orchestra.run.smithi137.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-11-15T01:44:15.935 INFO:teuthology.orchestra.run.smithi137.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-11-15T01:44:15.993 INFO:teuthology.orchestra.run.smithi055.stderr:bash: ntpq: command not found 2023-11-15T01:44:15.998 INFO:teuthology.orchestra.run.smithi055.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-15T01:44:15.998 INFO:teuthology.orchestra.run.smithi055.stdout:=============================================================================== 2023-11-15T01:44:15.998 INFO:teuthology.orchestra.run.smithi055.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-15T01:44:15.998 INFO:teuthology.orchestra.run.smithi055.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-15T01:44:15.999 INFO:teuthology.orchestra.run.smithi055.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-15T01:44:15.999 INFO:teuthology.orchestra.run.smithi055.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-15T01:44:15.999 INFO:teuthology.orchestra.run.smithi137.stderr:bash: ntpq: command not found 2023-11-15T01:44:16.002 INFO:teuthology.orchestra.run.smithi137.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-15T01:44:16.003 INFO:teuthology.orchestra.run.smithi137.stdout:=============================================================================== 2023-11-15T01:44:16.003 INFO:teuthology.orchestra.run.smithi137.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-15T01:44:16.003 INFO:teuthology.orchestra.run.smithi137.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-15T01:44:16.003 INFO:teuthology.orchestra.run.smithi137.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-15T01:44:16.003 INFO:teuthology.orchestra.run.smithi137.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-15T01:44:16.004 INFO:teuthology.run_tasks:Running task pexec... 2023-11-15T01:44:16.017 INFO:teuthology.task.pexec:Executing custom commands... 2023-11-15T01:44:16.017 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi055.front.sepia.ceph.com 2023-11-15T01:44:16.018 DEBUG:teuthology.orchestra.run.smithi055:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-11-15T01:44:16.018 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi137.front.sepia.ceph.com 2023-11-15T01:44:16.019 DEBUG:teuthology.orchestra.run.smithi137:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-11-15T01:44:16.925 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:30:47 ago on Wed 15 Nov 2023 01:13:29 AM UTC. 2023-11-15T01:44:17.051 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:30:04 ago on Wed 15 Nov 2023 01:14:12 AM UTC. 2023-11-15T01:44:19.066 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2023-11-15T01:44:19.067 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T01:44:19.067 INFO:teuthology.orchestra.run.smithi137.stdout: Package Architecture Version Repository Size 2023-11-15T01:44:19.068 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T01:44:19.068 INFO:teuthology.orchestra.run.smithi137.stdout:Resetting modules: 2023-11-15T01:44:19.068 INFO:teuthology.orchestra.run.smithi137.stdout: container-tools 2023-11-15T01:44:19.068 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:44:19.068 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2023-11-15T01:44:19.069 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T01:44:19.069 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:44:19.365 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2023-11-15T01:44:19.422 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2023-11-15T01:44:19.424 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-15T01:44:19.424 INFO:teuthology.orchestra.run.smithi055.stdout: Package Architecture Version Repository Size 2023-11-15T01:44:19.424 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-15T01:44:19.424 INFO:teuthology.orchestra.run.smithi055.stdout:Resetting modules: 2023-11-15T01:44:19.425 INFO:teuthology.orchestra.run.smithi055.stdout: container-tools 2023-11-15T01:44:19.425 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-15T01:44:19.425 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2023-11-15T01:44:19.425 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-15T01:44:19.425 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-15T01:44:19.711 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2023-11-15T01:44:20.325 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:30:51 ago on Wed 15 Nov 2023 01:13:29 AM UTC. 2023-11-15T01:44:20.775 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:30:08 ago on Wed 15 Nov 2023 01:14:12 AM UTC. 2023-11-15T01:44:22.417 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2023-11-15T01:44:22.595 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T01:44:22.595 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2023-11-15T01:44:22.595 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T01:44:22.596 INFO:teuthology.orchestra.run.smithi137.stdout:Installing group/module packages: 2023-11-15T01:44:22.596 INFO:teuthology.orchestra.run.smithi137.stdout: aardvark-dns x86_64 2:1.8.0-1.module_el8+712+4cd1bd69 appstream 1.0 M 2023-11-15T01:44:22.596 INFO:teuthology.orchestra.run.smithi137.stdout: buildah x86_64 1:1.32.0-1.module_el8+712+4cd1bd69 appstream 9.6 M 2023-11-15T01:44:22.596 INFO:teuthology.orchestra.run.smithi137.stdout: cockpit-podman noarch 78-1.module_el8+712+4cd1bd69 appstream 750 k 2023-11-15T01:44:22.597 INFO:teuthology.orchestra.run.smithi137.stdout: crun x86_64 1.9.2-1.module_el8+712+4cd1bd69 appstream 241 k 2023-11-15T01:44:22.597 INFO:teuthology.orchestra.run.smithi137.stdout: netavark x86_64 2:1.8.0-2.module_el8+712+4cd1bd69 appstream 3.9 M 2023-11-15T01:44:22.597 INFO:teuthology.orchestra.run.smithi137.stdout: python3-podman noarch 4.7.0-1.module_el8+712+4cd1bd69 appstream 178 k 2023-11-15T01:44:22.597 INFO:teuthology.orchestra.run.smithi137.stdout: skopeo x86_64 2:1.13.3-1.module_el8+712+4cd1bd69 appstream 8.1 M 2023-11-15T01:44:22.597 INFO:teuthology.orchestra.run.smithi137.stdout: toolbox x86_64 0.0.99.4-6.module_el8+712+4cd1bd69 appstream 2.5 M 2023-11-15T01:44:22.598 INFO:teuthology.orchestra.run.smithi137.stdout: udica noarch 0.2.6-20.module_el8+712+4cd1bd69 appstream 53 k 2023-11-15T01:44:22.598 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2023-11-15T01:44:22.598 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2023-11-15T01:44:22.598 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2023-11-15T01:44:22.598 INFO:teuthology.orchestra.run.smithi137.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2023-11-15T01:44:22.599 INFO:teuthology.orchestra.run.smithi137.stdout:Installing weak dependencies: 2023-11-15T01:44:22.599 INFO:teuthology.orchestra.run.smithi137.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2023-11-15T01:44:22.599 INFO:teuthology.orchestra.run.smithi137.stdout:Installing module profiles: 2023-11-15T01:44:22.599 INFO:teuthology.orchestra.run.smithi137.stdout: container-tools/common 2023-11-15T01:44:22.600 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:44:22.600 INFO:teuthology.orchestra.run.smithi137.stdout:Enabling module streams: 2023-11-15T01:44:22.600 INFO:teuthology.orchestra.run.smithi137.stdout: container-tools rhel8 2023-11-15T01:44:22.600 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:44:22.600 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2023-11-15T01:44:22.601 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T01:44:22.601 INFO:teuthology.orchestra.run.smithi137.stdout:Install 13 Packages 2023-11-15T01:44:22.601 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:44:22.602 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 27 M 2023-11-15T01:44:22.602 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 89 M 2023-11-15T01:44:22.603 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2023-11-15T01:44:22.961 INFO:teuthology.orchestra.run.smithi137.stdout:(1/13): cockpit-podman-78-1.module_el8+712+4cd1 2.3 MB/s | 750 kB 00:00 2023-11-15T01:44:22.987 INFO:teuthology.orchestra.run.smithi137.stdout:(2/13): aardvark-dns-1.8.0-1.module_el8+712+4cd 2.9 MB/s | 1.0 MB 00:00 2023-11-15T01:44:23.020 INFO:teuthology.orchestra.run.smithi137.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 646 kB/s | 38 kB 00:00 2023-11-15T01:44:23.054 INFO:teuthology.orchestra.run.smithi137.stdout:(4/13): crun-1.9.2-1.module_el8+712+4cd1bd69.x8 3.5 MB/s | 241 kB 00:00 2023-11-15T01:44:23.112 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2023-11-15T01:44:23.229 INFO:teuthology.orchestra.run.smithi137.stdout:(5/13): python3-podman-4.7.0-1.module_el8+712+4 1.0 MB/s | 178 kB 00:00 2023-11-15T01:44:23.296 INFO:teuthology.orchestra.run.smithi137.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 372 kB/s | 25 kB 00:00 2023-11-15T01:44:23.316 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-15T01:44:23.316 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repository Size 2023-11-15T01:44:23.317 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-15T01:44:23.317 INFO:teuthology.orchestra.run.smithi055.stdout:Installing group/module packages: 2023-11-15T01:44:23.317 INFO:teuthology.orchestra.run.smithi055.stdout: aardvark-dns x86_64 2:1.8.0-1.module_el8+712+4cd1bd69 appstream 1.0 M 2023-11-15T01:44:23.317 INFO:teuthology.orchestra.run.smithi055.stdout: buildah x86_64 1:1.32.0-1.module_el8+712+4cd1bd69 appstream 9.6 M 2023-11-15T01:44:23.317 INFO:teuthology.orchestra.run.smithi055.stdout: cockpit-podman noarch 78-1.module_el8+712+4cd1bd69 appstream 750 k 2023-11-15T01:44:23.318 INFO:teuthology.orchestra.run.smithi055.stdout: crun x86_64 1.9.2-1.module_el8+712+4cd1bd69 appstream 241 k 2023-11-15T01:44:23.318 INFO:teuthology.orchestra.run.smithi055.stdout: netavark x86_64 2:1.8.0-2.module_el8+712+4cd1bd69 appstream 3.9 M 2023-11-15T01:44:23.318 INFO:teuthology.orchestra.run.smithi055.stdout: python3-podman noarch 4.7.0-1.module_el8+712+4cd1bd69 appstream 178 k 2023-11-15T01:44:23.318 INFO:teuthology.orchestra.run.smithi055.stdout: skopeo x86_64 2:1.13.3-1.module_el8+712+4cd1bd69 appstream 8.1 M 2023-11-15T01:44:23.319 INFO:teuthology.orchestra.run.smithi055.stdout: toolbox x86_64 0.0.99.4-6.module_el8+712+4cd1bd69 appstream 2.5 M 2023-11-15T01:44:23.319 INFO:teuthology.orchestra.run.smithi055.stdout: udica noarch 0.2.6-20.module_el8+712+4cd1bd69 appstream 53 k 2023-11-15T01:44:23.319 INFO:teuthology.orchestra.run.smithi055.stdout:Installing dependencies: 2023-11-15T01:44:23.319 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2023-11-15T01:44:23.319 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2023-11-15T01:44:23.319 INFO:teuthology.orchestra.run.smithi055.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2023-11-15T01:44:23.320 INFO:teuthology.orchestra.run.smithi055.stdout:Installing weak dependencies: 2023-11-15T01:44:23.320 INFO:teuthology.orchestra.run.smithi055.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2023-11-15T01:44:23.320 INFO:teuthology.orchestra.run.smithi055.stdout:Installing module profiles: 2023-11-15T01:44:23.320 INFO:teuthology.orchestra.run.smithi055.stdout: container-tools/common 2023-11-15T01:44:23.320 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-15T01:44:23.321 INFO:teuthology.orchestra.run.smithi055.stdout:Enabling module streams: 2023-11-15T01:44:23.321 INFO:teuthology.orchestra.run.smithi055.stdout: container-tools rhel8 2023-11-15T01:44:23.321 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-15T01:44:23.321 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2023-11-15T01:44:23.321 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-15T01:44:23.322 INFO:teuthology.orchestra.run.smithi055.stdout:Install 13 Packages 2023-11-15T01:44:23.322 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-15T01:44:23.325 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 27 M 2023-11-15T01:44:23.326 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 89 M 2023-11-15T01:44:23.326 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2023-11-15T01:44:23.422 INFO:teuthology.orchestra.run.smithi137.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 751 kB/s | 94 kB 00:00 2023-11-15T01:44:23.522 INFO:teuthology.orchestra.run.smithi137.stdout:(8/13): netavark-1.8.0-2.module_el8+712+4cd1bd6 7.8 MB/s | 3.9 MB 00:00 2023-11-15T01:44:23.734 INFO:teuthology.orchestra.run.smithi055.stdout:(1/13): cockpit-podman-78-1.module_el8+712+4cd1 2.0 MB/s | 750 kB 00:00 2023-11-15T01:44:23.814 INFO:teuthology.orchestra.run.smithi137.stdout:(9/13): buildah-1.32.0-1.module_el8+712+4cd1bd6 8.2 MB/s | 9.6 MB 00:01 2023-11-15T01:44:23.908 INFO:teuthology.orchestra.run.smithi137.stdout:(10/13): toolbox-0.0.99.4-6.module_el8+712+4cd1 6.6 MB/s | 2.5 MB 00:00 2023-11-15T01:44:23.933 INFO:teuthology.orchestra.run.smithi137.stdout:(11/13): udica-0.2.6-20.module_el8+712+4cd1bd69 442 kB/s | 53 kB 00:00 2023-11-15T01:44:23.951 INFO:teuthology.orchestra.run.smithi055.stdout:(2/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 175 kB/s | 38 kB 00:00 2023-11-15T01:44:23.967 INFO:teuthology.orchestra.run.smithi137.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 699 kB/s | 41 kB 00:00 2023-11-15T01:44:23.985 INFO:teuthology.orchestra.run.smithi055.stdout:(3/13): aardvark-dns-1.8.0-1.module_el8+712+4cd 1.6 MB/s | 1.0 MB 00:00 2023-11-15T01:44:24.076 INFO:teuthology.orchestra.run.smithi055.stdout:(4/13): crun-1.9.2-1.module_el8+712+4cd1bd69.x8 1.9 MB/s | 241 kB 00:00 2023-11-15T01:44:24.152 INFO:teuthology.orchestra.run.smithi055.stdout:(5/13): python3-podman-4.7.0-1.module_el8+712+4 2.3 MB/s | 178 kB 00:00 2023-11-15T01:44:24.194 INFO:teuthology.orchestra.run.smithi055.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 593 kB/s | 25 kB 00:00 2023-11-15T01:44:24.217 INFO:teuthology.orchestra.run.smithi137.stdout:(13/13): skopeo-1.13.3-1.module_el8+712+4cd1bd6 10 MB/s | 8.1 MB 00:00 2023-11-15T01:44:24.217 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2023-11-15T01:44:24.218 INFO:teuthology.orchestra.run.smithi137.stdout:Total 16 MB/s | 27 MB 00:01 2023-11-15T01:44:24.277 INFO:teuthology.orchestra.run.smithi055.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 1.1 MB/s | 94 kB 00:00 2023-11-15T01:44:24.453 INFO:teuthology.orchestra.run.smithi055.stdout:(8/13): buildah-1.32.0-1.module_el8+712+4cd1bd6 8.8 MB/s | 9.6 MB 00:01 2023-11-15T01:44:24.482 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2023-11-15T01:44:24.503 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2023-11-15T01:44:24.503 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2023-11-15T01:44:24.520 INFO:teuthology.orchestra.run.smithi055.stdout:(9/13): netavark-1.8.0-2.module_el8+712+4cd1bd6 7.3 MB/s | 3.9 MB 00:00 2023-11-15T01:44:24.620 INFO:teuthology.orchestra.run.smithi055.stdout:(10/13): toolbox-0.0.99.4-6.module_el8+712+4cd1 15 MB/s | 2.5 MB 00:00 2023-11-15T01:44:24.637 INFO:teuthology.orchestra.run.smithi055.stdout:(11/13): udica-0.2.6-20.module_el8+712+4cd1bd69 450 kB/s | 53 kB 00:00 2023-11-15T01:44:24.662 INFO:teuthology.orchestra.run.smithi055.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 981 kB/s | 41 kB 00:00 2023-11-15T01:44:24.768 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2023-11-15T01:44:24.772 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2023-11-15T01:44:25.046 INFO:teuthology.orchestra.run.smithi055.stdout:(13/13): skopeo-1.13.3-1.module_el8+712+4cd1bd6 11 MB/s | 8.1 MB 00:00 2023-11-15T01:44:25.047 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2023-11-15T01:44:25.047 INFO:teuthology.orchestra.run.smithi055.stdout:Total 15 MB/s | 27 MB 00:01 2023-11-15T01:44:25.338 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2023-11-15T01:44:25.361 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2023-11-15T01:44:25.362 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2023-11-15T01:44:25.489 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2023-11-15T01:44:25.635 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2023-11-15T01:44:25.639 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2023-11-15T01:44:25.742 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2023-11-15T01:44:25.896 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2023-11-15T01:44:26.119 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2023-11-15T01:44:26.319 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2023-11-15T01:44:26.348 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-11-15T01:44:26.535 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2023-11-15T01:44:26.703 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2023-11-15T01:44:26.882 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2023-11-15T01:44:26.931 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 5/13 2023-11-15T01:44:27.121 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-11-15T01:44:27.704 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 5/13 2023-11-15T01:44:28.026 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2023-11-15T01:44:28.294 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 7/13 2023-11-15T01:44:28.482 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 8/13 2023-11-15T01:44:28.758 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2023-11-15T01:44:28.766 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 9/13 2023-11-15T01:44:28.980 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 7/13 2023-11-15T01:44:29.181 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 10/13 2023-11-15T01:44:29.192 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 8/13 2023-11-15T01:44:29.441 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 9/13 2023-11-15T01:44:29.856 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 10/13 2023-11-15T01:44:30.068 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2023-11-15T01:44:30.283 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 12/13 2023-11-15T01:44:30.486 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2023-11-15T01:44:30.799 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2023-11-15T01:44:30.907 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2023-11-15T01:44:30.908 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 1/13 2023-11-15T01:44:30.908 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2/13 2023-11-15T01:44:30.908 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 3/13 2023-11-15T01:44:30.908 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-11-15T01:44:30.909 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 5/13 2023-11-15T01:44:30.909 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2023-11-15T01:44:30.909 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 7/13 2023-11-15T01:44:30.909 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2023-11-15T01:44:30.910 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2023-11-15T01:44:30.910 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 10/13 2023-11-15T01:44:30.910 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2023-11-15T01:44:30.910 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 12/13 2023-11-15T01:44:31.010 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 12/13 2023-11-15T01:44:31.192 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2023-11-15T01:44:31.672 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2023-11-15T01:44:31.673 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 1/13 2023-11-15T01:44:31.673 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2/13 2023-11-15T01:44:31.673 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 3/13 2023-11-15T01:44:31.673 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-11-15T01:44:31.674 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 5/13 2023-11-15T01:44:31.674 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2023-11-15T01:44:31.674 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 7/13 2023-11-15T01:44:31.674 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2023-11-15T01:44:31.674 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2023-11-15T01:44:31.675 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 10/13 2023-11-15T01:44:31.675 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2023-11-15T01:44:31.675 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 12/13 2023-11-15T01:44:31.701 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2023-11-15T01:44:31.702 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:44:31.702 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2023-11-15T01:44:31.702 INFO:teuthology.orchestra.run.smithi137.stdout: aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86_64 2023-11-15T01:44:31.703 INFO:teuthology.orchestra.run.smithi137.stdout: buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2023-11-15T01:44:31.703 INFO:teuthology.orchestra.run.smithi137.stdout: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 2023-11-15T01:44:31.703 INFO:teuthology.orchestra.run.smithi137.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2023-11-15T01:44:31.703 INFO:teuthology.orchestra.run.smithi137.stdout: crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 2023-11-15T01:44:31.703 INFO:teuthology.orchestra.run.smithi137.stdout: netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 2023-11-15T01:44:31.703 INFO:teuthology.orchestra.run.smithi137.stdout: python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noarch 2023-11-15T01:44:31.704 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-11-15T01:44:31.704 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-11-15T01:44:31.704 INFO:teuthology.orchestra.run.smithi137.stdout: skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 2023-11-15T01:44:31.704 INFO:teuthology.orchestra.run.smithi137.stdout: toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 2023-11-15T01:44:31.704 INFO:teuthology.orchestra.run.smithi137.stdout: udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 2023-11-15T01:44:31.705 INFO:teuthology.orchestra.run.smithi137.stdout: yajl-2.1.0-12.el8.x86_64 2023-11-15T01:44:31.705 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:44:31.705 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2023-11-15T01:44:32.033 DEBUG:teuthology.parallel:result is None 2023-11-15T01:44:32.515 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2023-11-15T01:44:32.515 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-15T01:44:32.515 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2023-11-15T01:44:32.516 INFO:teuthology.orchestra.run.smithi055.stdout: aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86_64 2023-11-15T01:44:32.516 INFO:teuthology.orchestra.run.smithi055.stdout: buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2023-11-15T01:44:32.516 INFO:teuthology.orchestra.run.smithi055.stdout: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 2023-11-15T01:44:32.516 INFO:teuthology.orchestra.run.smithi055.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2023-11-15T01:44:32.516 INFO:teuthology.orchestra.run.smithi055.stdout: crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 2023-11-15T01:44:32.517 INFO:teuthology.orchestra.run.smithi055.stdout: netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 2023-11-15T01:44:32.517 INFO:teuthology.orchestra.run.smithi055.stdout: python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noarch 2023-11-15T01:44:32.517 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-11-15T01:44:32.517 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-11-15T01:44:32.517 INFO:teuthology.orchestra.run.smithi055.stdout: skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 2023-11-15T01:44:32.518 INFO:teuthology.orchestra.run.smithi055.stdout: toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 2023-11-15T01:44:32.518 INFO:teuthology.orchestra.run.smithi055.stdout: udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 2023-11-15T01:44:32.518 INFO:teuthology.orchestra.run.smithi055.stdout: yajl-2.1.0-12.el8.x86_64 2023-11-15T01:44:32.518 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-15T01:44:32.518 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2023-11-15T01:44:32.775 DEBUG:teuthology.parallel:result is None 2023-11-15T01:44:32.775 INFO:teuthology.run_tasks:Running task pexec... 2023-11-15T01:44:32.785 INFO:teuthology.task.pexec:Executing custom commands... 2023-11-15T01:44:32.786 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi055.front.sepia.ceph.com 2023-11-15T01:44:32.786 DEBUG:teuthology.orchestra.run.smithi055:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-11-15T01:44:32.787 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi137.front.sepia.ceph.com 2023-11-15T01:44:32.787 DEBUG:teuthology.orchestra.run.smithi137:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-11-15T01:44:32.969 DEBUG:teuthology.parallel:result is None 2023-11-15T01:44:33.012 DEBUG:teuthology.parallel:result is None 2023-11-15T01:44:33.013 INFO:teuthology.run_tasks:Running task install... 2023-11-15T01:44:33.022 DEBUG:teuthology.task.install:project ceph 2023-11-15T01:44:33.022 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '620f25c1794f62207424fc9319427943fb8d1fae'}} 2023-11-15T01:44:33.023 DEBUG:teuthology.task.install:config {'exclude_packages': ['ceph-volume'], 'tag': 'v16.2.4', 'flavor': 'default', 'sha1': '620f25c1794f62207424fc9319427943fb8d1fae'} 2023-11-15T01:44:33.023 INFO:teuthology.task.install:Using flavor: default 2023-11-15T01:44:33.029 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-11-15T01:44:33.029 INFO:teuthology.task.install:extra packages: [] 2023-11-15T01:44:33.029 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '620f25c1794f62207424fc9319427943fb8d1fae', 'tag': 'v16.2.4', 'wait_for_package': False} 2023-11-15T01:44:33.030 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-11-15T01:44:33.030 INFO:teuthology.packaging:ref: None 2023-11-15T01:44:33.031 INFO:teuthology.packaging:tag: v16.2.4 2023-11-15T01:44:33.031 INFO:teuthology.packaging:branch: None 2023-11-15T01:44:33.031 INFO:teuthology.packaging:sha1: 620f25c1794f62207424fc9319427943fb8d1fae 2023-11-15T01:44:33.510 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-11-15T01:44:33.511 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-11-15T01:44:33.514 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '620f25c1794f62207424fc9319427943fb8d1fae', 'tag': 'v16.2.4', 'wait_for_package': False} 2023-11-15T01:44:33.514 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-11-15T01:44:33.515 INFO:teuthology.packaging:ref: None 2023-11-15T01:44:33.515 INFO:teuthology.packaging:tag: v16.2.4 2023-11-15T01:44:33.515 INFO:teuthology.packaging:branch: None 2023-11-15T01:44:33.515 INFO:teuthology.packaging:sha1: 620f25c1794f62207424fc9319427943fb8d1fae 2023-11-15T01:44:33.913 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-11-15T01:44:33.914 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-11-15T01:44:34.201 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2023-11-15T01:44:34.201 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2023-11-15T01:44:34.206 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2023-11-15T01:44:34.206 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2023-11-15T01:44:34.251 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-11-15T01:44:34.252 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2023-11-15T01:44:34.252 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-11-15T01:44:34.255 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-11-15T01:44:34.255 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2023-11-15T01:44:34.255 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-11-15T01:44:34.288 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-11-15T01:44:34.288 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-11-15T01:44:34.289 INFO:teuthology.packaging:ref: None 2023-11-15T01:44:34.289 INFO:teuthology.packaging:tag: v16.2.4 2023-11-15T01:44:34.289 INFO:teuthology.packaging:branch: None 2023-11-15T01:44:34.289 INFO:teuthology.packaging:sha1: 620f25c1794f62207424fc9319427943fb8d1fae 2023-11-15T01:44:34.290 DEBUG:teuthology.orchestra.run.smithi137:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-11-15T01:44:34.291 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-11-15T01:44:34.291 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-11-15T01:44:34.292 INFO:teuthology.packaging:ref: None 2023-11-15T01:44:34.292 INFO:teuthology.packaging:tag: v16.2.4 2023-11-15T01:44:34.292 INFO:teuthology.packaging:branch: None 2023-11-15T01:44:34.292 INFO:teuthology.packaging:sha1: 620f25c1794f62207424fc9319427943fb8d1fae 2023-11-15T01:44:34.292 DEBUG:teuthology.orchestra.run.smithi055:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-11-15T01:44:34.360 DEBUG:teuthology.orchestra.run.smithi137:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-11-15T01:44:34.367 DEBUG:teuthology.orchestra.run.smithi055:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-11-15T01:44:34.441 DEBUG:teuthology.orchestra.run.smithi137:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-11-15T01:44:34.455 DEBUG:teuthology.orchestra.run.smithi055:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-11-15T01:44:34.487 INFO:teuthology.orchestra.run.smithi055.stdout:check_obsoletes = 1 2023-11-15T01:44:34.488 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum clean all 2023-11-15T01:44:34.509 INFO:teuthology.orchestra.run.smithi137.stdout:check_obsoletes = 1 2023-11-15T01:44:34.511 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum clean all 2023-11-15T01:44:34.794 INFO:teuthology.orchestra.run.smithi055.stdout:76 files removed 2023-11-15T01:44:34.821 INFO:teuthology.orchestra.run.smithi137.stdout:76 files removed 2023-11-15T01:44:34.826 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph-radosgw 2023-11-15T01:44:34.848 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-radosgw 2023-11-15T01:44:35.368 INFO:teuthology.orchestra.run.smithi137.stdout:ceph packages for x86_64 643 kB/s | 74 kB 00:00 2023-11-15T01:44:35.379 INFO:teuthology.orchestra.run.smithi055.stdout:ceph packages for x86_64 859 kB/s | 74 kB 00:00 2023-11-15T01:44:35.535 INFO:teuthology.orchestra.run.smithi137.stdout:ceph noarch packages 117 kB/s | 15 kB 00:00 2023-11-15T01:44:35.571 INFO:teuthology.orchestra.run.smithi055.stdout:ceph noarch packages 101 kB/s | 15 kB 00:00 2023-11-15T01:44:35.710 INFO:teuthology.orchestra.run.smithi137.stdout:ceph source packages 12 kB/s | 1.7 kB 00:00 2023-11-15T01:44:35.714 INFO:teuthology.orchestra.run.smithi055.stdout:ceph source packages 16 kB/s | 1.7 kB 00:00 2023-11-15T01:44:43.431 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS Stream 8 - AppStream 4.4 MB/s | 34 MB 00:07 2023-11-15T01:44:44.877 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 8 - AppStream 3.7 MB/s | 34 MB 00:09 2023-11-15T01:44:52.705 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS Stream 8 - BaseOS 23 MB/s | 53 MB 00:02 2023-11-15T01:44:55.235 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 8 - BaseOS 13 MB/s | 53 MB 00:04 2023-11-15T01:45:01.648 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS Stream 8 - Extras 32 kB/s | 18 kB 00:00 2023-11-15T01:45:02.007 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS Stream 8 - Extras common packages 44 kB/s | 6.9 kB 00:00 2023-11-15T01:45:05.042 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS Stream 8 - PowerTools 2.3 MB/s | 6.6 MB 00:02 2023-11-15T01:45:06.618 INFO:teuthology.orchestra.run.smithi055.stdout:Copr repo for python3-asyncssh owned by ceph 25 kB/s | 3.5 kB 00:00 2023-11-15T01:45:07.169 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 8 - Extras 4.1 kB/s | 18 kB 00:04 2023-11-15T01:45:07.511 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 8 - Extras common packages 44 kB/s | 6.9 kB 00:00 2023-11-15T01:45:08.255 INFO:teuthology.orchestra.run.smithi055.stdout:Extra Packages for Enterprise Linux 11 MB/s | 16 MB 00:01 2023-11-15T01:45:11.329 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 8 - PowerTools 1.8 MB/s | 6.6 MB 00:03 2023-11-15T01:45:12.510 INFO:teuthology.orchestra.run.smithi055.stdout:lab-extras 415 kB/s | 24 kB 00:00 2023-11-15T01:45:12.738 INFO:teuthology.orchestra.run.smithi137.stdout:Copr repo for python3-asyncssh owned by ceph 31 kB/s | 3.5 kB 00:00 2023-11-15T01:45:13.708 INFO:teuthology.orchestra.run.smithi137.stdout:Extra Packages for Enterprise Linux 20 MB/s | 16 MB 00:00 2023-11-15T01:45:17.522 INFO:teuthology.orchestra.run.smithi137.stdout:lab-extras 391 kB/s | 24 kB 00:00 2023-11-15T01:45:17.535 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2023-11-15T01:45:17.539 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-15T01:45:17.539 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repository Size 2023-11-15T01:45:17.539 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-15T01:45:17.539 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2023-11-15T01:45:17.540 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2023-11-15T01:45:17.540 INFO:teuthology.orchestra.run.smithi055.stdout:Upgrading: 2023-11-15T01:45:17.540 INFO:teuthology.orchestra.run.smithi055.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2023-11-15T01:45:17.540 INFO:teuthology.orchestra.run.smithi055.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2023-11-15T01:45:17.541 INFO:teuthology.orchestra.run.smithi055.stdout:Installing dependencies: 2023-11-15T01:45:17.541 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2023-11-15T01:45:17.541 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2023-11-15T01:45:17.541 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2023-11-15T01:45:17.541 INFO:teuthology.orchestra.run.smithi055.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2023-11-15T01:45:17.542 INFO:teuthology.orchestra.run.smithi055.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2023-11-15T01:45:17.542 INFO:teuthology.orchestra.run.smithi055.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-11-15T01:45:17.542 INFO:teuthology.orchestra.run.smithi055.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2023-11-15T01:45:17.542 INFO:teuthology.orchestra.run.smithi055.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2023-11-15T01:45:17.542 INFO:teuthology.orchestra.run.smithi055.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-11-15T01:45:17.543 INFO:teuthology.orchestra.run.smithi055.stdout: libpmemobj x86_64 1.6.1-1.el8 appstream 145 k 2023-11-15T01:45:17.543 INFO:teuthology.orchestra.run.smithi055.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2023-11-15T01:45:17.543 INFO:teuthology.orchestra.run.smithi055.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2023-11-15T01:45:17.543 INFO:teuthology.orchestra.run.smithi055.stdout: librdkafka x86_64 0.11.4-3.el8 appstream 354 k 2023-11-15T01:45:17.543 INFO:teuthology.orchestra.run.smithi055.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2023-11-15T01:45:17.544 INFO:teuthology.orchestra.run.smithi055.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-11-15T01:45:17.544 INFO:teuthology.orchestra.run.smithi055.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2023-11-15T01:45:17.544 INFO:teuthology.orchestra.run.smithi055.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2023-11-15T01:45:17.544 INFO:teuthology.orchestra.run.smithi055.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2023-11-15T01:45:17.544 INFO:teuthology.orchestra.run.smithi055.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2023-11-15T01:45:17.545 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2023-11-15T01:45:17.545 INFO:teuthology.orchestra.run.smithi055.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2023-11-15T01:45:17.545 INFO:teuthology.orchestra.run.smithi055.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2023-11-15T01:45:17.545 INFO:teuthology.orchestra.run.smithi055.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2023-11-15T01:45:17.545 INFO:teuthology.orchestra.run.smithi055.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2023-11-15T01:45:17.546 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-15T01:45:17.546 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2023-11-15T01:45:17.546 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-15T01:45:17.546 INFO:teuthology.orchestra.run.smithi055.stdout:Install 25 Packages 2023-11-15T01:45:17.546 INFO:teuthology.orchestra.run.smithi055.stdout:Upgrade 2 Packages 2023-11-15T01:45:17.547 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-15T01:45:17.547 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 59 M 2023-11-15T01:45:17.547 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2023-11-15T01:45:17.820 INFO:teuthology.orchestra.run.smithi055.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 43 MB/s | 6.0 MB 00:00 2023-11-15T01:45:17.837 INFO:teuthology.orchestra.run.smithi055.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 1.5 MB/s | 25 kB 00:00 2023-11-15T01:45:17.870 INFO:teuthology.orchestra.run.smithi055.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 24 MB/s | 801 kB 00:00 2023-11-15T01:45:17.912 INFO:teuthology.orchestra.run.smithi055.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 11 MB/s | 486 kB 00:00 2023-11-15T01:45:18.013 INFO:teuthology.orchestra.run.smithi055.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 41 MB/s | 4.1 MB 00:00 2023-11-15T01:45:18.029 INFO:teuthology.orchestra.run.smithi055.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 2.7 MB/s | 44 kB 00:00 2023-11-15T01:45:18.055 INFO:teuthology.orchestra.run.smithi055.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 3.1 MB/s | 80 kB 00:00 2023-11-15T01:45:18.080 INFO:teuthology.orchestra.run.smithi055.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 8.4 MB/s | 213 kB 00:00 2023-11-15T01:45:18.114 INFO:teuthology.orchestra.run.smithi055.stdout:(9/27): python3-rados-16.2.4-0.el8.x86_64.rpm 11 MB/s | 387 kB 00:00 2023-11-15T01:45:18.148 INFO:teuthology.orchestra.run.smithi055.stdout:(10/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 11 MB/s | 368 kB 00:00 2023-11-15T01:45:18.173 INFO:teuthology.orchestra.run.smithi055.stdout:(11/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 4.5 MB/s | 114 kB 00:00 2023-11-15T01:45:18.374 INFO:teuthology.orchestra.run.smithi055.stdout:(12/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 18 MB/s | 13 MB 00:00 2023-11-15T01:45:18.399 INFO:teuthology.orchestra.run.smithi055.stdout:(13/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 645 kB/s | 145 kB 00:00 2023-11-15T01:45:18.449 INFO:teuthology.orchestra.run.smithi055.stdout:(14/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 5.1 MB/s | 259 kB 00:00 2023-11-15T01:45:18.716 INFO:teuthology.orchestra.run.smithi055.stdout:(15/27): ceph-common-16.2.4-0.el8.x86_64.rpm 23 MB/s | 24 MB 00:01 2023-11-15T01:45:18.741 INFO:teuthology.orchestra.run.smithi055.stdout:(16/27): librdkafka-0.11.4-3.el8.x86_64.rpm 964 kB/s | 354 kB 00:00 2023-11-15T01:45:18.766 INFO:teuthology.orchestra.run.smithi055.stdout:(17/27): python3-prettytable-0.7.2-14.el8.noarc 138 kB/s | 44 kB 00:00 2023-11-15T01:45:19.075 INFO:teuthology.orchestra.run.smithi055.stdout:(18/27): librabbitmq-0.9.0-4.el8.x86_64.rpm 141 kB/s | 47 kB 00:00 2023-11-15T01:45:19.134 INFO:teuthology.orchestra.run.smithi055.stdout:(19/27): daxctl-libs-71.1-7.el8.x86_64.rpm 101 kB/s | 42 kB 00:00 2023-11-15T01:45:19.159 INFO:teuthology.orchestra.run.smithi055.stdout:(20/27): ndctl-libs-71.1-7.el8.x86_64.rpm 204 kB/s | 80 kB 00:00 2023-11-15T01:45:19.351 INFO:teuthology.orchestra.run.smithi055.stdout:(21/27): leveldb-1.22-1.el8.x86_64.rpm 945 kB/s | 181 kB 00:00 2023-11-15T01:45:19.376 INFO:teuthology.orchestra.run.smithi055.stdout:(22/27): gperftools-libs-2.7-9.el8.x86_64.rpm 1.2 MB/s | 306 kB 00:00 2023-11-15T01:45:19.401 INFO:teuthology.orchestra.run.smithi055.stdout:(23/27): fmt-6.2.1-1.el8.x86_64.rpm 357 kB/s | 116 kB 00:00 2023-11-15T01:45:19.427 INFO:teuthology.orchestra.run.smithi055.stdout:(24/27): liboath-2.6.2-3.el8.x86_64.rpm 781 kB/s | 59 kB 00:00 2023-11-15T01:45:19.452 INFO:teuthology.orchestra.run.smithi055.stdout:(25/27): libunwind-1.3.1-3.el8.x86_64.rpm 993 kB/s | 75 kB 00:00 2023-11-15T01:45:19.535 INFO:teuthology.orchestra.run.smithi055.stdout:(26/27): librados2-16.2.4-0.el8.x86_64.rpm 28 MB/s | 3.7 MB 00:00 2023-11-15T01:45:19.594 INFO:teuthology.orchestra.run.smithi055.stdout:(27/27): librbd1-16.2.4-0.el8.x86_64.rpm 24 MB/s | 4.0 MB 00:00 2023-11-15T01:45:19.596 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2023-11-15T01:45:19.597 INFO:teuthology.orchestra.run.smithi055.stdout:Total 29 MB/s | 59 MB 00:02 2023-11-15T01:45:19.671 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2023-11-15T01:45:19.726 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2023-11-15T01:45:19.726 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2023-11-15T01:45:20.109 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2023-11-15T01:45:20.113 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2023-11-15T01:45:21.535 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2023-11-15T01:45:21.744 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2023-11-15T01:45:21.945 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2023-11-15T01:45:22.102 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2023-11-15T01:45:22.123 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2023-11-15T01:45:22.126 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T01:45:22.127 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2023-11-15T01:45:22.127 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T01:45:22.127 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2023-11-15T01:45:22.127 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2023-11-15T01:45:22.127 INFO:teuthology.orchestra.run.smithi137.stdout:Upgrading: 2023-11-15T01:45:22.127 INFO:teuthology.orchestra.run.smithi137.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2023-11-15T01:45:22.128 INFO:teuthology.orchestra.run.smithi137.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2023-11-15T01:45:22.128 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2023-11-15T01:45:22.128 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2023-11-15T01:45:22.128 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2023-11-15T01:45:22.128 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2023-11-15T01:45:22.129 INFO:teuthology.orchestra.run.smithi137.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2023-11-15T01:45:22.129 INFO:teuthology.orchestra.run.smithi137.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2023-11-15T01:45:22.129 INFO:teuthology.orchestra.run.smithi137.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-11-15T01:45:22.129 INFO:teuthology.orchestra.run.smithi137.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2023-11-15T01:45:22.129 INFO:teuthology.orchestra.run.smithi137.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2023-11-15T01:45:22.130 INFO:teuthology.orchestra.run.smithi137.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-11-15T01:45:22.130 INFO:teuthology.orchestra.run.smithi137.stdout: libpmemobj x86_64 1.6.1-1.el8 appstream 145 k 2023-11-15T01:45:22.130 INFO:teuthology.orchestra.run.smithi137.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2023-11-15T01:45:22.130 INFO:teuthology.orchestra.run.smithi137.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2023-11-15T01:45:22.130 INFO:teuthology.orchestra.run.smithi137.stdout: librdkafka x86_64 0.11.4-3.el8 appstream 354 k 2023-11-15T01:45:22.131 INFO:teuthology.orchestra.run.smithi137.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2023-11-15T01:45:22.131 INFO:teuthology.orchestra.run.smithi137.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-11-15T01:45:22.131 INFO:teuthology.orchestra.run.smithi137.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2023-11-15T01:45:22.131 INFO:teuthology.orchestra.run.smithi137.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2023-11-15T01:45:22.131 INFO:teuthology.orchestra.run.smithi137.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2023-11-15T01:45:22.132 INFO:teuthology.orchestra.run.smithi137.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2023-11-15T01:45:22.132 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2023-11-15T01:45:22.132 INFO:teuthology.orchestra.run.smithi137.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2023-11-15T01:45:22.132 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2023-11-15T01:45:22.132 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2023-11-15T01:45:22.132 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2023-11-15T01:45:22.133 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:45:22.133 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2023-11-15T01:45:22.133 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T01:45:22.133 INFO:teuthology.orchestra.run.smithi137.stdout:Install 25 Packages 2023-11-15T01:45:22.133 INFO:teuthology.orchestra.run.smithi137.stdout:Upgrade 2 Packages 2023-11-15T01:45:22.133 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:45:22.134 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 59 M 2023-11-15T01:45:22.134 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2023-11-15T01:45:22.550 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2023-11-15T01:45:22.727 INFO:teuthology.orchestra.run.smithi055.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2023-11-15T01:45:22.782 INFO:teuthology.orchestra.run.smithi137.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 42 MB/s | 6.0 MB 00:00 2023-11-15T01:45:22.807 INFO:teuthology.orchestra.run.smithi137.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 997 kB/s | 25 kB 00:00 2023-11-15T01:45:22.833 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2023-11-15T01:45:22.840 INFO:teuthology.orchestra.run.smithi137.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 24 MB/s | 801 kB 00:00 2023-11-15T01:45:22.882 INFO:teuthology.orchestra.run.smithi137.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 11 MB/s | 486 kB 00:00 2023-11-15T01:45:22.991 INFO:teuthology.orchestra.run.smithi137.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 38 MB/s | 4.1 MB 00:00 2023-11-15T01:45:23.058 INFO:teuthology.orchestra.run.smithi137.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 669 kB/s | 44 kB 00:00 2023-11-15T01:45:23.075 INFO:teuthology.orchestra.run.smithi137.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 4.6 MB/s | 80 kB 00:00 2023-11-15T01:45:23.100 INFO:teuthology.orchestra.run.smithi137.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 8.3 MB/s | 213 kB 00:00 2023-11-15T01:45:23.102 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2023-11-15T01:45:23.142 INFO:teuthology.orchestra.run.smithi137.stdout:(9/27): python3-rados-16.2.4-0.el8.x86_64.rpm 9.1 MB/s | 387 kB 00:00 2023-11-15T01:45:23.168 INFO:teuthology.orchestra.run.smithi137.stdout:(10/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 14 MB/s | 368 kB 00:00 2023-11-15T01:45:23.193 INFO:teuthology.orchestra.run.smithi137.stdout:(11/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 4.5 MB/s | 114 kB 00:00 2023-11-15T01:45:23.269 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2023-11-15T01:45:23.325 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2023-11-15T01:45:23.327 INFO:teuthology.orchestra.run.smithi137.stdout:(12/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 18 MB/s | 13 MB 00:00 2023-11-15T01:45:23.521 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 7/29 2023-11-15T01:45:23.627 INFO:teuthology.orchestra.run.smithi137.stdout:(13/27): ceph-common-16.2.4-0.el8.x86_64.rpm 24 MB/s | 24 MB 00:00 2023-11-15T01:45:23.653 INFO:teuthology.orchestra.run.smithi137.stdout:(14/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 316 kB/s | 145 kB 00:00 2023-11-15T01:45:23.660 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 8/29 2023-11-15T01:45:23.677 INFO:teuthology.orchestra.run.smithi137.stdout:(15/27): librdkafka-0.11.4-3.el8.x86_64.rpm 1.0 MB/s | 354 kB 00:00 2023-11-15T01:45:23.694 INFO:teuthology.orchestra.run.smithi137.stdout:(16/27): python3-prettytable-0.7.2-14.el8.noarc 1.0 MB/s | 44 kB 00:00 2023-11-15T01:45:23.745 INFO:teuthology.orchestra.run.smithi137.stdout:(17/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 2.2 MB/s | 259 kB 00:00 2023-11-15T01:45:24.087 INFO:teuthology.orchestra.run.smithi137.stdout:(18/27): daxctl-libs-71.1-7.el8.x86_64.rpm 103 kB/s | 42 kB 00:00 2023-11-15T01:45:24.120 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 8/29 2023-11-15T01:45:24.129 INFO:teuthology.orchestra.run.smithi137.stdout:(19/27): librabbitmq-0.9.0-4.el8.x86_64.rpm 108 kB/s | 47 kB 00:00 2023-11-15T01:45:24.154 INFO:teuthology.orchestra.run.smithi137.stdout:(20/27): ndctl-libs-71.1-7.el8.x86_64.rpm 196 kB/s | 80 kB 00:00 2023-11-15T01:45:24.251 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2023-11-15T01:45:24.333 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2023-11-15T01:45:24.500 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2023-11-15T01:45:24.646 INFO:teuthology.orchestra.run.smithi137.stdout:(21/27): fmt-6.2.1-1.el8.x86_64.rpm 208 kB/s | 116 kB 00:00 2023-11-15T01:45:24.668 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 11/29 2023-11-15T01:45:24.696 INFO:teuthology.orchestra.run.smithi137.stdout:(22/27): leveldb-1.22-1.el8.x86_64.rpm 334 kB/s | 181 kB 00:00 2023-11-15T01:45:24.739 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 11/29 2023-11-15T01:45:24.763 INFO:teuthology.orchestra.run.smithi137.stdout:(23/27): gperftools-libs-2.7-9.el8.x86_64.rpm 483 kB/s | 306 kB 00:00 2023-11-15T01:45:24.788 INFO:teuthology.orchestra.run.smithi137.stdout:(24/27): liboath-2.6.2-3.el8.x86_64.rpm 414 kB/s | 59 kB 00:00 2023-11-15T01:45:24.822 INFO:teuthology.orchestra.run.smithi137.stdout:(25/27): libunwind-1.3.1-3.el8.x86_64.rpm 599 kB/s | 75 kB 00:00 2023-11-15T01:45:24.901 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2023-11-15T01:45:25.062 INFO:teuthology.orchestra.run.smithi137.stdout:(26/27): librados2-16.2.4-0.el8.x86_64.rpm 13 MB/s | 3.7 MB 00:00 2023-11-15T01:45:25.079 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2023-11-15T01:45:25.235 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 14/29 2023-11-15T01:45:25.306 INFO:teuthology.orchestra.run.smithi137.stdout:(27/27): librbd1-16.2.4-0.el8.x86_64.rpm 7.8 MB/s | 4.0 MB 00:00 2023-11-15T01:45:25.311 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2023-11-15T01:45:25.311 INFO:teuthology.orchestra.run.smithi137.stdout:Total 18 MB/s | 59 MB 00:03 2023-11-15T01:45:25.316 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 14/29 2023-11-15T01:45:25.380 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2023-11-15T01:45:25.429 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2023-11-15T01:45:25.429 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2023-11-15T01:45:25.443 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2023-11-15T01:45:25.774 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2023-11-15T01:45:25.777 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2023-11-15T01:45:25.905 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2023-11-15T01:45:26.084 INFO:teuthology.orchestra.run.smithi055.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2023-11-15T01:45:26.188 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2023-11-15T01:45:26.365 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2023-11-15T01:45:26.584 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2023-11-15T01:45:26.734 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2023-11-15T01:45:26.807 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2023-11-15T01:45:26.938 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2023-11-15T01:45:26.982 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2023-11-15T01:45:27.136 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2023-11-15T01:45:27.178 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2023-11-15T01:45:27.324 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2023-11-15T01:45:27.389 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2023-11-15T01:45:27.518 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2023-11-15T01:45:27.594 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2023-11-15T01:45:27.988 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2023-11-15T01:45:28.195 INFO:teuthology.orchestra.run.smithi137.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2023-11-15T01:45:28.272 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2023-11-15T01:45:28.493 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2023-11-15T01:45:28.661 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2023-11-15T01:45:28.710 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2023-11-15T01:45:28.900 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 7/29 2023-11-15T01:45:29.110 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 8/29 2023-11-15T01:45:29.533 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 8/29 2023-11-15T01:45:29.685 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2023-11-15T01:45:29.767 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2023-11-15T01:45:29.982 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2023-11-15T01:45:30.144 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 11/29 2023-11-15T01:45:30.208 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 11/29 2023-11-15T01:45:30.301 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-11-15T01:45:30.357 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2023-11-15T01:45:30.510 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-11-15T01:45:30.611 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2023-11-15T01:45:30.777 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 14/29 2023-11-15T01:45:30.843 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 14/29 2023-11-15T01:45:31.027 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2023-11-15T01:45:31.323 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-11-15T01:45:31.444 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2023-11-15T01:45:31.624 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2023-11-15T01:45:31.627 INFO:teuthology.orchestra.run.smithi137.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2023-11-15T01:45:31.710 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2023-11-15T01:45:31.742 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2023-11-15T01:45:31.876 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2023-11-15T01:45:31.922 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2023-11-15T01:45:32.136 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2023-11-15T01:45:32.285 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2023-11-15T01:45:32.347 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2023-11-15T01:45:32.585 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2023-11-15T01:45:32.759 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2023-11-15T01:45:32.984 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2023-11-15T01:45:33.169 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2023-11-15T01:45:35.761 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-11-15T01:45:36.043 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-11-15T01:45:36.759 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-11-15T01:45:37.031 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2023-11-15T01:45:37.127 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2023-11-15T01:45:37.309 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2023-11-15T01:45:53.075 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2023-11-15T01:45:53.076 INFO:teuthology.orchestra.run.smithi055.stdout:skipping the directory /sys 2023-11-15T01:45:53.076 INFO:teuthology.orchestra.run.smithi055.stdout:skipping the directory /proc 2023-11-15T01:45:53.076 INFO:teuthology.orchestra.run.smithi055.stdout:skipping the directory /dev 2023-11-15T01:45:53.076 INFO:teuthology.orchestra.run.smithi055.stdout:skipping the directory /run 2023-11-15T01:45:53.077 INFO:teuthology.orchestra.run.smithi055.stdout:skipping the directory /mnt 2023-11-15T01:45:53.077 INFO:teuthology.orchestra.run.smithi055.stdout:skipping the directory /var/tmp 2023-11-15T01:45:53.077 INFO:teuthology.orchestra.run.smithi055.stdout:skipping the directory /home 2023-11-15T01:45:53.077 INFO:teuthology.orchestra.run.smithi055.stdout:skipping the directory /tmp 2023-11-15T01:45:53.077 INFO:teuthology.orchestra.run.smithi055.stdout:skipping the directory /dev 2023-11-15T01:45:53.078 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-15T01:45:54.513 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2023-11-15T01:45:54.602 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2023-11-15T01:45:54.602 INFO:teuthology.orchestra.run.smithi055.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2023-11-15T01:45:54.801 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2023-11-15T01:45:54.802 INFO:teuthology.orchestra.run.smithi055.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2023-11-15T01:45:55.539 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2023-11-15T01:45:55.539 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2023-11-15T01:45:55.539 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2023-11-15T01:45:55.540 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2023-11-15T01:45:55.540 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2023-11-15T01:45:55.540 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2023-11-15T01:45:55.540 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2023-11-15T01:45:55.540 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2023-11-15T01:45:55.541 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2023-11-15T01:45:55.541 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2023-11-15T01:45:55.541 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2023-11-15T01:45:55.541 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2023-11-15T01:45:55.541 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2023-11-15T01:45:55.542 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2023-11-15T01:45:55.542 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2023-11-15T01:45:55.542 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 15/29 2023-11-15T01:45:55.542 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2023-11-15T01:45:55.542 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2023-11-15T01:45:55.543 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 18/29 2023-11-15T01:45:55.543 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 19/29 2023-11-15T01:45:55.543 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 20/29 2023-11-15T01:45:55.543 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2023-11-15T01:45:55.543 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2023-11-15T01:45:55.544 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2023-11-15T01:45:55.544 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2023-11-15T01:45:55.544 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2023-11-15T01:45:55.544 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2023-11-15T01:45:55.544 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2023-11-15T01:45:55.545 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2023-11-15T01:45:55.852 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2023-11-15T01:45:55.852 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /sys 2023-11-15T01:45:55.852 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /proc 2023-11-15T01:45:55.852 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /dev 2023-11-15T01:45:55.852 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /run 2023-11-15T01:45:55.853 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /mnt 2023-11-15T01:45:55.853 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /var/tmp 2023-11-15T01:45:55.853 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /home 2023-11-15T01:45:55.853 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /tmp 2023-11-15T01:45:55.853 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /dev 2023-11-15T01:45:55.854 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:45:56.820 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2023-11-15T01:45:56.820 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-15T01:45:56.821 INFO:teuthology.orchestra.run.smithi055.stdout:Upgraded: 2023-11-15T01:45:56.821 INFO:teuthology.orchestra.run.smithi055.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2023-11-15T01:45:56.821 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2023-11-15T01:45:56.821 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2023-11-15T01:45:56.822 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2023-11-15T01:45:56.822 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2023-11-15T01:45:56.822 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2023-11-15T01:45:56.822 INFO:teuthology.orchestra.run.smithi055.stdout: daxctl-libs-71.1-7.el8.x86_64 2023-11-15T01:45:56.822 INFO:teuthology.orchestra.run.smithi055.stdout: fmt-6.2.1-1.el8.x86_64 2023-11-15T01:45:56.822 INFO:teuthology.orchestra.run.smithi055.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-11-15T01:45:56.823 INFO:teuthology.orchestra.run.smithi055.stdout: leveldb-1.22-1.el8.x86_64 2023-11-15T01:45:56.823 INFO:teuthology.orchestra.run.smithi055.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2023-11-15T01:45:56.823 INFO:teuthology.orchestra.run.smithi055.stdout: liboath-2.6.2-3.el8.x86_64 2023-11-15T01:45:56.823 INFO:teuthology.orchestra.run.smithi055.stdout: libpmemobj-1.6.1-1.el8.x86_64 2023-11-15T01:45:56.823 INFO:teuthology.orchestra.run.smithi055.stdout: librabbitmq-0.9.0-4.el8.x86_64 2023-11-15T01:45:56.824 INFO:teuthology.orchestra.run.smithi055.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2023-11-15T01:45:56.824 INFO:teuthology.orchestra.run.smithi055.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-11-15T01:45:56.824 INFO:teuthology.orchestra.run.smithi055.stdout: librgw2-2:16.2.4-0.el8.x86_64 2023-11-15T01:45:56.824 INFO:teuthology.orchestra.run.smithi055.stdout: libunwind-1.3.1-3.el8.x86_64 2023-11-15T01:45:56.824 INFO:teuthology.orchestra.run.smithi055.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-11-15T01:45:56.825 INFO:teuthology.orchestra.run.smithi055.stdout: ndctl-libs-71.1-7.el8.x86_64 2023-11-15T01:45:56.825 INFO:teuthology.orchestra.run.smithi055.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2023-11-15T01:45:56.825 INFO:teuthology.orchestra.run.smithi055.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2023-11-15T01:45:56.825 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2023-11-15T01:45:56.825 INFO:teuthology.orchestra.run.smithi055.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-11-15T01:45:56.826 INFO:teuthology.orchestra.run.smithi055.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2023-11-15T01:45:56.826 INFO:teuthology.orchestra.run.smithi055.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2023-11-15T01:45:56.826 INFO:teuthology.orchestra.run.smithi055.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2023-11-15T01:45:56.826 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-15T01:45:56.826 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2023-11-15T01:45:56.986 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph-test 2023-11-15T01:45:57.219 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2023-11-15T01:45:57.282 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2023-11-15T01:45:57.283 INFO:teuthology.orchestra.run.smithi137.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2023-11-15T01:45:57.475 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2023-11-15T01:45:57.475 INFO:teuthology.orchestra.run.smithi137.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2023-11-15T01:45:57.977 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:00:45 ago on Wed 15 Nov 2023 01:45:12 AM UTC. 2023-11-15T01:45:58.225 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2023-11-15T01:45:58.226 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2023-11-15T01:45:58.226 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2023-11-15T01:45:58.226 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2023-11-15T01:45:58.227 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2023-11-15T01:45:58.227 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2023-11-15T01:45:58.227 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2023-11-15T01:45:58.227 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2023-11-15T01:45:58.228 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2023-11-15T01:45:58.228 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2023-11-15T01:45:58.228 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2023-11-15T01:45:58.228 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2023-11-15T01:45:58.229 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2023-11-15T01:45:58.229 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2023-11-15T01:45:58.229 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2023-11-15T01:45:58.229 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 15/29 2023-11-15T01:45:58.229 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2023-11-15T01:45:58.230 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2023-11-15T01:45:58.230 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 18/29 2023-11-15T01:45:58.230 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 19/29 2023-11-15T01:45:58.230 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 20/29 2023-11-15T01:45:58.231 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2023-11-15T01:45:58.231 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2023-11-15T01:45:58.231 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2023-11-15T01:45:58.231 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2023-11-15T01:45:58.232 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2023-11-15T01:45:58.232 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2023-11-15T01:45:58.232 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2023-11-15T01:45:58.232 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2023-11-15T01:45:59.371 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2023-11-15T01:45:59.371 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:45:59.371 INFO:teuthology.orchestra.run.smithi137.stdout:Upgraded: 2023-11-15T01:45:59.372 INFO:teuthology.orchestra.run.smithi137.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2023-11-15T01:45:59.372 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2023-11-15T01:45:59.372 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2023-11-15T01:45:59.372 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2023-11-15T01:45:59.372 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2023-11-15T01:45:59.373 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2023-11-15T01:45:59.373 INFO:teuthology.orchestra.run.smithi137.stdout: daxctl-libs-71.1-7.el8.x86_64 2023-11-15T01:45:59.373 INFO:teuthology.orchestra.run.smithi137.stdout: fmt-6.2.1-1.el8.x86_64 2023-11-15T01:45:59.373 INFO:teuthology.orchestra.run.smithi137.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-11-15T01:45:59.373 INFO:teuthology.orchestra.run.smithi137.stdout: leveldb-1.22-1.el8.x86_64 2023-11-15T01:45:59.374 INFO:teuthology.orchestra.run.smithi137.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2023-11-15T01:45:59.374 INFO:teuthology.orchestra.run.smithi137.stdout: liboath-2.6.2-3.el8.x86_64 2023-11-15T01:45:59.374 INFO:teuthology.orchestra.run.smithi137.stdout: libpmemobj-1.6.1-1.el8.x86_64 2023-11-15T01:45:59.374 INFO:teuthology.orchestra.run.smithi137.stdout: librabbitmq-0.9.0-4.el8.x86_64 2023-11-15T01:45:59.374 INFO:teuthology.orchestra.run.smithi137.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2023-11-15T01:45:59.375 INFO:teuthology.orchestra.run.smithi137.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-11-15T01:45:59.375 INFO:teuthology.orchestra.run.smithi137.stdout: librgw2-2:16.2.4-0.el8.x86_64 2023-11-15T01:45:59.375 INFO:teuthology.orchestra.run.smithi137.stdout: libunwind-1.3.1-3.el8.x86_64 2023-11-15T01:45:59.376 INFO:teuthology.orchestra.run.smithi137.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-11-15T01:45:59.376 INFO:teuthology.orchestra.run.smithi137.stdout: ndctl-libs-71.1-7.el8.x86_64 2023-11-15T01:45:59.376 INFO:teuthology.orchestra.run.smithi137.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2023-11-15T01:45:59.376 INFO:teuthology.orchestra.run.smithi137.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2023-11-15T01:45:59.376 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2023-11-15T01:45:59.377 INFO:teuthology.orchestra.run.smithi137.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-11-15T01:45:59.377 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2023-11-15T01:45:59.377 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2023-11-15T01:45:59.377 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2023-11-15T01:45:59.378 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:45:59.378 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2023-11-15T01:45:59.603 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-test 2023-11-15T01:45:59.821 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2023-11-15T01:45:59.822 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-15T01:45:59.823 INFO:teuthology.orchestra.run.smithi055.stdout: Package Architecture Version Repository Size 2023-11-15T01:45:59.823 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-15T01:45:59.823 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2023-11-15T01:45:59.823 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2023-11-15T01:45:59.824 INFO:teuthology.orchestra.run.smithi055.stdout:Installing dependencies: 2023-11-15T01:45:59.824 INFO:teuthology.orchestra.run.smithi055.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2023-11-15T01:45:59.824 INFO:teuthology.orchestra.run.smithi055.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2023-11-15T01:45:59.824 INFO:teuthology.orchestra.run.smithi055.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2023-11-15T01:45:59.824 INFO:teuthology.orchestra.run.smithi055.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2023-11-15T01:45:59.824 INFO:teuthology.orchestra.run.smithi055.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2023-11-15T01:45:59.825 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-15T01:45:59.825 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2023-11-15T01:45:59.825 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-15T01:45:59.825 INFO:teuthology.orchestra.run.smithi055.stdout:Install 6 Packages 2023-11-15T01:45:59.825 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-15T01:45:59.826 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 53 M 2023-11-15T01:45:59.826 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 229 M 2023-11-15T01:45:59.827 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2023-11-15T01:45:59.954 INFO:teuthology.orchestra.run.smithi055.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 1.8 MB/s | 169 kB 00:00 2023-11-15T01:46:00.088 INFO:teuthology.orchestra.run.smithi055.stdout:(2/6): jq-1.6-8.el8.x86_64.rpm 902 kB/s | 203 kB 00:00 2023-11-15T01:46:00.180 INFO:teuthology.orchestra.run.smithi055.stdout:(3/6): oniguruma-6.8.2-2.el8.x86_64.rpm 831 kB/s | 187 kB 00:00 2023-11-15T01:46:00.205 INFO:teuthology.orchestra.run.smithi055.stdout:(4/6): socat-1.7.4.1-1.el8.x86_64.rpm 2.7 MB/s | 323 kB 00:00 2023-11-15T01:46:00.239 INFO:teuthology.orchestra.run.smithi055.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.2 MB/s | 71 kB 00:00 2023-11-15T01:46:00.455 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:00:43 ago on Wed 15 Nov 2023 01:45:17 AM UTC. 2023-11-15T01:46:00.739 INFO:teuthology.orchestra.run.smithi055.stdout:(6/6): ceph-test-16.2.4-0.el8.x86_64.rpm 59 MB/s | 52 MB 00:00 2023-11-15T01:46:00.741 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2023-11-15T01:46:00.742 INFO:teuthology.orchestra.run.smithi055.stdout:Total 58 MB/s | 53 MB 00:00 2023-11-15T01:46:00.787 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2023-11-15T01:46:00.811 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2023-11-15T01:46:00.811 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2023-11-15T01:46:01.146 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2023-11-15T01:46:01.150 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2023-11-15T01:46:01.657 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2023-11-15T01:46:01.888 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2023-11-15T01:46:02.078 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2023-11-15T01:46:02.114 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2023-11-15T01:46:02.115 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T01:46:02.115 INFO:teuthology.orchestra.run.smithi137.stdout: Package Architecture Version Repository Size 2023-11-15T01:46:02.115 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T01:46:02.116 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2023-11-15T01:46:02.116 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2023-11-15T01:46:02.116 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2023-11-15T01:46:02.116 INFO:teuthology.orchestra.run.smithi137.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2023-11-15T01:46:02.116 INFO:teuthology.orchestra.run.smithi137.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2023-11-15T01:46:02.117 INFO:teuthology.orchestra.run.smithi137.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2023-11-15T01:46:02.117 INFO:teuthology.orchestra.run.smithi137.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2023-11-15T01:46:02.117 INFO:teuthology.orchestra.run.smithi137.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2023-11-15T01:46:02.117 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:46:02.117 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2023-11-15T01:46:02.118 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T01:46:02.118 INFO:teuthology.orchestra.run.smithi137.stdout:Install 6 Packages 2023-11-15T01:46:02.118 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:46:02.119 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 53 M 2023-11-15T01:46:02.119 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 229 M 2023-11-15T01:46:02.119 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2023-11-15T01:46:02.222 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2023-11-15T01:46:02.224 INFO:teuthology.orchestra.run.smithi137.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 2.4 MB/s | 169 kB 00:00 2023-11-15T01:46:02.305 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2023-11-15T01:46:02.375 INFO:teuthology.orchestra.run.smithi137.stdout:(2/6): jq-1.6-8.el8.x86_64.rpm 933 kB/s | 203 kB 00:00 2023-11-15T01:46:02.416 INFO:teuthology.orchestra.run.smithi137.stdout:(3/6): oniguruma-6.8.2-2.el8.x86_64.rpm 977 kB/s | 187 kB 00:00 2023-11-15T01:46:02.475 INFO:teuthology.orchestra.run.smithi137.stdout:(4/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.2 MB/s | 71 kB 00:00 2023-11-15T01:46:02.481 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2023-11-15T01:46:02.559 INFO:teuthology.orchestra.run.smithi137.stdout:(5/6): socat-1.7.4.1-1.el8.x86_64.rpm 1.7 MB/s | 323 kB 00:00 2023-11-15T01:46:02.613 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2023-11-15T01:46:03.059 INFO:teuthology.orchestra.run.smithi137.stdout:(6/6): ceph-test-16.2.4-0.el8.x86_64.rpm 58 MB/s | 52 MB 00:00 2023-11-15T01:46:03.061 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2023-11-15T01:46:03.062 INFO:teuthology.orchestra.run.smithi137.stdout:Total 56 MB/s | 53 MB 00:00 2023-11-15T01:46:03.105 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2023-11-15T01:46:03.126 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2023-11-15T01:46:03.126 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2023-11-15T01:46:03.430 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2023-11-15T01:46:03.433 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2023-11-15T01:46:03.955 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2023-11-15T01:46:04.199 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2023-11-15T01:46:04.407 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2023-11-15T01:46:04.621 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2023-11-15T01:46:04.706 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2023-11-15T01:46:04.879 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2023-11-15T01:46:05.088 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2023-11-15T01:46:07.942 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2023-11-15T01:46:08.154 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2023-11-15T01:46:08.490 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2023-11-15T01:46:08.490 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2023-11-15T01:46:08.490 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2023-11-15T01:46:08.491 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2023-11-15T01:46:08.491 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2023-11-15T01:46:08.491 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2023-11-15T01:46:09.258 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2023-11-15T01:46:09.259 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-15T01:46:09.259 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2023-11-15T01:46:09.259 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-8.el8.x86_64 2023-11-15T01:46:09.259 INFO:teuthology.orchestra.run.smithi055.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-2.el8.x86_64 2023-11-15T01:46:09.260 INFO:teuthology.orchestra.run.smithi055.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2023-11-15T01:46:09.260 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-15T01:46:09.260 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2023-11-15T01:46:09.489 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph 2023-11-15T01:46:09.968 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2023-11-15T01:46:10.194 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2023-11-15T01:46:10.472 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:00:58 ago on Wed 15 Nov 2023 01:45:12 AM UTC. 2023-11-15T01:46:10.537 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2023-11-15T01:46:10.538 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2023-11-15T01:46:10.538 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2023-11-15T01:46:10.538 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2023-11-15T01:46:10.539 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2023-11-15T01:46:10.539 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2023-11-15T01:46:11.260 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2023-11-15T01:46:11.261 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:46:11.261 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2023-11-15T01:46:11.261 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-8.el8.x86_64 2023-11-15T01:46:11.261 INFO:teuthology.orchestra.run.smithi137.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-2.el8.x86_64 2023-11-15T01:46:11.262 INFO:teuthology.orchestra.run.smithi137.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2023-11-15T01:46:11.262 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:46:11.262 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2023-11-15T01:46:11.485 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph 2023-11-15T01:46:12.327 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2023-11-15T01:46:12.331 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-15T01:46:12.331 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repository Size 2023-11-15T01:46:12.331 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-15T01:46:12.331 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2023-11-15T01:46:12.331 INFO:teuthology.orchestra.run.smithi055.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2023-11-15T01:46:12.332 INFO:teuthology.orchestra.run.smithi055.stdout:Installing dependencies: 2023-11-15T01:46:12.332 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2023-11-15T01:46:12.332 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2023-11-15T01:46:12.332 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2023-11-15T01:46:12.333 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2023-11-15T01:46:12.333 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2023-11-15T01:46:12.333 INFO:teuthology.orchestra.run.smithi055.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-11-15T01:46:12.333 INFO:teuthology.orchestra.run.smithi055.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-11-15T01:46:12.333 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2023-11-15T01:46:12.333 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-11-15T01:46:12.334 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-11-15T01:46:12.334 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cryptography x86_64 3.2.1-6.el8 baseos 641 k 2023-11-15T01:46:12.334 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-11-15T01:46:12.334 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-11-15T01:46:12.335 INFO:teuthology.orchestra.run.smithi055.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-11-15T01:46:12.335 INFO:teuthology.orchestra.run.smithi055.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2023-11-15T01:46:12.335 INFO:teuthology.orchestra.run.smithi055.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2023-11-15T01:46:12.335 INFO:teuthology.orchestra.run.smithi055.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-11-15T01:46:12.336 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-11-15T01:46:12.336 INFO:teuthology.orchestra.run.smithi055.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-11-15T01:46:12.336 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2023-11-15T01:46:12.336 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2023-11-15T01:46:12.336 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2023-11-15T01:46:12.337 INFO:teuthology.orchestra.run.smithi055.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-11-15T01:46:12.337 INFO:teuthology.orchestra.run.smithi055.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-11-15T01:46:12.337 INFO:teuthology.orchestra.run.smithi055.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-11-15T01:46:12.337 INFO:teuthology.orchestra.run.smithi055.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-11-15T01:46:12.337 INFO:teuthology.orchestra.run.smithi055.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-11-15T01:46:12.338 INFO:teuthology.orchestra.run.smithi055.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-11-15T01:46:12.338 INFO:teuthology.orchestra.run.smithi055.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-11-15T01:46:12.338 INFO:teuthology.orchestra.run.smithi055.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2023-11-15T01:46:12.338 INFO:teuthology.orchestra.run.smithi055.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-11-15T01:46:12.338 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-15T01:46:12.339 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2023-11-15T01:46:12.339 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-15T01:46:12.339 INFO:teuthology.orchestra.run.smithi055.stdout:Install 32 Packages 2023-11-15T01:46:12.339 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-15T01:46:12.340 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 31 M 2023-11-15T01:46:12.340 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 111 M 2023-11-15T01:46:12.340 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2023-11-15T01:46:12.344 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:00:55 ago on Wed 15 Nov 2023 01:45:17 AM UTC. 2023-11-15T01:46:12.992 INFO:teuthology.orchestra.run.smithi055.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 108 kB/s | 6.3 kB 00:00 2023-11-15T01:46:13.050 INFO:teuthology.orchestra.run.smithi055.stdout:(2/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2023-11-15T01:46:13.101 INFO:teuthology.orchestra.run.smithi055.stdout:(3/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 14 MB/s | 2.4 MB 00:00 2023-11-15T01:46:13.126 INFO:teuthology.orchestra.run.smithi055.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 8.7 MB/s | 220 kB 00:00 2023-11-15T01:46:13.210 INFO:teuthology.orchestra.run.smithi055.stdout:(5/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 23 MB/s | 5.0 MB 00:00 2023-11-15T01:46:13.243 INFO:teuthology.orchestra.run.smithi055.stdout:(6/32): python3-mako-1.0.6-14.el8.noarch.rpm 1.3 MB/s | 157 kB 00:00 2023-11-15T01:46:13.293 INFO:teuthology.orchestra.run.smithi055.stdout:(7/32): python3-markupsafe-0.23-19.el8.x86_64.r 465 kB/s | 39 kB 00:00 2023-11-15T01:46:13.344 INFO:teuthology.orchestra.run.smithi055.stdout:(8/32): python3-pytz-2017.2-11.el8.noarch.rpm 1.1 MB/s | 57 kB 00:00 2023-11-15T01:46:13.369 INFO:teuthology.orchestra.run.smithi055.stdout:(9/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 824 kB/s | 103 kB 00:00 2023-11-15T01:46:13.561 INFO:teuthology.orchestra.run.smithi055.stdout:(10/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 36 MB/s | 18 MB 00:00 2023-11-15T01:46:13.661 INFO:teuthology.orchestra.run.smithi055.stdout:(11/32): python3-werkzeug-0.12.2-4.el8.noarch.r 1.4 MB/s | 457 kB 00:00 2023-11-15T01:46:13.978 INFO:teuthology.orchestra.run.smithi055.stdout:(12/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 408 kB/s | 249 kB 00:00 2023-11-15T01:46:14.010 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2023-11-15T01:46:14.013 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T01:46:14.013 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2023-11-15T01:46:14.014 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T01:46:14.014 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2023-11-15T01:46:14.014 INFO:teuthology.orchestra.run.smithi137.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2023-11-15T01:46:14.014 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2023-11-15T01:46:14.015 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2023-11-15T01:46:14.015 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2023-11-15T01:46:14.015 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2023-11-15T01:46:14.015 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2023-11-15T01:46:14.015 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2023-11-15T01:46:14.016 INFO:teuthology.orchestra.run.smithi137.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-11-15T01:46:14.016 INFO:teuthology.orchestra.run.smithi137.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-11-15T01:46:14.016 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2023-11-15T01:46:14.016 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-11-15T01:46:14.016 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-11-15T01:46:14.016 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cryptography x86_64 3.2.1-6.el8 baseos 641 k 2023-11-15T01:46:14.017 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-11-15T01:46:14.017 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-11-15T01:46:14.017 INFO:teuthology.orchestra.run.smithi137.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-11-15T01:46:14.017 INFO:teuthology.orchestra.run.smithi137.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2023-11-15T01:46:14.017 INFO:teuthology.orchestra.run.smithi137.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2023-11-15T01:46:14.018 INFO:teuthology.orchestra.run.smithi137.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-11-15T01:46:14.018 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-11-15T01:46:14.018 INFO:teuthology.orchestra.run.smithi137.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-11-15T01:46:14.018 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2023-11-15T01:46:14.018 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2023-11-15T01:46:14.019 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2023-11-15T01:46:14.019 INFO:teuthology.orchestra.run.smithi137.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-11-15T01:46:14.019 INFO:teuthology.orchestra.run.smithi137.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-11-15T01:46:14.019 INFO:teuthology.orchestra.run.smithi137.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-11-15T01:46:14.019 INFO:teuthology.orchestra.run.smithi137.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-11-15T01:46:14.020 INFO:teuthology.orchestra.run.smithi137.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-11-15T01:46:14.020 INFO:teuthology.orchestra.run.smithi137.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-11-15T01:46:14.020 INFO:teuthology.orchestra.run.smithi137.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-11-15T01:46:14.020 INFO:teuthology.orchestra.run.smithi137.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2023-11-15T01:46:14.020 INFO:teuthology.orchestra.run.smithi137.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-11-15T01:46:14.020 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:46:14.021 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2023-11-15T01:46:14.021 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T01:46:14.021 INFO:teuthology.orchestra.run.smithi137.stdout:Install 32 Packages 2023-11-15T01:46:14.021 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:46:14.022 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 31 M 2023-11-15T01:46:14.022 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 111 M 2023-11-15T01:46:14.022 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2023-11-15T01:46:14.104 INFO:teuthology.orchestra.run.smithi055.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 247 kB/s | 109 kB 00:00 2023-11-15T01:46:14.146 INFO:teuthology.orchestra.run.smithi055.stdout:(14/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 261 kB/s | 44 kB 00:00 2023-11-15T01:46:14.171 INFO:teuthology.orchestra.run.smithi055.stdout:(15/32): python3-cryptography-3.2.1-6.el8.x86_6 1.0 MB/s | 641 kB 00:00 2023-11-15T01:46:14.186 INFO:teuthology.orchestra.run.smithi137.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 106 kB/s | 6.3 kB 00:00 2023-11-15T01:46:14.237 INFO:teuthology.orchestra.run.smithi137.stdout:(2/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 14 MB/s | 1.6 MB 00:00 2023-11-15T01:46:14.280 INFO:teuthology.orchestra.run.smithi055.stdout:(16/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 1.3 MB/s | 173 kB 00:00 2023-11-15T01:46:14.287 INFO:teuthology.orchestra.run.smithi137.stdout:(3/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 15 MB/s | 2.4 MB 00:00 2023-11-15T01:46:14.312 INFO:teuthology.orchestra.run.smithi137.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 8.7 MB/s | 220 kB 00:00 2023-11-15T01:46:14.321 INFO:teuthology.orchestra.run.smithi055.stdout:(17/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 852 kB/s | 185 kB 00:00 2023-11-15T01:46:14.347 INFO:teuthology.orchestra.run.smithi055.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 166 kB/s | 11 kB 00:00 2023-11-15T01:46:14.380 INFO:teuthology.orchestra.run.smithi055.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 301 kB/s | 18 kB 00:00 2023-11-15T01:46:14.396 INFO:teuthology.orchestra.run.smithi137.stdout:(5/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 24 MB/s | 5.0 MB 00:00 2023-11-15T01:46:14.405 INFO:teuthology.orchestra.run.smithi055.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 832 kB/s | 49 kB 00:00 2023-11-15T01:46:14.431 INFO:teuthology.orchestra.run.smithi055.stdout:(21/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.4 MB/s | 384 kB 00:00 2023-11-15T01:46:14.438 INFO:teuthology.orchestra.run.smithi137.stdout:(6/32): python3-mako-1.0.6-14.el8.noarch.rpm 1.2 MB/s | 157 kB 00:00 2023-11-15T01:46:14.455 INFO:teuthology.orchestra.run.smithi137.stdout:(7/32): python3-markupsafe-0.23-19.el8.x86_64.r 664 kB/s | 39 kB 00:00 2023-11-15T01:46:14.456 INFO:teuthology.orchestra.run.smithi055.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 777 kB/s | 59 kB 00:00 2023-11-15T01:46:14.490 INFO:teuthology.orchestra.run.smithi055.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 266 kB/s | 16 kB 00:00 2023-11-15T01:46:14.557 INFO:teuthology.orchestra.run.smithi055.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 187 kB/s | 19 kB 00:00 2023-11-15T01:46:14.582 INFO:teuthology.orchestra.run.smithi055.stdout:(25/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.6 MB/s | 283 kB 00:00 2023-11-15T01:46:14.607 INFO:teuthology.orchestra.run.smithi055.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 204 kB/s | 24 kB 00:00 2023-11-15T01:46:14.632 INFO:teuthology.orchestra.run.smithi055.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 388 kB/s | 29 kB 00:00 2023-11-15T01:46:14.658 INFO:teuthology.orchestra.run.smithi055.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 551 kB/s | 27 kB 00:00 2023-11-15T01:46:14.683 INFO:teuthology.orchestra.run.smithi055.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 3.3 MB/s | 255 kB 00:00 2023-11-15T01:46:14.705 INFO:teuthology.orchestra.run.smithi137.stdout:(8/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 39 MB/s | 18 MB 00:00 2023-11-15T01:46:14.717 INFO:teuthology.orchestra.run.smithi055.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 1.4 MB/s | 85 kB 00:00 2023-11-15T01:46:14.722 INFO:teuthology.orchestra.run.smithi137.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 213 kB/s | 57 kB 00:00 2023-11-15T01:46:14.742 INFO:teuthology.orchestra.run.smithi055.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 393 kB/s | 23 kB 00:00 2023-11-15T01:46:14.747 INFO:teuthology.orchestra.run.smithi137.stdout:(10/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 335 kB/s | 103 kB 00:00 2023-11-15T01:46:14.767 INFO:teuthology.orchestra.run.smithi055.stdout:(32/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.8 MB/s | 251 kB 00:00 2023-11-15T01:46:14.769 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2023-11-15T01:46:14.769 INFO:teuthology.orchestra.run.smithi055.stdout:Total 13 MB/s | 31 MB 00:02 2023-11-15T01:46:14.855 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2023-11-15T01:46:14.898 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2023-11-15T01:46:14.898 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2023-11-15T01:46:15.198 INFO:teuthology.orchestra.run.smithi137.stdout:(11/32): python3-werkzeug-0.12.2-4.el8.noarch.r 930 kB/s | 457 kB 00:00 2023-11-15T01:46:15.223 INFO:teuthology.orchestra.run.smithi137.stdout:(12/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 497 kB/s | 249 kB 00:00 2023-11-15T01:46:15.231 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2023-11-15T01:46:15.235 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2023-11-15T01:46:15.348 INFO:teuthology.orchestra.run.smithi137.stdout:(13/32): python3-cryptography-3.2.1-6.el8.x86_6 1.0 MB/s | 641 kB 00:00 2023-11-15T01:46:15.457 INFO:teuthology.orchestra.run.smithi137.stdout:(14/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 187 kB/s | 44 kB 00:00 2023-11-15T01:46:15.582 INFO:teuthology.orchestra.run.smithi137.stdout:(15/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 1.3 MB/s | 173 kB 00:00 2023-11-15T01:46:15.607 INFO:teuthology.orchestra.run.smithi137.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 716 kB/s | 185 kB 00:00 2023-11-15T01:46:15.674 INFO:teuthology.orchestra.run.smithi137.stdout:(17/32): python3-pycparser-2.14-14.el8.noarch.r 229 kB/s | 109 kB 00:00 2023-11-15T01:46:15.775 INFO:teuthology.orchestra.run.smithi137.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 66 kB/s | 11 kB 00:00 2023-11-15T01:46:15.800 INFO:teuthology.orchestra.run.smithi137.stdout:(19/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.7 MB/s | 384 kB 00:00 2023-11-15T01:46:15.842 INFO:teuthology.orchestra.run.smithi137.stdout:(20/32): python3-jaraco-functools-2.0-4.el8.noa 105 kB/s | 18 kB 00:00 2023-11-15T01:46:15.867 INFO:teuthology.orchestra.run.smithi137.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 529 kB/s | 49 kB 00:00 2023-11-15T01:46:15.892 INFO:teuthology.orchestra.run.smithi137.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 636 kB/s | 59 kB 00:00 2023-11-15T01:46:15.942 INFO:teuthology.orchestra.run.smithi137.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 208 kB/s | 16 kB 00:00 2023-11-15T01:46:15.968 INFO:teuthology.orchestra.run.smithi137.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 249 kB/s | 19 kB 00:00 2023-11-15T01:46:16.026 INFO:teuthology.orchestra.run.smithi137.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 285 kB/s | 24 kB 00:00 2023-11-15T01:46:16.052 INFO:teuthology.orchestra.run.smithi137.stdout:(26/32): python3-tempora-1.14.1-5.el8.noarch.rp 350 kB/s | 29 kB 00:00 2023-11-15T01:46:16.143 INFO:teuthology.orchestra.run.smithi137.stdout:(27/32): python3-trustme-0.6.0-4.el8.noarch.rpm 237 kB/s | 27 kB 00:00 2023-11-15T01:46:16.178 INFO:teuthology.orchestra.run.smithi137.stdout:(28/32): python3-waitress-1.4.3-1.el8.noarch.rp 2.0 MB/s | 255 kB 00:00 2023-11-15T01:46:16.261 INFO:teuthology.orchestra.run.smithi137.stdout:(29/32): python3-webtest-2.0.33-1.el8.noarch.rp 1.0 MB/s | 85 kB 00:00 2023-11-15T01:46:16.286 INFO:teuthology.orchestra.run.smithi137.stdout:(30/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.7 MB/s | 251 kB 00:00 2023-11-15T01:46:16.328 INFO:teuthology.orchestra.run.smithi137.stdout:(31/32): python3-pecan-1.3.2-9.el8.noarch.rpm 582 kB/s | 283 kB 00:00 2023-11-15T01:46:16.337 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2023-11-15T01:46:16.353 INFO:teuthology.orchestra.run.smithi137.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 250 kB/s | 23 kB 00:00 2023-11-15T01:46:16.355 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2023-11-15T01:46:16.355 INFO:teuthology.orchestra.run.smithi137.stdout:Total 13 MB/s | 31 MB 00:02 2023-11-15T01:46:16.437 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2023-11-15T01:46:16.476 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2023-11-15T01:46:16.477 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2023-11-15T01:46:16.544 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-11-15T01:46:16.739 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-11-15T01:46:16.806 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2023-11-15T01:46:16.809 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2023-11-15T01:46:16.984 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-11-15T01:46:17.178 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2023-11-15T01:46:17.342 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-11-15T01:46:17.501 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-11-15T01:46:17.632 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-11-15T01:46:17.776 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-11-15T01:46:17.965 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-11-15T01:46:17.976 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2023-11-15T01:46:18.167 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-11-15T01:46:18.224 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-11-15T01:46:18.402 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-11-15T01:46:18.427 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-11-15T01:46:18.664 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-11-15T01:46:18.719 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-11-15T01:46:18.925 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2023-11-15T01:46:19.048 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2023-11-15T01:46:19.140 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-11-15T01:46:19.302 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 14/32 2023-11-15T01:46:19.328 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-11-15T01:46:19.480 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-11-15T01:46:19.481 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-11-15T01:46:19.699 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-11-15T01:46:19.721 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-11-15T01:46:19.873 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-11-15T01:46:19.898 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-11-15T01:46:20.119 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-11-15T01:46:20.193 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-11-15T01:46:20.340 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-11-15T01:46:20.412 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-11-15T01:46:20.592 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2023-11-15T01:46:20.605 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-11-15T01:46:20.898 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-11-15T01:46:20.950 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2023-11-15T01:46:21.133 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-11-15T01:46:21.238 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 14/32 2023-11-15T01:46:21.358 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-11-15T01:46:21.410 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-11-15T01:46:21.592 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-11-15T01:46:21.700 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-11-15T01:46:21.869 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-11-15T01:46:21.950 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2023-11-15T01:46:22.210 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-11-15T01:46:22.257 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-11-15T01:46:22.407 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-11-15T01:46:22.629 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2023-11-15T01:46:22.654 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2023-11-15T01:46:22.791 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-11-15T01:46:22.851 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2023-11-15T01:46:23.073 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-11-15T01:46:23.290 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-11-15T01:46:23.553 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-11-15T01:46:23.897 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2023-11-15T01:46:24.249 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-11-15T01:46:24.547 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2023-11-15T01:46:24.652 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2023-11-15T01:46:24.725 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2023-11-15T01:46:24.848 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2023-11-15T01:46:25.204 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2023-11-15T01:46:25.400 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2023-11-15T01:46:25.647 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2023-11-15T01:46:25.828 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2023-11-15T01:46:25.850 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2023-11-15T01:46:25.971 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2023-11-15T01:46:26.387 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2023-11-15T01:46:26.445 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2023-11-15T01:46:26.445 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2023-11-15T01:46:26.445 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2023-11-15T01:46:26.445 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2023-11-15T01:46:26.446 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2023-11-15T01:46:26.446 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2023-11-15T01:46:26.446 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2023-11-15T01:46:26.446 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2023-11-15T01:46:26.447 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-11-15T01:46:26.447 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-11-15T01:46:26.447 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2023-11-15T01:46:26.447 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-11-15T01:46:26.447 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2023-11-15T01:46:26.447 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 13/32 2023-11-15T01:46:26.448 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-11-15T01:46:26.448 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-11-15T01:46:26.448 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-11-15T01:46:26.448 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-11-15T01:46:26.448 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-11-15T01:46:26.449 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-11-15T01:46:26.449 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-11-15T01:46:26.449 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-11-15T01:46:26.449 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-11-15T01:46:26.450 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-11-15T01:46:26.450 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-11-15T01:46:26.450 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-11-15T01:46:26.451 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-11-15T01:46:26.451 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-11-15T01:46:26.452 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-11-15T01:46:26.452 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2023-11-15T01:46:26.452 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-11-15T01:46:26.452 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-11-15T01:46:26.572 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2023-11-15T01:46:27.005 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2023-11-15T01:46:27.230 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2023-11-15T01:46:27.452 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2023-11-15T01:46:27.660 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2023-11-15T01:46:27.677 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2023-11-15T01:46:27.726 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-11-15T01:46:27.727 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-15T01:46:27.727 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2023-11-15T01:46:27.727 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-2:16.2.4-0.el8.x86_64 2023-11-15T01:46:27.727 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2023-11-15T01:46:27.728 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2023-11-15T01:46:27.728 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2023-11-15T01:46:27.728 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2023-11-15T01:46:27.728 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2023-11-15T01:46:27.729 INFO:teuthology.orchestra.run.smithi055.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-11-15T01:46:27.729 INFO:teuthology.orchestra.run.smithi055.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-11-15T01:46:27.729 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cffi-1.11.5-6.el8.x86_64 2023-11-15T01:46:27.730 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-11-15T01:46:27.730 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-11-15T01:46:27.730 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2023-11-15T01:46:27.730 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco-6.2-6.el8.noarch 2023-11-15T01:46:27.731 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-11-15T01:46:27.731 INFO:teuthology.orchestra.run.smithi055.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-11-15T01:46:27.731 INFO:teuthology.orchestra.run.smithi055.stdout: python3-mako-1.0.6-14.el8.noarch 2023-11-15T01:46:27.731 INFO:teuthology.orchestra.run.smithi055.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-11-15T01:46:27.731 INFO:teuthology.orchestra.run.smithi055.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-11-15T01:46:27.732 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-11-15T01:46:27.732 INFO:teuthology.orchestra.run.smithi055.stdout: python3-portend-2.6-1.el8.noarch 2023-11-15T01:46:27.732 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-11-15T01:46:27.732 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pycparser-2.14-14.el8.noarch 2023-11-15T01:46:27.733 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pytz-2017.2-11.el8.noarch 2023-11-15T01:46:27.733 INFO:teuthology.orchestra.run.smithi055.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-11-15T01:46:27.733 INFO:teuthology.orchestra.run.smithi055.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-11-15T01:46:27.733 INFO:teuthology.orchestra.run.smithi055.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-11-15T01:46:27.733 INFO:teuthology.orchestra.run.smithi055.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-11-15T01:46:27.734 INFO:teuthology.orchestra.run.smithi055.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-11-15T01:46:27.734 INFO:teuthology.orchestra.run.smithi055.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-11-15T01:46:27.735 INFO:teuthology.orchestra.run.smithi055.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-11-15T01:46:27.735 INFO:teuthology.orchestra.run.smithi055.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-11-15T01:46:27.735 INFO:teuthology.orchestra.run.smithi055.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-11-15T01:46:27.736 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-15T01:46:27.736 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2023-11-15T01:46:27.851 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2023-11-15T01:46:27.877 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph-base 2023-11-15T01:46:28.244 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2023-11-15T01:46:28.245 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2023-11-15T01:46:28.245 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2023-11-15T01:46:28.245 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2023-11-15T01:46:28.245 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2023-11-15T01:46:28.245 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2023-11-15T01:46:28.246 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2023-11-15T01:46:28.246 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2023-11-15T01:46:28.246 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-11-15T01:46:28.246 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-11-15T01:46:28.247 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2023-11-15T01:46:28.247 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-11-15T01:46:28.247 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2023-11-15T01:46:28.247 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 13/32 2023-11-15T01:46:28.247 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-11-15T01:46:28.248 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-11-15T01:46:28.248 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-11-15T01:46:28.248 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-11-15T01:46:28.248 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-11-15T01:46:28.248 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-11-15T01:46:28.249 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-11-15T01:46:28.249 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-11-15T01:46:28.249 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-11-15T01:46:28.249 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-11-15T01:46:28.250 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-11-15T01:46:28.250 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-11-15T01:46:28.250 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-11-15T01:46:28.250 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-11-15T01:46:28.250 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-11-15T01:46:28.251 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2023-11-15T01:46:28.251 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-11-15T01:46:28.251 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-11-15T01:46:28.866 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:01:16 ago on Wed 15 Nov 2023 01:45:12 AM UTC. 2023-11-15T01:46:29.506 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-11-15T01:46:29.507 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:46:29.507 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2023-11-15T01:46:29.507 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-2:16.2.4-0.el8.x86_64 2023-11-15T01:46:29.507 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2023-11-15T01:46:29.508 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2023-11-15T01:46:29.508 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2023-11-15T01:46:29.508 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2023-11-15T01:46:29.508 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2023-11-15T01:46:29.508 INFO:teuthology.orchestra.run.smithi137.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-11-15T01:46:29.509 INFO:teuthology.orchestra.run.smithi137.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-11-15T01:46:29.509 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cffi-1.11.5-6.el8.x86_64 2023-11-15T01:46:29.509 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-11-15T01:46:29.509 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-11-15T01:46:29.509 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2023-11-15T01:46:29.510 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-6.2-6.el8.noarch 2023-11-15T01:46:29.510 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-11-15T01:46:29.510 INFO:teuthology.orchestra.run.smithi137.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-11-15T01:46:29.510 INFO:teuthology.orchestra.run.smithi137.stdout: python3-mako-1.0.6-14.el8.noarch 2023-11-15T01:46:29.510 INFO:teuthology.orchestra.run.smithi137.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-11-15T01:46:29.511 INFO:teuthology.orchestra.run.smithi137.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-11-15T01:46:29.511 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-11-15T01:46:29.511 INFO:teuthology.orchestra.run.smithi137.stdout: python3-portend-2.6-1.el8.noarch 2023-11-15T01:46:29.511 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-11-15T01:46:29.512 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pycparser-2.14-14.el8.noarch 2023-11-15T01:46:29.512 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pytz-2017.2-11.el8.noarch 2023-11-15T01:46:29.512 INFO:teuthology.orchestra.run.smithi137.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-11-15T01:46:29.512 INFO:teuthology.orchestra.run.smithi137.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-11-15T01:46:29.512 INFO:teuthology.orchestra.run.smithi137.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-11-15T01:46:29.513 INFO:teuthology.orchestra.run.smithi137.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-11-15T01:46:29.513 INFO:teuthology.orchestra.run.smithi137.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-11-15T01:46:29.513 INFO:teuthology.orchestra.run.smithi137.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-11-15T01:46:29.513 INFO:teuthology.orchestra.run.smithi137.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-11-15T01:46:29.513 INFO:teuthology.orchestra.run.smithi137.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-11-15T01:46:29.513 INFO:teuthology.orchestra.run.smithi137.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-11-15T01:46:29.514 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:46:29.514 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2023-11-15T01:46:29.719 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-base 2023-11-15T01:46:30.489 INFO:teuthology.orchestra.run.smithi055.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T01:46:30.578 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:13 ago on Wed 15 Nov 2023 01:45:17 AM UTC. 2023-11-15T01:46:30.733 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2023-11-15T01:46:30.734 INFO:teuthology.orchestra.run.smithi055.stdout:Nothing to do. 2023-11-15T01:46:30.735 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2023-11-15T01:46:30.834 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install cephadm 2023-11-15T01:46:31.812 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:01:19 ago on Wed 15 Nov 2023 01:45:12 AM UTC. 2023-11-15T01:46:32.030 INFO:teuthology.orchestra.run.smithi137.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T01:46:32.248 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2023-11-15T01:46:32.250 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2023-11-15T01:46:32.250 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2023-11-15T01:46:32.382 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install cephadm 2023-11-15T01:46:33.235 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:16 ago on Wed 15 Nov 2023 01:45:17 AM UTC. 2023-11-15T01:46:33.680 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2023-11-15T01:46:33.681 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-15T01:46:33.681 INFO:teuthology.orchestra.run.smithi055.stdout: Package Architecture Version Repository Size 2023-11-15T01:46:33.682 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-15T01:46:33.682 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2023-11-15T01:46:33.682 INFO:teuthology.orchestra.run.smithi055.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2023-11-15T01:46:33.682 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-15T01:46:33.682 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2023-11-15T01:46:33.683 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-15T01:46:33.683 INFO:teuthology.orchestra.run.smithi055.stdout:Install 1 Package 2023-11-15T01:46:33.683 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-15T01:46:33.684 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 71 k 2023-11-15T01:46:33.684 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 290 k 2023-11-15T01:46:33.684 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2023-11-15T01:46:34.116 INFO:teuthology.orchestra.run.smithi055.stdout:cephadm-16.2.4-0.el8.noarch.rpm 164 kB/s | 71 kB 00:00 2023-11-15T01:46:34.117 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2023-11-15T01:46:34.117 INFO:teuthology.orchestra.run.smithi055.stdout:Total 164 kB/s | 71 kB 00:00 2023-11-15T01:46:34.118 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2023-11-15T01:46:34.126 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2023-11-15T01:46:34.126 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2023-11-15T01:46:34.129 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2023-11-15T01:46:34.132 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2023-11-15T01:46:34.176 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2023-11-15T01:46:34.654 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2023-11-15T01:46:34.846 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2023-11-15T01:46:34.904 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2023-11-15T01:46:34.905 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T01:46:34.905 INFO:teuthology.orchestra.run.smithi137.stdout: Package Architecture Version Repository Size 2023-11-15T01:46:34.905 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T01:46:34.905 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2023-11-15T01:46:34.906 INFO:teuthology.orchestra.run.smithi137.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2023-11-15T01:46:34.906 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:46:34.906 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2023-11-15T01:46:34.906 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T01:46:34.906 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2023-11-15T01:46:34.907 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:46:34.907 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 71 k 2023-11-15T01:46:34.907 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 290 k 2023-11-15T01:46:34.907 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2023-11-15T01:46:34.942 INFO:teuthology.orchestra.run.smithi137.stdout:cephadm-16.2.4-0.el8.noarch.rpm 1.9 MB/s | 71 kB 00:00 2023-11-15T01:46:34.942 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2023-11-15T01:46:34.943 INFO:teuthology.orchestra.run.smithi137.stdout:Total 1.9 MB/s | 71 kB 00:00 2023-11-15T01:46:34.943 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2023-11-15T01:46:34.949 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2023-11-15T01:46:34.950 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2023-11-15T01:46:34.951 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2023-11-15T01:46:34.954 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2023-11-15T01:46:34.998 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2023-11-15T01:46:35.107 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2023-11-15T01:46:35.501 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2023-11-15T01:46:35.742 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2023-11-15T01:46:35.891 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2023-11-15T01:46:35.891 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-15T01:46:35.891 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2023-11-15T01:46:35.892 INFO:teuthology.orchestra.run.smithi055.stdout: cephadm-2:16.2.4-0.el8.noarch 2023-11-15T01:46:35.892 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-15T01:46:35.892 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2023-11-15T01:46:35.967 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2023-11-15T01:46:36.095 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph-immutable-object-cache 2023-11-15T01:46:36.615 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2023-11-15T01:46:36.616 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:46:36.616 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2023-11-15T01:46:36.616 INFO:teuthology.orchestra.run.smithi137.stdout: cephadm-2:16.2.4-0.el8.noarch 2023-11-15T01:46:36.616 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:46:36.616 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2023-11-15T01:46:37.089 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:01:25 ago on Wed 15 Nov 2023 01:45:12 AM UTC. 2023-11-15T01:46:38.503 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-immutable-object-cache 2023-11-15T01:46:38.936 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2023-11-15T01:46:38.936 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-15T01:46:38.937 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repo Size 2023-11-15T01:46:38.937 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-15T01:46:38.937 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2023-11-15T01:46:38.937 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2023-11-15T01:46:38.937 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-15T01:46:38.938 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2023-11-15T01:46:38.938 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-15T01:46:38.938 INFO:teuthology.orchestra.run.smithi055.stdout:Install 1 Package 2023-11-15T01:46:38.938 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-15T01:46:38.939 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 163 k 2023-11-15T01:46:38.939 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 587 k 2023-11-15T01:46:38.939 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2023-11-15T01:46:38.978 INFO:teuthology.orchestra.run.smithi055.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 4.0 MB/s | 163 kB 00:00 2023-11-15T01:46:38.979 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2023-11-15T01:46:38.979 INFO:teuthology.orchestra.run.smithi055.stdout:Total 3.8 MB/s | 163 kB 00:00 2023-11-15T01:46:38.980 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2023-11-15T01:46:38.992 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2023-11-15T01:46:38.992 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2023-11-15T01:46:39.046 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2023-11-15T01:46:39.050 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2023-11-15T01:46:39.173 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2023-11-15T01:46:39.365 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:22 ago on Wed 15 Nov 2023 01:45:17 AM UTC. 2023-11-15T01:46:40.664 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T01:46:41.027 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2023-11-15T01:46:41.028 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T01:46:41.028 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repo Size 2023-11-15T01:46:41.029 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T01:46:41.029 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2023-11-15T01:46:41.029 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2023-11-15T01:46:41.029 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:46:41.029 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2023-11-15T01:46:41.030 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T01:46:41.030 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2023-11-15T01:46:41.030 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:46:41.030 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 163 k 2023-11-15T01:46:41.031 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 587 k 2023-11-15T01:46:41.031 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2023-11-15T01:46:41.072 INFO:teuthology.orchestra.run.smithi137.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 3.7 MB/s | 163 kB 00:00 2023-11-15T01:46:41.073 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2023-11-15T01:46:41.074 INFO:teuthology.orchestra.run.smithi137.stdout:Total 3.6 MB/s | 163 kB 00:00 2023-11-15T01:46:41.074 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2023-11-15T01:46:41.084 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2023-11-15T01:46:41.084 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2023-11-15T01:46:41.130 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2023-11-15T01:46:41.133 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2023-11-15T01:46:41.244 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2023-11-15T01:46:41.262 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T01:46:41.441 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T01:46:41.848 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T01:46:42.029 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T01:46:42.029 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-15T01:46:42.030 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2023-11-15T01:46:42.030 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2023-11-15T01:46:42.030 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-15T01:46:42.030 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2023-11-15T01:46:42.233 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph-mgr 2023-11-15T01:46:42.496 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T01:46:42.497 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:46:42.497 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2023-11-15T01:46:42.497 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2023-11-15T01:46:42.497 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:46:42.497 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2023-11-15T01:46:42.741 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-mgr 2023-11-15T01:46:43.228 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:01:31 ago on Wed 15 Nov 2023 01:45:12 AM UTC. 2023-11-15T01:46:43.612 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:26 ago on Wed 15 Nov 2023 01:45:17 AM UTC. 2023-11-15T01:46:44.855 INFO:teuthology.orchestra.run.smithi055.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T01:46:45.049 INFO:teuthology.orchestra.run.smithi137.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T01:46:45.098 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2023-11-15T01:46:45.099 INFO:teuthology.orchestra.run.smithi055.stdout:Nothing to do. 2023-11-15T01:46:45.099 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2023-11-15T01:46:45.198 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph-mgr-dashboard 2023-11-15T01:46:45.267 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2023-11-15T01:46:45.268 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2023-11-15T01:46:45.268 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2023-11-15T01:46:45.404 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-mgr-dashboard 2023-11-15T01:46:46.180 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:01:34 ago on Wed 15 Nov 2023 01:45:12 AM UTC. 2023-11-15T01:46:46.257 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:29 ago on Wed 15 Nov 2023 01:45:17 AM UTC. 2023-11-15T01:46:47.918 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2023-11-15T01:46:47.919 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T01:46:47.919 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2023-11-15T01:46:47.919 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T01:46:47.920 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2023-11-15T01:46:47.920 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2023-11-15T01:46:47.920 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2023-11-15T01:46:47.920 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2023-11-15T01:46:47.921 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2023-11-15T01:46:47.921 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2023-11-15T01:46:47.921 INFO:teuthology.orchestra.run.smithi137.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-11-15T01:46:47.921 INFO:teuthology.orchestra.run.smithi137.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-11-15T01:46:47.921 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:46:47.922 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2023-11-15T01:46:47.922 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T01:46:47.922 INFO:teuthology.orchestra.run.smithi137.stdout:Install 6 Packages 2023-11-15T01:46:47.922 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:46:47.923 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 3.0 M 2023-11-15T01:46:47.923 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 15 M 2023-11-15T01:46:47.923 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2023-11-15T01:46:48.030 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2023-11-15T01:46:48.031 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-15T01:46:48.031 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repository Size 2023-11-15T01:46:48.032 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-15T01:46:48.032 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2023-11-15T01:46:48.032 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2023-11-15T01:46:48.032 INFO:teuthology.orchestra.run.smithi055.stdout:Installing dependencies: 2023-11-15T01:46:48.032 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2023-11-15T01:46:48.033 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2023-11-15T01:46:48.033 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2023-11-15T01:46:48.033 INFO:teuthology.orchestra.run.smithi055.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-11-15T01:46:48.033 INFO:teuthology.orchestra.run.smithi055.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-11-15T01:46:48.033 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-15T01:46:48.034 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2023-11-15T01:46:48.034 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-15T01:46:48.034 INFO:teuthology.orchestra.run.smithi055.stdout:Install 6 Packages 2023-11-15T01:46:48.034 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-15T01:46:48.035 INFO:teuthology.orchestra.run.smithi137.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 546 kB/s | 24 kB 00:00 2023-11-15T01:46:48.035 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 3.0 M 2023-11-15T01:46:48.036 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 15 M 2023-11-15T01:46:48.036 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2023-11-15T01:46:48.109 INFO:teuthology.orchestra.run.smithi137.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 75 kB/s | 8.9 kB 00:00 2023-11-15T01:46:48.168 INFO:teuthology.orchestra.run.smithi137.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 15 MB/s | 2.7 MB 00:00 2023-11-15T01:46:48.265 INFO:teuthology.orchestra.run.smithi055.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 518 kB/s | 24 kB 00:00 2023-11-15T01:46:48.290 INFO:teuthology.orchestra.run.smithi055.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 125 kB/s | 8.9 kB 00:00 2023-11-15T01:46:48.310 INFO:teuthology.orchestra.run.smithi137.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 168 kB/s | 34 kB 00:00 2023-11-15T01:46:48.358 INFO:teuthology.orchestra.run.smithi055.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 19 MB/s | 2.7 MB 00:00 2023-11-15T01:46:48.360 INFO:teuthology.orchestra.run.smithi137.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.0 MB/s | 196 kB 00:00 2023-11-15T01:46:48.427 INFO:teuthology.orchestra.run.smithi137.stdout:(6/6): python3-jwt-1.6.1-2.el8.noarch.rpm 108 kB/s | 43 kB 00:00 2023-11-15T01:46:48.430 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2023-11-15T01:46:48.430 INFO:teuthology.orchestra.run.smithi137.stdout:Total 5.8 MB/s | 3.0 MB 00:00 2023-11-15T01:46:48.439 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2023-11-15T01:46:48.449 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2023-11-15T01:46:48.449 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2023-11-15T01:46:48.574 INFO:teuthology.orchestra.run.smithi055.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 119 kB/s | 34 kB 00:00 2023-11-15T01:46:48.610 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2023-11-15T01:46:48.614 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2023-11-15T01:46:48.641 INFO:teuthology.orchestra.run.smithi055.stdout:(5/6): python3-jwt-1.6.1-2.el8.noarch.rpm 113 kB/s | 43 kB 00:00 2023-11-15T01:46:48.825 INFO:teuthology.orchestra.run.smithi055.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 421 kB/s | 196 kB 00:00 2023-11-15T01:46:48.827 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2023-11-15T01:46:48.828 INFO:teuthology.orchestra.run.smithi055.stdout:Total 3.7 MB/s | 3.0 MB 00:00 2023-11-15T01:46:48.840 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2023-11-15T01:46:48.852 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2023-11-15T01:46:48.852 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2023-11-15T01:46:48.914 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2023-11-15T01:46:49.041 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2023-11-15T01:46:49.045 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2023-11-15T01:46:49.186 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-11-15T01:46:49.332 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2023-11-15T01:46:49.544 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-11-15T01:46:49.615 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-11-15T01:46:49.813 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-11-15T01:46:49.816 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-11-15T01:46:49.954 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2023-11-15T01:46:50.001 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-11-15T01:46:50.144 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2023-11-15T01:46:50.971 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2023-11-15T01:46:51.221 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2023-11-15T01:46:51.276 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2023-11-15T01:46:51.496 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2023-11-15T01:46:51.564 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2023-11-15T01:46:51.565 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2023-11-15T01:46:51.565 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2023-11-15T01:46:51.565 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2023-11-15T01:46:51.565 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-11-15T01:46:51.565 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-11-15T01:46:51.822 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2023-11-15T01:46:51.822 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2023-11-15T01:46:51.823 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2023-11-15T01:46:51.823 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2023-11-15T01:46:51.823 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-11-15T01:46:51.823 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-11-15T01:46:52.250 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-11-15T01:46:52.250 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:46:52.250 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2023-11-15T01:46:52.250 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2023-11-15T01:46:52.251 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2023-11-15T01:46:52.251 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2023-11-15T01:46:52.251 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-11-15T01:46:52.251 INFO:teuthology.orchestra.run.smithi137.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-11-15T01:46:52.251 INFO:teuthology.orchestra.run.smithi137.stdout: python3-routes-2.4.1-12.el8.noarch 2023-11-15T01:46:52.252 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:46:52.252 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2023-11-15T01:46:52.389 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-mgr-diskprediction-local 2023-11-15T01:46:52.569 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-11-15T01:46:52.569 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-15T01:46:52.569 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2023-11-15T01:46:52.570 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2023-11-15T01:46:52.570 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2023-11-15T01:46:52.570 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2023-11-15T01:46:52.570 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-11-15T01:46:52.570 INFO:teuthology.orchestra.run.smithi055.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-11-15T01:46:52.571 INFO:teuthology.orchestra.run.smithi055.stdout: python3-routes-2.4.1-12.el8.noarch 2023-11-15T01:46:52.571 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-15T01:46:52.571 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2023-11-15T01:46:52.689 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph-mgr-diskprediction-local 2023-11-15T01:46:53.257 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:36 ago on Wed 15 Nov 2023 01:45:17 AM UTC. 2023-11-15T01:46:53.681 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:01:41 ago on Wed 15 Nov 2023 01:45:12 AM UTC. 2023-11-15T01:46:54.933 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2023-11-15T01:46:54.934 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================================ 2023-11-15T01:46:54.934 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2023-11-15T01:46:54.934 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================================ 2023-11-15T01:46:54.934 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2023-11-15T01:46:54.934 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2023-11-15T01:46:54.935 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2023-11-15T01:46:54.935 INFO:teuthology.orchestra.run.smithi137.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2023-11-15T01:46:54.935 INFO:teuthology.orchestra.run.smithi137.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2023-11-15T01:46:54.935 INFO:teuthology.orchestra.run.smithi137.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2023-11-15T01:46:54.936 INFO:teuthology.orchestra.run.smithi137.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2023-11-15T01:46:54.936 INFO:teuthology.orchestra.run.smithi137.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2023-11-15T01:46:54.936 INFO:teuthology.orchestra.run.smithi137.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2023-11-15T01:46:54.936 INFO:teuthology.orchestra.run.smithi137.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 appstream 14 M 2023-11-15T01:46:54.936 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:46:54.937 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2023-11-15T01:46:54.937 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================================ 2023-11-15T01:46:54.937 INFO:teuthology.orchestra.run.smithi137.stdout:Install 8 Packages 2023-11-15T01:46:54.937 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:46:54.938 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 37 M 2023-11-15T01:46:54.938 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 210 M 2023-11-15T01:46:54.938 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2023-11-15T01:46:55.448 INFO:teuthology.orchestra.run.smithi137.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 31 MB/s | 8.6 MB 00:00 2023-11-15T01:46:55.531 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2023-11-15T01:46:55.533 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================================ 2023-11-15T01:46:55.533 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repository Size 2023-11-15T01:46:55.533 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================================ 2023-11-15T01:46:55.534 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2023-11-15T01:46:55.534 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2023-11-15T01:46:55.534 INFO:teuthology.orchestra.run.smithi055.stdout:Installing dependencies: 2023-11-15T01:46:55.534 INFO:teuthology.orchestra.run.smithi055.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2023-11-15T01:46:55.534 INFO:teuthology.orchestra.run.smithi055.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2023-11-15T01:46:55.535 INFO:teuthology.orchestra.run.smithi055.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2023-11-15T01:46:55.535 INFO:teuthology.orchestra.run.smithi055.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2023-11-15T01:46:55.535 INFO:teuthology.orchestra.run.smithi055.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2023-11-15T01:46:55.535 INFO:teuthology.orchestra.run.smithi055.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2023-11-15T01:46:55.535 INFO:teuthology.orchestra.run.smithi055.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 appstream 14 M 2023-11-15T01:46:55.536 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-15T01:46:55.536 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2023-11-15T01:46:55.536 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================================ 2023-11-15T01:46:55.536 INFO:teuthology.orchestra.run.smithi055.stdout:Install 8 Packages 2023-11-15T01:46:55.536 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-15T01:46:55.537 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 37 M 2023-11-15T01:46:55.537 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 210 M 2023-11-15T01:46:55.537 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2023-11-15T01:46:55.643 INFO:teuthology.orchestra.run.smithi137.stdout:(2/8): openblas-0.3.15-4.el8.x86_64.rpm 9.9 MB/s | 4.7 MB 00:00 2023-11-15T01:46:55.718 INFO:teuthology.orchestra.run.smithi137.stdout:(3/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 8.8 MB/s | 4.9 MB 00:00 2023-11-15T01:46:55.744 INFO:teuthology.orchestra.run.smithi137.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 2.2 MB/s | 225 kB 00:00 2023-11-15T01:46:56.019 INFO:teuthology.orchestra.run.smithi137.stdout:(5/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 6.5 MB/s | 3.7 MB 00:00 2023-11-15T01:46:56.081 INFO:teuthology.orchestra.run.smithi055.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 50 MB/s | 8.6 MB 00:00 2023-11-15T01:46:56.236 INFO:teuthology.orchestra.run.smithi137.stdout:(6/8): python3-scipy-1.0.0-21.module_el8.5.0+77 26 MB/s | 14 MB 00:00 2023-11-15T01:46:56.461 INFO:teuthology.orchestra.run.smithi137.stdout:(7/8): libgfortran-8.5.0-21.el8.x86_64.rpm 900 kB/s | 645 kB 00:00 2023-11-15T01:46:56.562 INFO:teuthology.orchestra.run.smithi137.stdout:(8/8): libquadmath-8.5.0-21.el8.x86_64.rpm 318 kB/s | 172 kB 00:00 2023-11-15T01:46:56.562 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2023-11-15T01:46:56.562 INFO:teuthology.orchestra.run.smithi137.stdout:Total 23 MB/s | 37 MB 00:01 2023-11-15T01:46:56.633 INFO:teuthology.orchestra.run.smithi055.stdout:(2/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 6.7 MB/s | 4.9 MB 00:00 2023-11-15T01:46:56.790 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2023-11-15T01:46:56.809 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2023-11-15T01:46:56.809 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2023-11-15T01:46:56.899 INFO:teuthology.orchestra.run.smithi055.stdout:(3/8): openblas-0.3.15-4.el8.x86_64.rpm 4.8 MB/s | 4.7 MB 00:00 2023-11-15T01:46:57.041 INFO:teuthology.orchestra.run.smithi055.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 551 kB/s | 225 kB 00:00 2023-11-15T01:46:57.129 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2023-11-15T01:46:57.132 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2023-11-15T01:46:57.150 INFO:teuthology.orchestra.run.smithi055.stdout:(5/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 3.5 MB/s | 3.7 MB 00:01 2023-11-15T01:46:57.451 INFO:teuthology.orchestra.run.smithi055.stdout:(6/8): python3-scipy-1.0.0-21.module_el8.5.0+77 25 MB/s | 14 MB 00:00 2023-11-15T01:46:57.584 INFO:teuthology.orchestra.run.smithi055.stdout:(7/8): libquadmath-8.5.0-21.el8.x86_64.rpm 398 kB/s | 172 kB 00:00 2023-11-15T01:46:57.659 INFO:teuthology.orchestra.run.smithi055.stdout:(8/8): libgfortran-8.5.0-21.el8.x86_64.rpm 1.0 MB/s | 645 kB 00:00 2023-11-15T01:46:57.660 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2023-11-15T01:46:57.660 INFO:teuthology.orchestra.run.smithi055.stdout:Total 17 MB/s | 37 MB 00:02 2023-11-15T01:46:57.802 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2023-11-15T01:46:57.911 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2023-11-15T01:46:57.931 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2023-11-15T01:46:57.931 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2023-11-15T01:46:57.971 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2023-11-15T01:46:58.089 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2023-11-15T01:46:58.221 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2023-11-15T01:46:58.283 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2023-11-15T01:46:58.286 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2023-11-15T01:46:58.799 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2023-11-15T01:46:58.917 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2023-11-15T01:46:58.988 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-11-15T01:46:59.056 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2023-11-15T01:46:59.206 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2023-11-15T01:46:59.356 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2023-11-15T01:46:59.585 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-11-15T01:46:59.821 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-11-15T01:47:00.019 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2023-11-15T01:47:00.181 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-11-15T01:47:00.497 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-11-15T01:47:00.865 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-11-15T01:47:01.006 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-11-15T01:47:01.087 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-11-15T01:47:01.337 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-11-15T01:47:01.788 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-11-15T01:47:02.121 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-11-15T01:47:02.689 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-11-15T01:47:03.099 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-11-15T01:47:04.426 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2023-11-15T01:47:04.662 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2023-11-15T01:47:04.674 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-11-15T01:47:04.979 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2023-11-15T01:47:04.980 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2023-11-15T01:47:04.980 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-11-15T01:47:04.980 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-11-15T01:47:04.980 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-11-15T01:47:04.980 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-11-15T01:47:04.981 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2023-11-15T01:47:04.981 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2023-11-15T01:47:05.676 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2023-11-15T01:47:05.677 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:47:05.677 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2023-11-15T01:47:05.677 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2023-11-15T01:47:05.678 INFO:teuthology.orchestra.run.smithi137.stdout: libgfortran-8.5.0-21.el8.x86_64 2023-11-15T01:47:05.678 INFO:teuthology.orchestra.run.smithi137.stdout: libquadmath-8.5.0-21.el8.x86_64 2023-11-15T01:47:05.678 INFO:teuthology.orchestra.run.smithi137.stdout: openblas-0.3.15-4.el8.x86_64 2023-11-15T01:47:05.678 INFO:teuthology.orchestra.run.smithi137.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-11-15T01:47:05.678 INFO:teuthology.orchestra.run.smithi137.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-11-15T01:47:05.679 INFO:teuthology.orchestra.run.smithi137.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-11-15T01:47:05.679 INFO:teuthology.orchestra.run.smithi137.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-11-15T01:47:05.679 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:47:05.679 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2023-11-15T01:47:06.104 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-mgr-rook 2023-11-15T01:47:06.166 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2023-11-15T01:47:06.364 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2023-11-15T01:47:06.732 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2023-11-15T01:47:06.732 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2023-11-15T01:47:06.732 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-11-15T01:47:06.733 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-11-15T01:47:06.733 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-11-15T01:47:06.733 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-11-15T01:47:06.733 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2023-11-15T01:47:06.733 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2023-11-15T01:47:06.968 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:49 ago on Wed 15 Nov 2023 01:45:17 AM UTC. 2023-11-15T01:47:07.507 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2023-11-15T01:47:07.508 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-15T01:47:07.508 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2023-11-15T01:47:07.508 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2023-11-15T01:47:07.508 INFO:teuthology.orchestra.run.smithi055.stdout: libgfortran-8.5.0-21.el8.x86_64 2023-11-15T01:47:07.508 INFO:teuthology.orchestra.run.smithi055.stdout: libquadmath-8.5.0-21.el8.x86_64 2023-11-15T01:47:07.509 INFO:teuthology.orchestra.run.smithi055.stdout: openblas-0.3.15-4.el8.x86_64 2023-11-15T01:47:07.509 INFO:teuthology.orchestra.run.smithi055.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-11-15T01:47:07.509 INFO:teuthology.orchestra.run.smithi055.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-11-15T01:47:07.509 INFO:teuthology.orchestra.run.smithi055.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-11-15T01:47:07.509 INFO:teuthology.orchestra.run.smithi055.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-11-15T01:47:07.510 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-15T01:47:07.510 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2023-11-15T01:47:07.622 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph-mgr-rook 2023-11-15T01:47:08.615 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2023-11-15T01:47:08.616 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T01:47:08.617 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2023-11-15T01:47:08.617 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T01:47:08.617 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2023-11-15T01:47:08.618 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2023-11-15T01:47:08.618 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2023-11-15T01:47:08.618 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-11-15T01:47:08.618 INFO:teuthology.orchestra.run.smithi137.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-11-15T01:47:08.618 INFO:teuthology.orchestra.run.smithi137.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-11-15T01:47:08.619 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2023-11-15T01:47:08.619 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2023-11-15T01:47:08.619 INFO:teuthology.orchestra.run.smithi137.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-11-15T01:47:08.619 INFO:teuthology.orchestra.run.smithi137.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2023-11-15T01:47:08.619 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2023-11-15T01:47:08.620 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2023-11-15T01:47:08.620 INFO:teuthology.orchestra.run.smithi137.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2023-11-15T01:47:08.620 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-11-15T01:47:08.620 INFO:teuthology.orchestra.run.smithi137.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-11-15T01:47:08.620 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:47:08.621 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2023-11-15T01:47:08.621 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T01:47:08.621 INFO:teuthology.orchestra.run.smithi137.stdout:Install 13 Packages 2023-11-15T01:47:08.621 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:47:08.622 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 1.9 M 2023-11-15T01:47:08.623 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 24 M 2023-11-15T01:47:08.623 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2023-11-15T01:47:08.630 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:01:56 ago on Wed 15 Nov 2023 01:45:12 AM UTC. 2023-11-15T01:47:09.261 INFO:teuthology.orchestra.run.smithi137.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 640 kB/s | 27 kB 00:00 2023-11-15T01:47:09.302 INFO:teuthology.orchestra.run.smithi137.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 238 kB/s | 20 kB 00:00 2023-11-15T01:47:09.320 INFO:teuthology.orchestra.run.smithi137.stdout:(3/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 263 kB/s | 27 kB 00:00 2023-11-15T01:47:09.345 INFO:teuthology.orchestra.run.smithi137.stdout:(4/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 1.5 MB/s | 126 kB 00:00 2023-11-15T01:47:09.370 INFO:teuthology.orchestra.run.smithi137.stdout:(5/13): python3-pyasn1-modules-0.3.7-6.el8.noar 1.6 MB/s | 110 kB 00:00 2023-11-15T01:47:09.587 INFO:teuthology.orchestra.run.smithi137.stdout:(6/13): python3-cachetools-3.1.1-4.el8.noarch.r 152 kB/s | 33 kB 00:00 2023-11-15T01:47:09.646 INFO:teuthology.orchestra.run.smithi137.stdout:(7/13): python3-certifi-2018.10.15-7.el8.noarch 279 kB/s | 16 kB 00:00 2023-11-15T01:47:09.671 INFO:teuthology.orchestra.run.smithi137.stdout:(8/13): python3-requests-oauthlib-1.0.0-1.el8.n 131 kB/s | 43 kB 00:00 2023-11-15T01:47:09.755 INFO:teuthology.orchestra.run.smithi137.stdout:(9/13): python3-google-auth-1.1.1-10.el8.noarch 748 kB/s | 81 kB 00:00 2023-11-15T01:47:09.788 INFO:teuthology.orchestra.run.smithi137.stdout:(10/13): python3-oauthlib-2.1.0-1.el8.noarch.rp 331 kB/s | 155 kB 00:00 2023-11-15T01:47:09.855 INFO:teuthology.orchestra.run.smithi137.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 645 kB/s | 64 kB 00:00 2023-11-15T01:47:10.013 INFO:teuthology.orchestra.run.smithi137.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 272 kB/s | 61 kB 00:00 2023-11-15T01:47:10.050 INFO:teuthology.orchestra.run.smithi137.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 3.1 MB/s | 1.2 MB 00:00 2023-11-15T01:47:10.051 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2023-11-15T01:47:10.051 INFO:teuthology.orchestra.run.smithi137.stdout:Total 1.3 MB/s | 1.9 MB 00:01 2023-11-15T01:47:10.111 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2023-11-15T01:47:10.127 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2023-11-15T01:47:10.127 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2023-11-15T01:47:10.214 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2023-11-15T01:47:10.217 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2023-11-15T01:47:10.494 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2023-11-15T01:47:10.496 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-15T01:47:10.497 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repository Size 2023-11-15T01:47:10.497 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-15T01:47:10.497 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2023-11-15T01:47:10.497 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2023-11-15T01:47:10.498 INFO:teuthology.orchestra.run.smithi055.stdout:Installing dependencies: 2023-11-15T01:47:10.498 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-11-15T01:47:10.498 INFO:teuthology.orchestra.run.smithi055.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-11-15T01:47:10.498 INFO:teuthology.orchestra.run.smithi055.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-11-15T01:47:10.499 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2023-11-15T01:47:10.499 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2023-11-15T01:47:10.499 INFO:teuthology.orchestra.run.smithi055.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-11-15T01:47:10.499 INFO:teuthology.orchestra.run.smithi055.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2023-11-15T01:47:10.499 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2023-11-15T01:47:10.500 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2023-11-15T01:47:10.500 INFO:teuthology.orchestra.run.smithi055.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2023-11-15T01:47:10.500 INFO:teuthology.orchestra.run.smithi055.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-11-15T01:47:10.500 INFO:teuthology.orchestra.run.smithi055.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-11-15T01:47:10.501 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-15T01:47:10.501 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2023-11-15T01:47:10.501 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-15T01:47:10.501 INFO:teuthology.orchestra.run.smithi055.stdout:Install 13 Packages 2023-11-15T01:47:10.501 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-15T01:47:10.502 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 1.9 M 2023-11-15T01:47:10.503 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 24 M 2023-11-15T01:47:10.503 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2023-11-15T01:47:10.764 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2023-11-15T01:47:11.054 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-11-15T01:47:11.252 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-11-15T01:47:11.324 INFO:teuthology.orchestra.run.smithi055.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 557 kB/s | 27 kB 00:00 2023-11-15T01:47:11.374 INFO:teuthology.orchestra.run.smithi055.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 203 kB/s | 20 kB 00:00 2023-11-15T01:47:11.399 INFO:teuthology.orchestra.run.smithi055.stdout:(3/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 215 kB/s | 27 kB 00:00 2023-11-15T01:47:11.424 INFO:teuthology.orchestra.run.smithi055.stdout:(4/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 1.2 MB/s | 126 kB 00:00 2023-11-15T01:47:11.450 INFO:teuthology.orchestra.run.smithi055.stdout:(5/13): python3-pyasn1-modules-0.3.7-6.el8.noar 1.4 MB/s | 110 kB 00:00 2023-11-15T01:47:11.521 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2023-11-15T01:47:11.567 INFO:teuthology.orchestra.run.smithi055.stdout:(6/13): python3-cachetools-3.1.1-4.el8.noarch.r 282 kB/s | 33 kB 00:00 2023-11-15T01:47:11.609 INFO:teuthology.orchestra.run.smithi055.stdout:(7/13): python3-certifi-2018.10.15-7.el8.noarch 392 kB/s | 16 kB 00:00 2023-11-15T01:47:11.676 INFO:teuthology.orchestra.run.smithi055.stdout:(8/13): python3-google-auth-1.1.1-10.el8.noarch 1.2 MB/s | 81 kB 00:00 2023-11-15T01:47:11.734 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-11-15T01:47:11.743 INFO:teuthology.orchestra.run.smithi055.stdout:(9/13): python3-requests-oauthlib-1.0.0-1.el8.n 134 kB/s | 43 kB 00:00 2023-11-15T01:47:11.801 INFO:teuthology.orchestra.run.smithi055.stdout:(10/13): python3-kubernetes-11.0.0-6.el8.noarch 9.3 MB/s | 1.2 MB 00:00 2023-11-15T01:47:11.843 INFO:teuthology.orchestra.run.smithi055.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 1.5 MB/s | 61 kB 00:00 2023-11-15T01:47:11.868 INFO:teuthology.orchestra.run.smithi055.stdout:(12/13): python3-oauthlib-2.1.0-1.el8.noarch.rp 330 kB/s | 155 kB 00:00 2023-11-15T01:47:11.894 INFO:teuthology.orchestra.run.smithi055.stdout:(13/13): python3-rsa-4.9-2.el8.noarch.rpm 426 kB/s | 64 kB 00:00 2023-11-15T01:47:11.896 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2023-11-15T01:47:11.896 INFO:teuthology.orchestra.run.smithi055.stdout:Total 1.4 MB/s | 1.9 MB 00:01 2023-11-15T01:47:11.961 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2023-11-15T01:47:11.973 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-11-15T01:47:11.978 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2023-11-15T01:47:11.978 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2023-11-15T01:47:12.076 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2023-11-15T01:47:12.080 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2023-11-15T01:47:12.189 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-11-15T01:47:12.488 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-11-15T01:47:12.605 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2023-11-15T01:47:12.714 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-11-15T01:47:12.839 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-11-15T01:47:13.063 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-11-15T01:47:13.278 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2023-11-15T01:47:13.488 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-11-15T01:47:13.690 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-11-15T01:47:13.789 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-11-15T01:47:13.916 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-11-15T01:47:14.083 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-11-15T01:47:14.195 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-11-15T01:47:14.259 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-11-15T01:47:14.422 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-11-15T01:47:14.484 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-11-15T01:47:14.643 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2023-11-15T01:47:14.943 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2023-11-15T01:47:14.944 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2023-11-15T01:47:14.944 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-11-15T01:47:14.944 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-11-15T01:47:14.944 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-11-15T01:47:14.944 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-11-15T01:47:14.945 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-11-15T01:47:14.945 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-11-15T01:47:14.945 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-11-15T01:47:14.945 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-11-15T01:47:14.945 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-11-15T01:47:14.946 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-11-15T01:47:14.946 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2023-11-15T01:47:15.623 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-11-15T01:47:15.710 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-11-15T01:47:15.710 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:47:15.710 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2023-11-15T01:47:15.710 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2023-11-15T01:47:15.711 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-11-15T01:47:15.711 INFO:teuthology.orchestra.run.smithi137.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-11-15T01:47:15.711 INFO:teuthology.orchestra.run.smithi137.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-11-15T01:47:15.711 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-11-15T01:47:15.711 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-11-15T01:47:15.712 INFO:teuthology.orchestra.run.smithi137.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-11-15T01:47:15.712 INFO:teuthology.orchestra.run.smithi137.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-11-15T01:47:15.712 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-11-15T01:47:15.712 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-11-15T01:47:15.712 INFO:teuthology.orchestra.run.smithi137.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-11-15T01:47:15.713 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rsa-4.9-2.el8.noarch 2023-11-15T01:47:15.713 INFO:teuthology.orchestra.run.smithi137.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-11-15T01:47:15.713 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:47:15.713 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2023-11-15T01:47:15.879 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-11-15T01:47:15.889 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-mgr-cephadm 2023-11-15T01:47:16.071 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-11-15T01:47:16.238 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-11-15T01:47:16.379 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2023-11-15T01:47:16.711 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2023-11-15T01:47:16.712 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2023-11-15T01:47:16.712 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-11-15T01:47:16.712 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-11-15T01:47:16.712 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-11-15T01:47:16.712 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-11-15T01:47:16.713 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-11-15T01:47:16.713 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-11-15T01:47:16.713 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-11-15T01:47:16.713 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-11-15T01:47:16.713 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-11-15T01:47:16.714 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-11-15T01:47:16.714 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2023-11-15T01:47:16.745 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:59 ago on Wed 15 Nov 2023 01:45:17 AM UTC. 2023-11-15T01:47:17.531 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-11-15T01:47:17.531 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-15T01:47:17.532 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2023-11-15T01:47:17.532 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2023-11-15T01:47:17.533 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-11-15T01:47:17.533 INFO:teuthology.orchestra.run.smithi055.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-11-15T01:47:17.533 INFO:teuthology.orchestra.run.smithi055.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-11-15T01:47:17.534 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-11-15T01:47:17.534 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-11-15T01:47:17.534 INFO:teuthology.orchestra.run.smithi055.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-11-15T01:47:17.535 INFO:teuthology.orchestra.run.smithi055.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-11-15T01:47:17.535 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-11-15T01:47:17.535 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-11-15T01:47:17.536 INFO:teuthology.orchestra.run.smithi055.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-11-15T01:47:17.536 INFO:teuthology.orchestra.run.smithi055.stdout: python3-rsa-4.9-2.el8.noarch 2023-11-15T01:47:17.536 INFO:teuthology.orchestra.run.smithi055.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-11-15T01:47:17.536 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-15T01:47:17.537 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2023-11-15T01:47:17.650 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph-mgr-cephadm 2023-11-15T01:47:18.407 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2023-11-15T01:47:18.408 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T01:47:18.408 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2023-11-15T01:47:18.408 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T01:47:18.408 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2023-11-15T01:47:18.409 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2023-11-15T01:47:18.409 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2023-11-15T01:47:18.409 INFO:teuthology.orchestra.run.smithi137.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2023-11-15T01:47:18.409 INFO:teuthology.orchestra.run.smithi137.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2023-11-15T01:47:18.409 INFO:teuthology.orchestra.run.smithi137.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2023-11-15T01:47:18.410 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2023-11-15T01:47:18.410 INFO:teuthology.orchestra.run.smithi137.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2023-11-15T01:47:18.410 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:47:18.410 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2023-11-15T01:47:18.410 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T01:47:18.410 INFO:teuthology.orchestra.run.smithi137.stdout:Install 6 Packages 2023-11-15T01:47:18.411 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:47:18.411 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 5.8 M 2023-11-15T01:47:18.412 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 24 M 2023-11-15T01:47:18.412 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2023-11-15T01:47:18.629 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:02:06 ago on Wed 15 Nov 2023 01:45:12 AM UTC. 2023-11-15T01:47:18.896 INFO:teuthology.orchestra.run.smithi137.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 2.4 MB/s | 89 kB 00:00 2023-11-15T01:47:19.021 INFO:teuthology.orchestra.run.smithi137.stdout:(2/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 3.3 MB/s | 538 kB 00:00 2023-11-15T01:47:19.047 INFO:teuthology.orchestra.run.smithi137.stdout:(3/6): python3-apipkg-1.5-6.el8.noarch.rpm 141 kB/s | 21 kB 00:00 2023-11-15T01:47:19.114 INFO:teuthology.orchestra.run.smithi137.stdout:(4/6): python3-remoto-1.2.1-1.el8.noarch.rpm 788 kB/s | 49 kB 00:00 2023-11-15T01:47:19.272 INFO:teuthology.orchestra.run.smithi137.stdout:(5/6): python3-babel-2.5.1-7.el8.noarch.rpm 12 MB/s | 4.8 MB 00:00 2023-11-15T01:47:19.331 INFO:teuthology.orchestra.run.smithi137.stdout:(6/6): python3-execnet-1.7.1-1.el8.noarch.rpm 1.1 MB/s | 355 kB 00:00 2023-11-15T01:47:19.332 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2023-11-15T01:47:19.333 INFO:teuthology.orchestra.run.smithi137.stdout:Total 6.3 MB/s | 5.8 MB 00:00 2023-11-15T01:47:19.384 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2023-11-15T01:47:19.394 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2023-11-15T01:47:19.394 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2023-11-15T01:47:19.502 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2023-11-15T01:47:19.505 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2023-11-15T01:47:19.818 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2023-11-15T01:47:20.066 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2023-11-15T01:47:20.369 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2023-11-15T01:47:20.479 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2023-11-15T01:47:20.481 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-15T01:47:20.481 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repository Size 2023-11-15T01:47:20.481 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-15T01:47:20.481 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2023-11-15T01:47:20.481 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2023-11-15T01:47:20.482 INFO:teuthology.orchestra.run.smithi055.stdout:Installing dependencies: 2023-11-15T01:47:20.482 INFO:teuthology.orchestra.run.smithi055.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2023-11-15T01:47:20.482 INFO:teuthology.orchestra.run.smithi055.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2023-11-15T01:47:20.482 INFO:teuthology.orchestra.run.smithi055.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2023-11-15T01:47:20.482 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2023-11-15T01:47:20.483 INFO:teuthology.orchestra.run.smithi055.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2023-11-15T01:47:20.485 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-15T01:47:20.486 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2023-11-15T01:47:20.486 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-15T01:47:20.486 INFO:teuthology.orchestra.run.smithi055.stdout:Install 6 Packages 2023-11-15T01:47:20.486 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-15T01:47:20.487 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 5.8 M 2023-11-15T01:47:20.487 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 24 M 2023-11-15T01:47:20.487 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2023-11-15T01:47:21.252 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2023-11-15T01:47:21.260 INFO:teuthology.orchestra.run.smithi055.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 2.1 MB/s | 89 kB 00:00 2023-11-15T01:47:21.393 INFO:teuthology.orchestra.run.smithi055.stdout:(2/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 3.0 MB/s | 538 kB 00:00 2023-11-15T01:47:21.410 INFO:teuthology.orchestra.run.smithi055.stdout:(3/6): python3-apipkg-1.5-6.el8.noarch.rpm 141 kB/s | 21 kB 00:00 2023-11-15T01:47:21.579 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2023-11-15T01:47:21.611 INFO:teuthology.orchestra.run.smithi055.stdout:(4/6): python3-babel-2.5.1-7.el8.noarch.rpm 12 MB/s | 4.8 MB 00:00 2023-11-15T01:47:21.636 INFO:teuthology.orchestra.run.smithi055.stdout:(5/6): python3-remoto-1.2.1-1.el8.noarch.rpm 217 kB/s | 49 kB 00:00 2023-11-15T01:47:21.669 INFO:teuthology.orchestra.run.smithi055.stdout:(6/6): python3-execnet-1.7.1-1.el8.noarch.rpm 1.3 MB/s | 355 kB 00:00 2023-11-15T01:47:21.670 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2023-11-15T01:47:21.670 INFO:teuthology.orchestra.run.smithi055.stdout:Total 4.9 MB/s | 5.8 MB 00:01 2023-11-15T01:47:21.727 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2023-11-15T01:47:21.739 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2023-11-15T01:47:21.739 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2023-11-15T01:47:21.847 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2023-11-15T01:47:21.850 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2023-11-15T01:47:21.988 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2023-11-15T01:47:22.136 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2023-11-15T01:47:22.209 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2023-11-15T01:47:22.397 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2023-11-15T01:47:22.497 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2023-11-15T01:47:22.498 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2023-11-15T01:47:22.498 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2023-11-15T01:47:22.498 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2023-11-15T01:47:22.499 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2023-11-15T01:47:22.499 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2023-11-15T01:47:22.672 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2023-11-15T01:47:23.238 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2023-11-15T01:47:23.238 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:47:23.238 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2023-11-15T01:47:23.238 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2023-11-15T01:47:23.239 INFO:teuthology.orchestra.run.smithi137.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2023-11-15T01:47:23.239 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jinja2-2.10.1-3.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2023-11-15T01:47:23.239 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:47:23.239 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2023-11-15T01:47:23.405 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-fuse 2023-11-15T01:47:23.690 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2023-11-15T01:47:24.058 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2023-11-15T01:47:24.266 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:02:07 ago on Wed 15 Nov 2023 01:45:17 AM UTC. 2023-11-15T01:47:24.380 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2023-11-15T01:47:24.569 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2023-11-15T01:47:24.895 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2023-11-15T01:47:24.895 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2023-11-15T01:47:24.895 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2023-11-15T01:47:24.896 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2023-11-15T01:47:24.896 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2023-11-15T01:47:24.896 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2023-11-15T01:47:25.651 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2023-11-15T01:47:25.652 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-15T01:47:25.652 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2023-11-15T01:47:25.652 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2023-11-15T01:47:25.652 INFO:teuthology.orchestra.run.smithi055.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2023-11-15T01:47:25.652 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jinja2-2.10.1-3.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2023-11-15T01:47:25.653 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-15T01:47:25.653 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2023-11-15T01:47:25.788 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph-fuse 2023-11-15T01:47:25.936 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2023-11-15T01:47:25.937 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T01:47:25.937 INFO:teuthology.orchestra.run.smithi137.stdout: Package Architecture Version Repository Size 2023-11-15T01:47:25.937 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T01:47:25.937 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2023-11-15T01:47:25.938 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2023-11-15T01:47:25.938 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2023-11-15T01:47:25.938 INFO:teuthology.orchestra.run.smithi137.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2023-11-15T01:47:25.938 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:47:25.938 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2023-11-15T01:47:25.939 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T01:47:25.939 INFO:teuthology.orchestra.run.smithi137.stdout:Install 2 Packages 2023-11-15T01:47:25.939 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:47:25.940 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 968 k 2023-11-15T01:47:25.940 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 3.2 M 2023-11-15T01:47:25.940 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2023-11-15T01:47:26.042 INFO:teuthology.orchestra.run.smithi137.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 13 MB/s | 885 kB 00:00 2023-11-15T01:47:26.301 INFO:teuthology.orchestra.run.smithi137.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 257 kB/s | 83 kB 00:00 2023-11-15T01:47:26.301 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2023-11-15T01:47:26.302 INFO:teuthology.orchestra.run.smithi137.stdout:Total 2.6 MB/s | 968 kB 00:00 2023-11-15T01:47:26.312 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2023-11-15T01:47:26.333 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2023-11-15T01:47:26.333 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2023-11-15T01:47:26.419 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2023-11-15T01:47:26.422 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2023-11-15T01:47:26.623 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2023-11-15T01:47:26.776 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:02:14 ago on Wed 15 Nov 2023 01:45:12 AM UTC. 2023-11-15T01:47:26.891 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2023-11-15T01:47:27.141 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2023-11-15T01:47:27.539 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2023-11-15T01:47:27.539 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2023-11-15T01:47:28.208 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2023-11-15T01:47:28.208 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:47:28.209 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2023-11-15T01:47:28.209 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2023-11-15T01:47:28.209 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:47:28.209 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2023-11-15T01:47:28.432 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install librados-devel 2023-11-15T01:47:28.617 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2023-11-15T01:47:28.618 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-15T01:47:28.618 INFO:teuthology.orchestra.run.smithi055.stdout: Package Architecture Version Repository Size 2023-11-15T01:47:28.618 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-15T01:47:28.619 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2023-11-15T01:47:28.619 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2023-11-15T01:47:28.619 INFO:teuthology.orchestra.run.smithi055.stdout:Installing dependencies: 2023-11-15T01:47:28.619 INFO:teuthology.orchestra.run.smithi055.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2023-11-15T01:47:28.619 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-15T01:47:28.620 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2023-11-15T01:47:28.620 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-15T01:47:28.620 INFO:teuthology.orchestra.run.smithi055.stdout:Install 2 Packages 2023-11-15T01:47:28.620 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-15T01:47:28.621 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 968 k 2023-11-15T01:47:28.621 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 3.2 M 2023-11-15T01:47:28.621 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2023-11-15T01:47:28.714 INFO:teuthology.orchestra.run.smithi055.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 16 MB/s | 885 kB 00:00 2023-11-15T01:47:29.056 INFO:teuthology.orchestra.run.smithi055.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 210 kB/s | 83 kB 00:00 2023-11-15T01:47:29.056 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2023-11-15T01:47:29.057 INFO:teuthology.orchestra.run.smithi055.stdout:Total 2.2 MB/s | 968 kB 00:00 2023-11-15T01:47:29.069 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2023-11-15T01:47:29.094 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2023-11-15T01:47:29.094 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2023-11-15T01:47:29.195 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2023-11-15T01:47:29.199 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2023-11-15T01:47:29.304 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:02:12 ago on Wed 15 Nov 2023 01:45:17 AM UTC. 2023-11-15T01:47:29.385 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2023-11-15T01:47:29.652 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2023-11-15T01:47:29.829 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2023-11-15T01:47:30.284 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2023-11-15T01:47:30.285 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2023-11-15T01:47:30.972 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2023-11-15T01:47:30.972 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T01:47:30.973 INFO:teuthology.orchestra.run.smithi137.stdout: Package Architecture Version Repository Size 2023-11-15T01:47:30.973 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T01:47:30.973 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2023-11-15T01:47:30.973 INFO:teuthology.orchestra.run.smithi137.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2023-11-15T01:47:30.973 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:47:30.974 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2023-11-15T01:47:30.974 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T01:47:30.974 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2023-11-15T01:47:30.974 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:47:30.975 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 131 k 2023-11-15T01:47:30.975 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 439 k 2023-11-15T01:47:30.975 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2023-11-15T01:47:31.022 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2023-11-15T01:47:31.023 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-15T01:47:31.023 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2023-11-15T01:47:31.023 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2023-11-15T01:47:31.023 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-15T01:47:31.023 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2023-11-15T01:47:31.029 INFO:teuthology.orchestra.run.smithi137.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 2.3 MB/s | 131 kB 00:00 2023-11-15T01:47:31.030 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2023-11-15T01:47:31.030 INFO:teuthology.orchestra.run.smithi137.stdout:Total 2.3 MB/s | 131 kB 00:00 2023-11-15T01:47:31.030 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2023-11-15T01:47:31.035 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2023-11-15T01:47:31.036 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2023-11-15T01:47:31.085 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2023-11-15T01:47:31.088 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2023-11-15T01:47:31.199 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2023-11-15T01:47:31.227 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install librados-devel 2023-11-15T01:47:31.464 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T01:47:31.757 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T01:47:32.228 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:02:20 ago on Wed 15 Nov 2023 01:45:12 AM UTC. 2023-11-15T01:47:32.415 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T01:47:32.416 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:47:32.416 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2023-11-15T01:47:32.416 INFO:teuthology.orchestra.run.smithi137.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2023-11-15T01:47:32.416 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:47:32.416 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2023-11-15T01:47:33.833 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install libcephfs2 2023-11-15T01:47:34.082 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2023-11-15T01:47:34.083 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-15T01:47:34.083 INFO:teuthology.orchestra.run.smithi055.stdout: Package Architecture Version Repository Size 2023-11-15T01:47:34.083 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-15T01:47:34.084 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2023-11-15T01:47:34.084 INFO:teuthology.orchestra.run.smithi055.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2023-11-15T01:47:34.084 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-15T01:47:34.084 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2023-11-15T01:47:34.084 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-15T01:47:34.085 INFO:teuthology.orchestra.run.smithi055.stdout:Install 1 Package 2023-11-15T01:47:34.085 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-15T01:47:34.085 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 131 k 2023-11-15T01:47:34.086 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 439 k 2023-11-15T01:47:34.086 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2023-11-15T01:47:34.504 INFO:teuthology.orchestra.run.smithi055.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 313 kB/s | 131 kB 00:00 2023-11-15T01:47:34.504 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2023-11-15T01:47:34.505 INFO:teuthology.orchestra.run.smithi055.stdout:Total 312 kB/s | 131 kB 00:00 2023-11-15T01:47:34.505 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2023-11-15T01:47:34.512 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2023-11-15T01:47:34.512 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2023-11-15T01:47:34.574 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2023-11-15T01:47:34.577 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2023-11-15T01:47:34.705 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:02:17 ago on Wed 15 Nov 2023 01:45:17 AM UTC. 2023-11-15T01:47:35.023 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2023-11-15T01:47:35.635 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T01:47:35.975 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T01:47:36.132 INFO:teuthology.orchestra.run.smithi137.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T01:47:36.345 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2023-11-15T01:47:36.346 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2023-11-15T01:47:36.347 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2023-11-15T01:47:36.475 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install libcephfs-devel 2023-11-15T01:47:36.717 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T01:47:36.718 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-15T01:47:36.718 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2023-11-15T01:47:36.718 INFO:teuthology.orchestra.run.smithi055.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2023-11-15T01:47:36.719 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-15T01:47:36.719 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2023-11-15T01:47:36.931 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install libcephfs2 2023-11-15T01:47:37.337 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:02:20 ago on Wed 15 Nov 2023 01:45:17 AM UTC. 2023-11-15T01:47:37.952 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:02:25 ago on Wed 15 Nov 2023 01:45:12 AM UTC. 2023-11-15T01:47:39.002 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2023-11-15T01:47:39.003 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T01:47:39.003 INFO:teuthology.orchestra.run.smithi137.stdout: Package Architecture Version Repository Size 2023-11-15T01:47:39.003 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T01:47:39.003 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2023-11-15T01:47:39.003 INFO:teuthology.orchestra.run.smithi137.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2023-11-15T01:47:39.004 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:47:39.004 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2023-11-15T01:47:39.004 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T01:47:39.004 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2023-11-15T01:47:39.004 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:47:39.005 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 24 k 2023-11-15T01:47:39.005 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 94 k 2023-11-15T01:47:39.005 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2023-11-15T01:47:39.045 INFO:teuthology.orchestra.run.smithi137.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 601 kB/s | 24 kB 00:00 2023-11-15T01:47:39.045 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2023-11-15T01:47:39.046 INFO:teuthology.orchestra.run.smithi137.stdout:Total 582 kB/s | 24 kB 00:00 2023-11-15T01:47:39.046 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2023-11-15T01:47:39.049 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2023-11-15T01:47:39.049 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2023-11-15T01:47:39.051 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2023-11-15T01:47:39.054 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2023-11-15T01:47:39.107 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2023-11-15T01:47:39.363 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T01:47:39.569 INFO:teuthology.orchestra.run.smithi055.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T01:47:39.634 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T01:47:39.815 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2023-11-15T01:47:39.816 INFO:teuthology.orchestra.run.smithi055.stdout:Nothing to do. 2023-11-15T01:47:39.816 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2023-11-15T01:47:39.922 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install libcephfs-devel 2023-11-15T01:47:40.261 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T01:47:40.261 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:47:40.261 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2023-11-15T01:47:40.261 INFO:teuthology.orchestra.run.smithi137.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2023-11-15T01:47:40.262 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:47:40.262 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2023-11-15T01:47:40.479 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install librados2 2023-11-15T01:47:40.918 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:02:28 ago on Wed 15 Nov 2023 01:45:12 AM UTC. 2023-11-15T01:47:41.349 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:02:24 ago on Wed 15 Nov 2023 01:45:17 AM UTC. 2023-11-15T01:47:42.779 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2023-11-15T01:47:42.780 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-15T01:47:42.780 INFO:teuthology.orchestra.run.smithi055.stdout: Package Architecture Version Repository Size 2023-11-15T01:47:42.780 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-15T01:47:42.780 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2023-11-15T01:47:42.780 INFO:teuthology.orchestra.run.smithi055.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2023-11-15T01:47:42.781 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-15T01:47:42.781 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2023-11-15T01:47:42.781 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-15T01:47:42.781 INFO:teuthology.orchestra.run.smithi055.stdout:Install 1 Package 2023-11-15T01:47:42.781 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-15T01:47:42.782 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 24 k 2023-11-15T01:47:42.782 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 94 k 2023-11-15T01:47:42.783 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2023-11-15T01:47:42.799 INFO:teuthology.orchestra.run.smithi137.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T01:47:42.825 INFO:teuthology.orchestra.run.smithi055.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 562 kB/s | 24 kB 00:00 2023-11-15T01:47:42.825 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2023-11-15T01:47:42.826 INFO:teuthology.orchestra.run.smithi055.stdout:Total 542 kB/s | 24 kB 00:00 2023-11-15T01:47:42.826 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2023-11-15T01:47:42.830 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2023-11-15T01:47:42.830 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2023-11-15T01:47:42.833 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2023-11-15T01:47:42.836 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2023-11-15T01:47:42.887 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2023-11-15T01:47:43.013 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2023-11-15T01:47:43.014 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2023-11-15T01:47:43.014 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2023-11-15T01:47:43.067 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T01:47:43.142 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install librbd1 2023-11-15T01:47:43.357 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T01:47:44.004 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:02:27 ago on Wed 15 Nov 2023 01:45:17 AM UTC. 2023-11-15T01:47:44.085 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T01:47:44.085 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-15T01:47:44.086 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2023-11-15T01:47:44.086 INFO:teuthology.orchestra.run.smithi055.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2023-11-15T01:47:44.086 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-15T01:47:44.087 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2023-11-15T01:47:44.196 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install librados2 2023-11-15T01:47:45.195 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:02:33 ago on Wed 15 Nov 2023 01:45:12 AM UTC. 2023-11-15T01:47:45.444 INFO:teuthology.orchestra.run.smithi137.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T01:47:45.658 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2023-11-15T01:47:45.659 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2023-11-15T01:47:45.659 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2023-11-15T01:47:45.771 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install python3-rados 2023-11-15T01:47:46.638 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:02:29 ago on Wed 15 Nov 2023 01:45:17 AM UTC. 2023-11-15T01:47:46.823 INFO:teuthology.orchestra.run.smithi055.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T01:47:47.062 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2023-11-15T01:47:47.064 INFO:teuthology.orchestra.run.smithi055.stdout:Nothing to do. 2023-11-15T01:47:47.064 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2023-11-15T01:47:47.173 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install librbd1 2023-11-15T01:47:48.087 INFO:teuthology.orchestra.run.smithi137.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T01:47:48.177 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:02:36 ago on Wed 15 Nov 2023 01:45:12 AM UTC. 2023-11-15T01:47:48.303 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2023-11-15T01:47:48.304 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2023-11-15T01:47:48.304 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2023-11-15T01:47:48.417 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install python3-rgw 2023-11-15T01:47:49.277 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:02:32 ago on Wed 15 Nov 2023 01:45:17 AM UTC. 2023-11-15T01:47:49.790 INFO:teuthology.orchestra.run.smithi055.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T01:47:50.034 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2023-11-15T01:47:50.036 INFO:teuthology.orchestra.run.smithi055.stdout:Nothing to do. 2023-11-15T01:47:50.036 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2023-11-15T01:47:50.136 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install python3-rados 2023-11-15T01:47:50.777 INFO:teuthology.orchestra.run.smithi137.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T01:47:50.992 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2023-11-15T01:47:50.993 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2023-11-15T01:47:50.994 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2023-11-15T01:47:51.110 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install python3-cephfs 2023-11-15T01:47:51.124 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:02:39 ago on Wed 15 Nov 2023 01:45:12 AM UTC. 2023-11-15T01:47:51.968 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:02:34 ago on Wed 15 Nov 2023 01:45:17 AM UTC. 2023-11-15T01:47:52.741 INFO:teuthology.orchestra.run.smithi055.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T01:47:52.982 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2023-11-15T01:47:52.984 INFO:teuthology.orchestra.run.smithi055.stdout:Nothing to do. 2023-11-15T01:47:52.984 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2023-11-15T01:47:53.096 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install python3-rgw 2023-11-15T01:47:53.418 INFO:teuthology.orchestra.run.smithi137.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T01:47:53.634 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2023-11-15T01:47:53.635 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2023-11-15T01:47:53.636 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2023-11-15T01:47:53.752 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install python3-rbd 2023-11-15T01:47:54.089 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:02:42 ago on Wed 15 Nov 2023 01:45:12 AM UTC. 2023-11-15T01:47:54.608 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:02:37 ago on Wed 15 Nov 2023 01:45:17 AM UTC. 2023-11-15T01:47:55.704 INFO:teuthology.orchestra.run.smithi055.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T01:47:55.949 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2023-11-15T01:47:55.950 INFO:teuthology.orchestra.run.smithi055.stdout:Nothing to do. 2023-11-15T01:47:55.951 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2023-11-15T01:47:56.046 INFO:teuthology.orchestra.run.smithi137.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T01:47:56.062 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install python3-cephfs 2023-11-15T01:47:56.263 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2023-11-15T01:47:56.264 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2023-11-15T01:47:56.264 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2023-11-15T01:47:56.383 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install rbd-fuse 2023-11-15T01:47:57.053 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:02:45 ago on Wed 15 Nov 2023 01:45:12 AM UTC. 2023-11-15T01:47:57.246 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:02:40 ago on Wed 15 Nov 2023 01:45:17 AM UTC. 2023-11-15T01:47:58.663 INFO:teuthology.orchestra.run.smithi055.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T01:47:58.909 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2023-11-15T01:47:58.911 INFO:teuthology.orchestra.run.smithi055.stdout:Nothing to do. 2023-11-15T01:47:58.911 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2023-11-15T01:47:58.923 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2023-11-15T01:47:58.924 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T01:47:58.924 INFO:teuthology.orchestra.run.smithi137.stdout: Package Architecture Version Repository Size 2023-11-15T01:47:58.924 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T01:47:58.925 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2023-11-15T01:47:58.925 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2023-11-15T01:47:58.925 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:47:58.925 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2023-11-15T01:47:58.925 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T01:47:58.926 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2023-11-15T01:47:58.926 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:47:58.926 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 89 k 2023-11-15T01:47:58.926 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 298 k 2023-11-15T01:47:58.926 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2023-11-15T01:47:58.984 INFO:teuthology.orchestra.run.smithi137.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 1.5 MB/s | 89 kB 00:00 2023-11-15T01:47:58.984 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2023-11-15T01:47:58.985 INFO:teuthology.orchestra.run.smithi137.stdout:Total 1.4 MB/s | 89 kB 00:00 2023-11-15T01:47:58.985 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2023-11-15T01:47:58.994 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2023-11-15T01:47:58.994 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2023-11-15T01:47:59.004 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install python3-rbd 2023-11-15T01:47:59.042 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2023-11-15T01:47:59.046 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2023-11-15T01:47:59.166 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2023-11-15T01:47:59.343 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T01:47:59.634 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T01:48:00.006 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:02:48 ago on Wed 15 Nov 2023 01:45:12 AM UTC. 2023-11-15T01:48:00.362 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T01:48:00.363 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:48:00.363 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2023-11-15T01:48:00.363 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2023-11-15T01:48:00.363 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:48:00.364 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2023-11-15T01:48:00.629 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install rbd-mirror 2023-11-15T01:48:01.498 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:02:44 ago on Wed 15 Nov 2023 01:45:17 AM UTC. 2023-11-15T01:48:01.614 INFO:teuthology.orchestra.run.smithi055.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T01:48:01.854 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2023-11-15T01:48:01.856 INFO:teuthology.orchestra.run.smithi055.stdout:Nothing to do. 2023-11-15T01:48:01.856 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2023-11-15T01:48:01.963 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install rbd-fuse 2023-11-15T01:48:02.961 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:02:50 ago on Wed 15 Nov 2023 01:45:12 AM UTC. 2023-11-15T01:48:03.145 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2023-11-15T01:48:03.146 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T01:48:03.146 INFO:teuthology.orchestra.run.smithi137.stdout: Package Architecture Version Repository Size 2023-11-15T01:48:03.146 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T01:48:03.147 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2023-11-15T01:48:03.147 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2023-11-15T01:48:03.147 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:48:03.147 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2023-11-15T01:48:03.147 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T01:48:03.148 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2023-11-15T01:48:03.148 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:48:03.148 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 3.7 M 2023-11-15T01:48:03.148 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 18 M 2023-11-15T01:48:03.148 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2023-11-15T01:48:03.246 INFO:teuthology.orchestra.run.smithi137.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 38 MB/s | 3.7 MB 00:00 2023-11-15T01:48:03.247 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2023-11-15T01:48:03.247 INFO:teuthology.orchestra.run.smithi137.stdout:Total 37 MB/s | 3.7 MB 00:00 2023-11-15T01:48:03.248 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2023-11-15T01:48:03.258 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2023-11-15T01:48:03.259 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2023-11-15T01:48:03.322 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2023-11-15T01:48:03.325 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2023-11-15T01:48:03.833 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2023-11-15T01:48:04.069 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T01:48:04.484 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T01:48:04.820 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2023-11-15T01:48:04.821 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-15T01:48:04.821 INFO:teuthology.orchestra.run.smithi055.stdout: Package Architecture Version Repository Size 2023-11-15T01:48:04.821 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-15T01:48:04.821 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2023-11-15T01:48:04.822 INFO:teuthology.orchestra.run.smithi055.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2023-11-15T01:48:04.822 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-15T01:48:04.822 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2023-11-15T01:48:04.822 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-15T01:48:04.823 INFO:teuthology.orchestra.run.smithi055.stdout:Install 1 Package 2023-11-15T01:48:04.823 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-15T01:48:04.823 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 89 k 2023-11-15T01:48:04.824 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 298 k 2023-11-15T01:48:04.824 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2023-11-15T01:48:04.856 INFO:teuthology.orchestra.run.smithi055.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 2.6 MB/s | 89 kB 00:00 2023-11-15T01:48:04.857 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2023-11-15T01:48:04.857 INFO:teuthology.orchestra.run.smithi055.stdout:Total 2.5 MB/s | 89 kB 00:00 2023-11-15T01:48:04.857 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2023-11-15T01:48:04.868 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2023-11-15T01:48:04.869 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2023-11-15T01:48:04.925 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2023-11-15T01:48:04.928 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2023-11-15T01:48:05.103 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2023-11-15T01:48:05.221 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T01:48:05.221 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:48:05.222 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2023-11-15T01:48:05.222 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2023-11-15T01:48:05.222 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:48:05.222 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2023-11-15T01:48:05.298 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T01:48:05.511 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install rbd-nbd 2023-11-15T01:48:05.633 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T01:48:06.384 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:02:49 ago on Wed 15 Nov 2023 01:45:17 AM UTC. 2023-11-15T01:48:06.408 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T01:48:06.409 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-15T01:48:06.409 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2023-11-15T01:48:06.409 INFO:teuthology.orchestra.run.smithi055.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2023-11-15T01:48:06.409 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-15T01:48:06.409 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2023-11-15T01:48:06.618 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install rbd-mirror 2023-11-15T01:48:07.614 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:02:55 ago on Wed 15 Nov 2023 01:45:12 AM UTC. 2023-11-15T01:48:08.043 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2023-11-15T01:48:08.043 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T01:48:08.043 INFO:teuthology.orchestra.run.smithi137.stdout: Package Architecture Version Repository Size 2023-11-15T01:48:08.044 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T01:48:08.044 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2023-11-15T01:48:08.044 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2023-11-15T01:48:08.044 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:48:08.044 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2023-11-15T01:48:08.045 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T01:48:08.045 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2023-11-15T01:48:08.045 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:48:08.046 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 173 k 2023-11-15T01:48:08.046 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 649 k 2023-11-15T01:48:08.046 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2023-11-15T01:48:08.090 INFO:teuthology.orchestra.run.smithi137.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 3.7 MB/s | 173 kB 00:00 2023-11-15T01:48:08.091 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2023-11-15T01:48:08.091 INFO:teuthology.orchestra.run.smithi137.stdout:Total 3.6 MB/s | 173 kB 00:00 2023-11-15T01:48:08.091 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2023-11-15T01:48:08.102 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2023-11-15T01:48:08.102 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2023-11-15T01:48:08.152 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2023-11-15T01:48:08.155 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2023-11-15T01:48:08.272 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2023-11-15T01:48:08.483 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T01:48:08.766 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T01:48:09.429 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T01:48:09.429 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:48:09.429 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2023-11-15T01:48:09.429 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2023-11-15T01:48:09.430 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:48:09.430 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2023-11-15T01:48:09.485 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2023-11-15T01:48:09.486 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-15T01:48:09.486 INFO:teuthology.orchestra.run.smithi055.stdout: Package Architecture Version Repository Size 2023-11-15T01:48:09.486 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-15T01:48:09.486 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2023-11-15T01:48:09.487 INFO:teuthology.orchestra.run.smithi055.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2023-11-15T01:48:09.487 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-15T01:48:09.487 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2023-11-15T01:48:09.487 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-15T01:48:09.487 INFO:teuthology.orchestra.run.smithi055.stdout:Install 1 Package 2023-11-15T01:48:09.488 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-15T01:48:09.488 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 3.7 M 2023-11-15T01:48:09.488 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 18 M 2023-11-15T01:48:09.489 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2023-11-15T01:48:09.574 INFO:teuthology.orchestra.run.smithi055.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 43 MB/s | 3.7 MB 00:00 2023-11-15T01:48:09.575 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2023-11-15T01:48:09.576 INFO:teuthology.orchestra.run.smithi055.stdout:Total 42 MB/s | 3.7 MB 00:00 2023-11-15T01:48:09.576 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2023-11-15T01:48:09.590 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2023-11-15T01:48:09.590 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2023-11-15T01:48:09.635 DEBUG:teuthology.parallel:result is None 2023-11-15T01:48:09.661 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2023-11-15T01:48:09.665 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2023-11-15T01:48:10.183 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2023-11-15T01:48:10.372 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T01:48:10.844 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T01:48:11.596 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T01:48:11.597 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-15T01:48:11.597 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2023-11-15T01:48:11.597 INFO:teuthology.orchestra.run.smithi055.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2023-11-15T01:48:11.597 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-15T01:48:11.597 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2023-11-15T01:48:11.795 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install rbd-nbd 2023-11-15T01:48:12.810 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:03:00 ago on Wed 15 Nov 2023 01:45:12 AM UTC. 2023-11-15T01:48:14.658 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2023-11-15T01:48:14.659 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-15T01:48:14.659 INFO:teuthology.orchestra.run.smithi055.stdout: Package Architecture Version Repository Size 2023-11-15T01:48:14.659 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-15T01:48:14.660 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2023-11-15T01:48:14.660 INFO:teuthology.orchestra.run.smithi055.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2023-11-15T01:48:14.660 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-15T01:48:14.660 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2023-11-15T01:48:14.660 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-11-15T01:48:14.661 INFO:teuthology.orchestra.run.smithi055.stdout:Install 1 Package 2023-11-15T01:48:14.661 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-15T01:48:14.661 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 173 k 2023-11-15T01:48:14.662 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 649 k 2023-11-15T01:48:14.662 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2023-11-15T01:48:14.711 INFO:teuthology.orchestra.run.smithi055.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 3.4 MB/s | 173 kB 00:00 2023-11-15T01:48:14.712 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2023-11-15T01:48:14.712 INFO:teuthology.orchestra.run.smithi055.stdout:Total 3.3 MB/s | 173 kB 00:00 2023-11-15T01:48:14.712 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2023-11-15T01:48:14.724 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2023-11-15T01:48:14.724 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2023-11-15T01:48:14.783 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2023-11-15T01:48:14.786 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2023-11-15T01:48:14.913 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2023-11-15T01:48:15.105 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T01:48:15.438 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T01:48:16.191 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T01:48:16.191 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-15T01:48:16.192 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2023-11-15T01:48:16.192 INFO:teuthology.orchestra.run.smithi055.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2023-11-15T01:48:16.192 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-11-15T01:48:16.192 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2023-11-15T01:48:16.406 DEBUG:teuthology.parallel:result is None 2023-11-15T01:48:16.407 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-11-15T01:48:16.407 INFO:teuthology.packaging:ref: None 2023-11-15T01:48:16.407 INFO:teuthology.packaging:tag: v16.2.4 2023-11-15T01:48:16.408 INFO:teuthology.packaging:branch: None 2023-11-15T01:48:16.408 INFO:teuthology.packaging:sha1: 620f25c1794f62207424fc9319427943fb8d1fae 2023-11-15T01:48:16.630 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-11-15T01:48:16.631 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-11-15T01:48:16.877 DEBUG:teuthology.orchestra.run.smithi055:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-11-15T01:48:16.906 INFO:teuthology.orchestra.run.smithi055.stdout:16.2.4-0.el8 2023-11-15T01:48:16.907 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2023-11-15T01:48:16.907 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2023-11-15T01:48:16.908 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-11-15T01:48:16.908 INFO:teuthology.packaging:ref: None 2023-11-15T01:48:16.909 INFO:teuthology.packaging:tag: v16.2.4 2023-11-15T01:48:16.909 INFO:teuthology.packaging:branch: None 2023-11-15T01:48:16.909 INFO:teuthology.packaging:sha1: 620f25c1794f62207424fc9319427943fb8d1fae 2023-11-15T01:48:17.221 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-11-15T01:48:17.223 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-11-15T01:48:17.367 DEBUG:teuthology.orchestra.run.smithi137:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-11-15T01:48:17.394 INFO:teuthology.orchestra.run.smithi137.stdout:16.2.4-0.el8 2023-11-15T01:48:17.395 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2023-11-15T01:48:17.395 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2023-11-15T01:48:17.396 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-11-15T01:48:17.397 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2023-11-15T01:48:17.397 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-11-15T01:48:17.433 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2023-11-15T01:48:17.433 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-11-15T01:48:17.467 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-11-15T01:48:17.467 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2023-11-15T01:48:17.468 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/usr/bin/daemon-helper 2023-11-15T01:48:17.507 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-11-15T01:48:17.582 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2023-11-15T01:48:17.583 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/usr/bin/daemon-helper 2023-11-15T01:48:17.614 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-11-15T01:48:17.684 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-11-15T01:48:17.684 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2023-11-15T01:48:17.685 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/usr/bin/adjust-ulimits 2023-11-15T01:48:17.719 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-11-15T01:48:17.793 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2023-11-15T01:48:17.793 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/usr/bin/adjust-ulimits 2023-11-15T01:48:17.822 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-11-15T01:48:17.889 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-11-15T01:48:17.890 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2023-11-15T01:48:17.890 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/usr/bin/stdin-killer 2023-11-15T01:48:17.924 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-11-15T01:48:17.999 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2023-11-15T01:48:17.999 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/usr/bin/stdin-killer 2023-11-15T01:48:18.028 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-11-15T01:48:18.095 INFO:teuthology.run_tasks:Running task print... 2023-11-15T01:48:18.105 INFO:teuthology.task.print:**** done install task... 2023-11-15T01:48:18.106 INFO:teuthology.run_tasks:Running task cephadm... 2023-11-15T01:48:18.356 INFO:tasks.cephadm:Config: {'avoid_pacific_features': True, 'cephadm_branch': 'v16.2.4', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'docker.io/ceph/ceph:v16.2.4', 'roleless': True, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': '620f25c1794f62207424fc9319427943fb8d1fae'} 2023-11-15T01:48:18.357 INFO:tasks.cephadm:Cluster image is docker.io/ceph/ceph:v16.2.4 2023-11-15T01:48:18.358 INFO:tasks.cephadm:Cluster fsid is 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 2023-11-15T01:48:18.358 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2023-11-15T01:48:18.358 INFO:tasks.cephadm:No mon roles; fabricating mons 2023-11-15T01:48:18.358 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi055': '172.21.15.55', 'mon.smithi137': '172.21.15.137'} 2023-11-15T01:48:18.359 INFO:tasks.cephadm:Normalizing hostnames... 2023-11-15T01:48:18.359 DEBUG:teuthology.orchestra.run.smithi055:> sudo hostname $(hostname -s) 2023-11-15T01:48:18.395 DEBUG:teuthology.orchestra.run.smithi137:> sudo hostname $(hostname -s) 2023-11-15T01:48:18.424 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v16.2.4)... 2023-11-15T01:48:18.425 DEBUG:teuthology.orchestra.run.smithi055:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-11-15T01:48:18.582 INFO:teuthology.orchestra.run.smithi055.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Nov 15 01:48 /home/ubuntu/cephtest/cephadm 2023-11-15T01:48:18.583 DEBUG:teuthology.orchestra.run.smithi137:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-11-15T01:48:18.720 INFO:teuthology.orchestra.run.smithi137.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Nov 15 01:48 /home/ubuntu/cephtest/cephadm 2023-11-15T01:48:18.720 DEBUG:teuthology.orchestra.run.smithi055:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-11-15T01:48:18.745 DEBUG:teuthology.orchestra.run.smithi137:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-11-15T01:48:18.775 DEBUG:teuthology.orchestra.run.smithi055:> sudo mkdir -p /etc/ceph 2023-11-15T01:48:18.819 DEBUG:teuthology.orchestra.run.smithi137:> sudo mkdir -p /etc/ceph 2023-11-15T01:48:18.848 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod 777 /etc/ceph 2023-11-15T01:48:18.896 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 777 /etc/ceph 2023-11-15T01:48:18.924 INFO:tasks.cephadm:Writing seed config... 2023-11-15T01:48:18.944 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2023-11-15T01:48:18.945 INFO:tasks.cephadm: override: [client] debug client = 20 2023-11-15T01:48:18.945 INFO:tasks.cephadm: override: [client] debug ms = 1 2023-11-15T01:48:18.945 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2023-11-15T01:48:18.945 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2023-11-15T01:48:18.946 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2023-11-15T01:48:18.946 INFO:tasks.cephadm: override: [mds] debug mds = 20 2023-11-15T01:48:18.946 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2023-11-15T01:48:18.946 INFO:tasks.cephadm: override: [mds] debug ms = 1 2023-11-15T01:48:18.947 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2023-11-15T01:48:18.947 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2023-11-15T01:48:18.947 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2023-11-15T01:48:18.948 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2023-11-15T01:48:18.948 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2023-11-15T01:48:18.948 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2023-11-15T01:48:18.949 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2023-11-15T01:48:18.949 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2023-11-15T01:48:18.949 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2023-11-15T01:48:18.949 INFO:tasks.cephadm: override: [mon] debug mon = 20 2023-11-15T01:48:18.950 INFO:tasks.cephadm: override: [mon] debug ms = 1 2023-11-15T01:48:18.950 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2023-11-15T01:48:18.950 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2023-11-15T01:48:18.950 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2023-11-15T01:48:18.951 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2023-11-15T01:48:18.951 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2023-11-15T01:48:18.951 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2023-11-15T01:48:18.952 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2023-11-15T01:48:18.952 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2023-11-15T01:48:18.952 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2023-11-15T01:48:18.952 INFO:tasks.cephadm: override: [osd] debug ms = 1 2023-11-15T01:48:18.953 INFO:tasks.cephadm: override: [osd] debug osd = 20 2023-11-15T01:48:18.953 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2023-11-15T01:48:18.953 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2023-11-15T01:48:18.953 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2023-11-15T01:48:18.954 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2023-11-15T01:48:18.954 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2023-11-15T01:48:18.954 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2023-11-15T01:48:18.954 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2023-11-15T01:48:18.955 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2023-11-15T01:48:18.956 DEBUG:teuthology.orchestra.run.smithi055:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2023-11-15T01:48:18.978 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2023-11-15T01:48:18.979 DEBUG:teuthology.orchestra.run.smithi055:mon.smithi055> sudo journalctl -f -n 0 -u ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7@mon.smithi055.service 2023-11-15T01:48:19.021 INFO:tasks.cephadm:Bootstrapping... 2023-11-15T01:48:19.021 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 -v bootstrap --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.55 && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2023-11-15T01:48:19.055 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:-- Logs begin at Wed 2023-11-15 00:53:00 UTC. -- 2023-11-15T01:48:19.318 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2023-11-15T01:48:20.125 INFO:teuthology.orchestra.run.smithi055.stderr:/bin/podman: 4.6.1 2023-11-15T01:48:20.135 INFO:teuthology.orchestra.run.smithi055.stderr:Verifying podman|docker is present... 2023-11-15T01:48:20.135 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2023-11-15T01:48:20.161 INFO:teuthology.orchestra.run.smithi055.stderr:/bin/podman: 4.6.1 2023-11-15T01:48:20.173 INFO:teuthology.orchestra.run.smithi055.stderr:Verifying lvm2 is present... 2023-11-15T01:48:20.173 INFO:teuthology.orchestra.run.smithi055.stderr:Verifying time synchronization is in place... 2023-11-15T01:48:20.173 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: systemctl is-enabled chrony.service 2023-11-15T01:48:20.182 INFO:teuthology.orchestra.run.smithi055.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2023-11-15T01:48:20.182 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: systemctl is-active chrony.service 2023-11-15T01:48:20.189 INFO:teuthology.orchestra.run.smithi055.stderr:systemctl: inactive 2023-11-15T01:48:20.189 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: systemctl is-enabled chronyd.service 2023-11-15T01:48:20.196 INFO:teuthology.orchestra.run.smithi055.stderr:systemctl: enabled 2023-11-15T01:48:20.197 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: systemctl is-active chronyd.service 2023-11-15T01:48:20.203 INFO:teuthology.orchestra.run.smithi055.stderr:systemctl: active 2023-11-15T01:48:20.204 INFO:teuthology.orchestra.run.smithi055.stderr:Unit chronyd.service is enabled and running 2023-11-15T01:48:20.204 INFO:teuthology.orchestra.run.smithi055.stderr:Repeating the final host check... 2023-11-15T01:48:20.204 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2023-11-15T01:48:20.226 INFO:teuthology.orchestra.run.smithi055.stderr:/bin/podman: 4.6.1 2023-11-15T01:48:20.237 INFO:teuthology.orchestra.run.smithi055.stderr:podman|docker (/bin/podman) is present 2023-11-15T01:48:20.237 INFO:teuthology.orchestra.run.smithi055.stderr:systemctl is present 2023-11-15T01:48:20.238 INFO:teuthology.orchestra.run.smithi055.stderr:lvcreate is present 2023-11-15T01:48:20.238 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: systemctl is-enabled chrony.service 2023-11-15T01:48:20.244 INFO:teuthology.orchestra.run.smithi055.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2023-11-15T01:48:20.244 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: systemctl is-active chrony.service 2023-11-15T01:48:20.250 INFO:teuthology.orchestra.run.smithi055.stderr:systemctl: inactive 2023-11-15T01:48:20.251 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: systemctl is-enabled chronyd.service 2023-11-15T01:48:20.257 INFO:teuthology.orchestra.run.smithi055.stderr:systemctl: enabled 2023-11-15T01:48:20.258 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: systemctl is-active chronyd.service 2023-11-15T01:48:20.264 INFO:teuthology.orchestra.run.smithi055.stderr:systemctl: active 2023-11-15T01:48:20.264 INFO:teuthology.orchestra.run.smithi055.stderr:Unit chronyd.service is enabled and running 2023-11-15T01:48:20.265 INFO:teuthology.orchestra.run.smithi055.stderr:Host looks OK 2023-11-15T01:48:20.265 INFO:teuthology.orchestra.run.smithi055.stderr:Cluster fsid: 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 2023-11-15T01:48:20.266 INFO:teuthology.orchestra.run.smithi055.stderr:Acquiring lock 139856514181832 on /run/cephadm/0cb4fc0c-8359-11ee-8dbe-212e2dc638e7.lock 2023-11-15T01:48:20.266 INFO:teuthology.orchestra.run.smithi055.stderr:Lock 139856514181832 acquired on /run/cephadm/0cb4fc0c-8359-11ee-8dbe-212e2dc638e7.lock 2023-11-15T01:48:20.266 INFO:teuthology.orchestra.run.smithi055.stderr:Verifying IP 172.21.15.55 port 3300 ... 2023-11-15T01:48:20.266 INFO:teuthology.orchestra.run.smithi055.stderr:Verifying IP 172.21.15.55 port 6789 ... 2023-11-15T01:48:20.267 INFO:teuthology.orchestra.run.smithi055.stderr:Base mon IP is 172.21.15.55, final addrv is [v2:172.21.15.55:3300,v1:172.21.15.55:6789] 2023-11-15T01:48:20.267 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: /sbin/ip route ls 2023-11-15T01:48:20.269 INFO:teuthology.orchestra.run.smithi055.stderr:/sbin/ip: default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.55 metric 100 2023-11-15T01:48:20.269 INFO:teuthology.orchestra.run.smithi055.stderr:/sbin/ip: 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.55 metric 100 2023-11-15T01:48:20.270 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: /sbin/ip -6 route ls 2023-11-15T01:48:20.272 INFO:teuthology.orchestra.run.smithi055.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2023-11-15T01:48:20.272 INFO:teuthology.orchestra.run.smithi055.stderr:/sbin/ip: fe80::/64 dev ens1f0 proto kernel metric 256 pref medium 2023-11-15T01:48:20.273 INFO:teuthology.orchestra.run.smithi055.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 1024 expires 1797sec hoplimit 64 pref medium 2023-11-15T01:48:20.273 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: /sbin/ip -6 addr ls 2023-11-15T01:48:20.275 INFO:teuthology.orchestra.run.smithi055.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2023-11-15T01:48:20.276 INFO:teuthology.orchestra.run.smithi055.stderr:/sbin/ip: inet6 ::1/128 scope host 2023-11-15T01:48:20.276 INFO:teuthology.orchestra.run.smithi055.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2023-11-15T01:48:20.276 INFO:teuthology.orchestra.run.smithi055.stderr:/sbin/ip: 4: ens1f0: mtu 1500 state UP qlen 1000 2023-11-15T01:48:20.276 INFO:teuthology.orchestra.run.smithi055.stderr:/sbin/ip: inet6 fe80::ae1f:6bff:fef8:2606/64 scope link 2023-11-15T01:48:20.276 INFO:teuthology.orchestra.run.smithi055.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2023-11-15T01:48:20.277 INFO:teuthology.orchestra.run.smithi055.stderr:Mon IP 172.21.15.55 is in CIDR network 172.21.0.0/20 2023-11-15T01:48:20.277 INFO:teuthology.orchestra.run.smithi055.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2023-11-15T01:48:20.278 INFO:teuthology.orchestra.run.smithi055.stderr:Pulling container image docker.io/ceph/ceph:v16.2.4... 2023-11-15T01:48:20.279 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: /bin/podman pull docker.io/ceph/ceph:v16.2.4 2023-11-15T01:48:20.376 INFO:teuthology.orchestra.run.smithi055.stderr:/bin/podman: Trying to pull docker.io/ceph/ceph:v16.2.4... 2023-11-15T01:48:26.852 INFO:teuthology.orchestra.run.smithi055.stderr:/bin/podman: Getting image source signatures 2023-11-15T01:48:26.861 INFO:teuthology.orchestra.run.smithi055.stderr:/bin/podman: Copying blob sha256:551101690791055467b1840be133b2e6b029d263771bc6c46a96bb7acdacac39 2023-11-15T01:48:26.861 INFO:teuthology.orchestra.run.smithi055.stderr:/bin/podman: Copying blob sha256:7a0437f04f83f084b7ed68ad9c4a4947e12fc4e1b006b38129bac89114ec3621 2023-11-15T01:48:41.104 INFO:teuthology.orchestra.run.smithi055.stderr:/bin/podman: Copying config sha256:8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2023-11-15T01:48:41.153 INFO:teuthology.orchestra.run.smithi055.stderr:/bin/podman: Writing manifest to image destination 2023-11-15T01:48:41.399 INFO:teuthology.orchestra.run.smithi055.stderr:/bin/podman: 8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2023-11-15T01:48:41.411 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi055 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --version 2023-11-15T01:48:41.985 INFO:teuthology.orchestra.run.smithi055.stderr:ceph: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2023-11-15T01:48:42.322 INFO:teuthology.orchestra.run.smithi055.stderr:Ceph version: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2023-11-15T01:48:42.322 INFO:teuthology.orchestra.run.smithi055.stderr:Extracting ceph user uid/gid from container image... 2023-11-15T01:48:42.322 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi055 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 -c %u %g /var/lib/ceph 2023-11-15T01:48:42.588 INFO:teuthology.orchestra.run.smithi055.stderr:stat: 167 167 2023-11-15T01:48:42.927 INFO:teuthology.orchestra.run.smithi055.stderr:Creating initial keys... 2023-11-15T01:48:42.927 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi055 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2023-11-15T01:48:43.204 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-authtool: AQB7I1RljxGyCxAAo/ObRJdq6F5d0NW2bLpUvQ== 2023-11-15T01:48:43.553 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi055 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2023-11-15T01:48:43.821 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-authtool: AQB7I1RlGwg0MBAA+FbR0K3tLo/9C97erk5GFQ== 2023-11-15T01:48:44.159 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi055 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2023-11-15T01:48:44.429 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-authtool: AQB8I1RllPAXGRAAFYEl7Zqq8kUZ+46yLGLgCg== 2023-11-15T01:48:44.767 INFO:teuthology.orchestra.run.smithi055.stderr:Creating initial monmap... 2023-11-15T01:48:44.768 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/monmaptool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi055 -e CEPH_USE_RANDOM_NONCE=1 -v /tmp/ceph-tmpiji6ns6_:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --create --clobber --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 --addv smithi055 [v2:172.21.15.55:3300,v1:172.21.15.55:6789] /tmp/monmap 2023-11-15T01:48:45.071 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2023-11-15T01:48:45.072 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 2023-11-15T01:48:45.072 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-11-15T01:48:45.405 INFO:teuthology.orchestra.run.smithi055.stderr:monmaptool for smithi055 [v2:172.21.15.55:3300,v1:172.21.15.55:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2023-11-15T01:48:45.405 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/monmaptool: set fsid to 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 2023-11-15T01:48:45.405 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-11-15T01:48:45.405 INFO:teuthology.orchestra.run.smithi055.stderr: 2023-11-15T01:48:45.406 INFO:teuthology.orchestra.run.smithi055.stderr:Creating mon... 2023-11-15T01:48:45.406 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-mon --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi055 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/0cb4fc0c-8359-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /var/lib/ceph/0cb4fc0c-8359-11ee-8dbe-212e2dc638e7/mon.smithi055:/var/lib/ceph/mon/ceph-smithi055:z -v /tmp/ceph-tmpvbntucsj:/tmp/keyring:z -v /tmp/ceph-tmpiji6ns6_:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --mkfs -i smithi055 --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -c /dev/null --monmap /tmp/monmap --keyring /tmp/keyring --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true 2023-11-15T01:48:45.728 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.700+0000 7fe5ed66d700 0 set uid:gid to 167:167 (ceph:ceph) 2023-11-15T01:48:45.728 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.700+0000 7fe5ed66d700 1 imported monmap: 2023-11-15T01:48:45.729 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: epoch 0 2023-11-15T01:48:45.729 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 2023-11-15T01:48:45.729 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: last_changed 2023-11-15T01:48:45.055593+0000 2023-11-15T01:48:45.729 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: created 2023-11-15T01:48:45.055593+0000 2023-11-15T01:48:45.730 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: min_mon_release 0 (unknown) 2023-11-15T01:48:45.730 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: election_strategy: 1 2023-11-15T01:48:45.730 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: 0: [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] mon.smithi055 2023-11-15T01:48:45.730 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: 2023-11-15T01:48:45.730 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.700+0000 7fe5ed66d700 0 /usr/bin/ceph-mon: set fsid to 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 2023-11-15T01:48:45.731 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: RocksDB version: 6.8.1 2023-11-15T01:48:45.731 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: 2023-11-15T01:48:45.731 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2023-11-15T01:48:45.731 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: Compile date May 13 2021 2023-11-15T01:48:45.731 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: DB SUMMARY 2023-11-15T01:48:45.732 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: 2023-11-15T01:48:45.732 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi055/store.db dir, Total Num: 0, files: 2023-11-15T01:48:45.732 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: 2023-11-15T01:48:45.732 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi055/store.db: 2023-11-15T01:48:45.732 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: 2023-11-15T01:48:45.733 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: Options.error_if_exists: 0 2023-11-15T01:48:45.733 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: Options.create_if_missing: 1 2023-11-15T01:48:45.733 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: Options.paranoid_checks: 1 2023-11-15T01:48:45.733 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: Options.env: 0x560b291641c0 2023-11-15T01:48:45.733 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: Options.fs: Posix File System 2023-11-15T01:48:45.734 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: Options.info_log: 0x560b2a3daec0 2023-11-15T01:48:45.734 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: Options.max_file_opening_threads: 16 2023-11-15T01:48:45.734 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: Options.statistics: (nil) 2023-11-15T01:48:45.734 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: Options.use_fsync: 0 2023-11-15T01:48:45.734 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: Options.max_log_file_size: 0 2023-11-15T01:48:45.735 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2023-11-15T01:48:45.735 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: Options.log_file_time_to_roll: 0 2023-11-15T01:48:45.735 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: Options.keep_log_file_num: 1000 2023-11-15T01:48:45.735 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: Options.recycle_log_file_num: 0 2023-11-15T01:48:45.735 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: Options.allow_fallocate: 1 2023-11-15T01:48:45.736 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: Options.allow_mmap_reads: 0 2023-11-15T01:48:45.736 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: Options.allow_mmap_writes: 0 2023-11-15T01:48:45.736 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: Options.use_direct_reads: 0 2023-11-15T01:48:45.736 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-11-15T01:48:45.736 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: Options.create_missing_column_families: 0 2023-11-15T01:48:45.737 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: Options.db_log_dir: 2023-11-15T01:48:45.737 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi055/store.db 2023-11-15T01:48:45.737 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: Options.table_cache_numshardbits: 6 2023-11-15T01:48:45.737 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: Options.max_subcompactions: 1 2023-11-15T01:48:45.737 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: Options.max_background_flushes: -1 2023-11-15T01:48:45.738 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: Options.WAL_ttl_seconds: 0 2023-11-15T01:48:45.738 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: Options.WAL_size_limit_MB: 0 2023-11-15T01:48:45.738 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-11-15T01:48:45.738 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2023-11-15T01:48:45.738 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: Options.is_fd_close_on_exec: 1 2023-11-15T01:48:45.739 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: Options.advise_random_on_open: 1 2023-11-15T01:48:45.739 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: Options.db_write_buffer_size: 0 2023-11-15T01:48:45.739 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: Options.write_buffer_manager: 0x560b2a3e1530 2023-11-15T01:48:45.739 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2023-11-15T01:48:45.739 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2023-11-15T01:48:45.740 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2023-11-15T01:48:45.740 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: Options.use_adaptive_mutex: 0 2023-11-15T01:48:45.740 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: Options.rate_limiter: (nil) 2023-11-15T01:48:45.740 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-11-15T01:48:45.740 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: Options.wal_recovery_mode: 2 2023-11-15T01:48:45.741 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: Options.enable_thread_tracking: 0 2023-11-15T01:48:45.741 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: Options.enable_pipelined_write: 0 2023-11-15T01:48:45.741 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: Options.unordered_write: 0 2023-11-15T01:48:45.741 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2023-11-15T01:48:45.741 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-11-15T01:48:45.742 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2023-11-15T01:48:45.742 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2023-11-15T01:48:45.742 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: Options.row_cache: None 2023-11-15T01:48:45.742 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: Options.wal_filter: None 2023-11-15T01:48:45.742 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2023-11-15T01:48:45.743 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: Options.allow_ingest_behind: 0 2023-11-15T01:48:45.743 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: Options.preserve_deletes: 0 2023-11-15T01:48:45.743 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: Options.two_write_queues: 0 2023-11-15T01:48:45.743 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: Options.manual_wal_flush: 0 2023-11-15T01:48:45.743 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: Options.atomic_flush: 0 2023-11-15T01:48:45.744 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-11-15T01:48:45.744 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: Options.persist_stats_to_disk: 0 2023-11-15T01:48:45.744 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: Options.write_dbid_to_manifest: 0 2023-11-15T01:48:45.744 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: Options.log_readahead_size: 0 2023-11-15T01:48:45.744 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2023-11-15T01:48:45.745 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: Options.max_background_jobs: 2 2023-11-15T01:48:45.745 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: Options.max_background_compactions: -1 2023-11-15T01:48:45.745 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2023-11-15T01:48:45.745 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-11-15T01:48:45.745 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: Options.delayed_write_rate : 16777216 2023-11-15T01:48:45.746 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: Options.max_total_wal_size: 0 2023-11-15T01:48:45.746 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-11-15T01:48:45.746 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: Options.stats_dump_period_sec: 600 2023-11-15T01:48:45.746 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: Options.stats_persist_period_sec: 600 2023-11-15T01:48:45.746 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2023-11-15T01:48:45.747 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: Options.max_open_files: -1 2023-11-15T01:48:45.747 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: Options.bytes_per_sync: 0 2023-11-15T01:48:45.747 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: Options.wal_bytes_per_sync: 0 2023-11-15T01:48:45.747 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: Options.strict_bytes_per_sync: 0 2023-11-15T01:48:45.747 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: Options.compaction_readahead_size: 0 2023-11-15T01:48:45.748 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: Compression algorithms supported: 2023-11-15T01:48:45.748 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2023-11-15T01:48:45.748 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: kZSTD supported: 0 2023-11-15T01:48:45.748 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: kXpressCompression supported: 0 2023-11-15T01:48:45.748 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: kLZ4HCCompression supported: 1 2023-11-15T01:48:45.749 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: kLZ4Compression supported: 1 2023-11-15T01:48:45.749 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: kBZip2Compression supported: 0 2023-11-15T01:48:45.749 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: kZlibCompression supported: 1 2023-11-15T01:48:45.749 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: kSnappyCompression supported: 1 2023-11-15T01:48:45.749 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.701+0000 7fe5ed66d700 4 rocksdb: Fast CRC32 supported: Supported on x86 2023-11-15T01:48:45.749 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.702+0000 7fe5ed66d700 4 rocksdb: [db_impl/db_impl_open.cc:273] Creating manifest 1 2023-11-15T01:48:45.750 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: 2023-11-15T01:48:45.750 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.705+0000 7fe5ed66d700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi055/store.db/MANIFEST-000001 2023-11-15T01:48:45.750 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: 2023-11-15T01:48:45.750 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.705+0000 7fe5ed66d700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2023-11-15T01:48:45.750 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: 2023-11-15T01:48:45.751 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.705+0000 7fe5ed66d700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-11-15T01:48:45.751 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.705+0000 7fe5ed66d700 4 rocksdb: Options.merge_operator: 2023-11-15T01:48:45.753 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.705+0000 7fe5ed66d700 4 rocksdb: Options.compaction_filter: None 2023-11-15T01:48:45.753 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.705+0000 7fe5ed66d700 4 rocksdb: Options.compaction_filter_factory: None 2023-11-15T01:48:45.753 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.705+0000 7fe5ed66d700 4 rocksdb: Options.memtable_factory: SkipListFactory 2023-11-15T01:48:45.754 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.705+0000 7fe5ed66d700 4 rocksdb: Options.table_factory: BlockBasedTable 2023-11-15T01:48:45.754 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.705+0000 7fe5ed66d700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x560b2a2f2120) 2023-11-15T01:48:45.754 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks: 1 2023-11-15T01:48:45.754 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks_with_high_priority: 0 2023-11-15T01:48:45.754 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: pin_l0_filter_and_index_blocks_in_cache: 0 2023-11-15T01:48:45.755 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: pin_top_level_index_and_filter: 1 2023-11-15T01:48:45.755 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: index_type: 0 2023-11-15T01:48:45.755 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: data_block_index_type: 0 2023-11-15T01:48:45.755 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: index_shortening: 1 2023-11-15T01:48:45.755 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: data_block_hash_table_util_ratio: 0.750000 2023-11-15T01:48:45.756 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: hash_index_allow_collision: 1 2023-11-15T01:48:45.756 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: checksum: 1 2023-11-15T01:48:45.756 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: no_block_cache: 0 2023-11-15T01:48:45.756 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: block_cache: 0x560b2a328d10 2023-11-15T01:48:45.756 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: block_cache_name: BinnedLRUCache 2023-11-15T01:48:45.756 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: block_cache_options: 2023-11-15T01:48:45.757 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: capacity : 536870912 2023-11-15T01:48:45.757 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: num_shard_bits : 4 2023-11-15T01:48:45.757 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: strict_capacity_limit : 0 2023-11-15T01:48:45.757 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: high_pri_pool_ratio: 0.000 2023-11-15T01:48:45.757 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: block_cache_compressed: (nil) 2023-11-15T01:48:45.758 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: persistent_cache: (nil) 2023-11-15T01:48:45.758 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: block_size: 4096 2023-11-15T01:48:45.758 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: block_size_deviation: 10 2023-11-15T01:48:45.758 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: block_restart_interval: 16 2023-11-15T01:48:45.758 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: index_block_restart_interval: 1 2023-11-15T01:48:45.759 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: metadata_block_size: 4096 2023-11-15T01:48:45.759 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: partition_filters: 0 2023-11-15T01:48:45.759 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: use_delta_encoding: 1 2023-11-15T01:48:45.759 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: filter_policy: rocksdb.BuiltinBloomFilter 2023-11-15T01:48:45.759 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: whole_key_filtering: 1 2023-11-15T01:48:45.760 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: verify_compression: 0 2023-11-15T01:48:45.760 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: read_amp_bytes_per_bit: 0 2023-11-15T01:48:45.760 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: format_version: 2 2023-11-15T01:48:45.760 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: enable_index_compression: 1 2023-11-15T01:48:45.760 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: block_align: 0 2023-11-15T01:48:45.760 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: 2023-11-15T01:48:45.761 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.705+0000 7fe5ed66d700 4 rocksdb: Options.write_buffer_size: 33554432 2023-11-15T01:48:45.761 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.705+0000 7fe5ed66d700 4 rocksdb: Options.max_write_buffer_number: 2 2023-11-15T01:48:45.761 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.705+0000 7fe5ed66d700 4 rocksdb: Options.compression: NoCompression 2023-11-15T01:48:45.761 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.705+0000 7fe5ed66d700 4 rocksdb: Options.bottommost_compression: Disabled 2023-11-15T01:48:45.761 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.705+0000 7fe5ed66d700 4 rocksdb: Options.prefix_extractor: nullptr 2023-11-15T01:48:45.762 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.705+0000 7fe5ed66d700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-11-15T01:48:45.762 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.705+0000 7fe5ed66d700 4 rocksdb: Options.num_levels: 7 2023-11-15T01:48:45.762 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.705+0000 7fe5ed66d700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-11-15T01:48:45.762 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.705+0000 7fe5ed66d700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-11-15T01:48:45.762 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.705+0000 7fe5ed66d700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-11-15T01:48:45.763 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.705+0000 7fe5ed66d700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-11-15T01:48:45.763 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.705+0000 7fe5ed66d700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2023-11-15T01:48:45.763 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.705+0000 7fe5ed66d700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-11-15T01:48:45.763 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.705+0000 7fe5ed66d700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-11-15T01:48:45.763 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.705+0000 7fe5ed66d700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-11-15T01:48:45.764 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.705+0000 7fe5ed66d700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2023-11-15T01:48:45.764 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.705+0000 7fe5ed66d700 4 rocksdb: Options.compression_opts.window_bits: -14 2023-11-15T01:48:45.764 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.705+0000 7fe5ed66d700 4 rocksdb: Options.compression_opts.level: 32767 2023-11-15T01:48:45.764 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.705+0000 7fe5ed66d700 4 rocksdb: Options.compression_opts.strategy: 0 2023-11-15T01:48:45.764 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.705+0000 7fe5ed66d700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-11-15T01:48:45.765 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.705+0000 7fe5ed66d700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-11-15T01:48:45.765 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.705+0000 7fe5ed66d700 4 rocksdb: Options.compression_opts.enabled: false 2023-11-15T01:48:45.765 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.705+0000 7fe5ed66d700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-11-15T01:48:45.765 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.705+0000 7fe5ed66d700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-11-15T01:48:45.765 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.705+0000 7fe5ed66d700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2023-11-15T01:48:45.765 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.705+0000 7fe5ed66d700 4 rocksdb: Options.target_file_size_base: 67108864 2023-11-15T01:48:45.766 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.705+0000 7fe5ed66d700 4 rocksdb: Options.target_file_size_multiplier: 1 2023-11-15T01:48:45.766 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.705+0000 7fe5ed66d700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2023-11-15T01:48:45.766 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.705+0000 7fe5ed66d700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-11-15T01:48:45.766 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.705+0000 7fe5ed66d700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-11-15T01:48:45.766 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.705+0000 7fe5ed66d700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-11-15T01:48:45.767 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.705+0000 7fe5ed66d700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-11-15T01:48:45.767 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.705+0000 7fe5ed66d700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-11-15T01:48:45.767 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.705+0000 7fe5ed66d700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-11-15T01:48:45.767 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.705+0000 7fe5ed66d700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-11-15T01:48:45.767 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.705+0000 7fe5ed66d700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-11-15T01:48:45.768 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.705+0000 7fe5ed66d700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-11-15T01:48:45.768 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.705+0000 7fe5ed66d700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-11-15T01:48:45.768 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.705+0000 7fe5ed66d700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2023-11-15T01:48:45.768 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.705+0000 7fe5ed66d700 4 rocksdb: Options.arena_block_size: 4194304 2023-11-15T01:48:45.768 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.705+0000 7fe5ed66d700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-11-15T01:48:45.768 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.705+0000 7fe5ed66d700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-11-15T01:48:45.769 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.705+0000 7fe5ed66d700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2023-11-15T01:48:45.769 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.705+0000 7fe5ed66d700 4 rocksdb: Options.disable_auto_compactions: 0 2023-11-15T01:48:45.769 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.705+0000 7fe5ed66d700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-11-15T01:48:45.769 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.705+0000 7fe5ed66d700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-11-15T01:48:45.769 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.705+0000 7fe5ed66d700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-11-15T01:48:45.770 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.705+0000 7fe5ed66d700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-11-15T01:48:45.770 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.705+0000 7fe5ed66d700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-11-15T01:48:45.770 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.705+0000 7fe5ed66d700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-11-15T01:48:45.770 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.705+0000 7fe5ed66d700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-11-15T01:48:45.770 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.705+0000 7fe5ed66d700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-11-15T01:48:45.771 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.705+0000 7fe5ed66d700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-11-15T01:48:45.771 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.705+0000 7fe5ed66d700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-11-15T01:48:45.771 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.705+0000 7fe5ed66d700 4 rocksdb: Options.table_properties_collectors: 2023-11-15T01:48:45.771 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.705+0000 7fe5ed66d700 4 rocksdb: Options.inplace_update_support: 0 2023-11-15T01:48:45.771 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.705+0000 7fe5ed66d700 4 rocksdb: Options.inplace_update_num_locks: 10000 2023-11-15T01:48:45.772 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.705+0000 7fe5ed66d700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-11-15T01:48:45.772 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.705+0000 7fe5ed66d700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2023-11-15T01:48:45.772 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.705+0000 7fe5ed66d700 4 rocksdb: Options.memtable_huge_page_size: 0 2023-11-15T01:48:45.772 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.705+0000 7fe5ed66d700 4 rocksdb: Options.bloom_locality: 0 2023-11-15T01:48:45.773 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.705+0000 7fe5ed66d700 4 rocksdb: Options.max_successive_merges: 0 2023-11-15T01:48:45.773 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.705+0000 7fe5ed66d700 4 rocksdb: Options.optimize_filters_for_hits: 0 2023-11-15T01:48:45.773 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.705+0000 7fe5ed66d700 4 rocksdb: Options.paranoid_file_checks: 0 2023-11-15T01:48:45.773 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.705+0000 7fe5ed66d700 4 rocksdb: Options.force_consistency_checks: 0 2023-11-15T01:48:45.773 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.705+0000 7fe5ed66d700 4 rocksdb: Options.report_bg_io_stats: 0 2023-11-15T01:48:45.774 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.705+0000 7fe5ed66d700 4 rocksdb: Options.ttl: 2592000 2023-11-15T01:48:45.774 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.705+0000 7fe5ed66d700 4 rocksdb: Options.periodic_compaction_seconds: 0 2023-11-15T01:48:45.774 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.705+0000 7fe5ed66d700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi055/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-11-15T01:48:45.774 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: 2023-11-15T01:48:45.774 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.705+0000 7fe5ed66d700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2023-11-15T01:48:45.774 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: 2023-11-15T01:48:45.775 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.706+0000 7fe5ed66d700 4 rocksdb: DB pointer 0x560b2a3ed800 2023-11-15T01:48:45.775 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.706+0000 7fe5d6506700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2023-11-15T01:48:45.775 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.706+0000 7fe5d6506700 4 rocksdb: [db_impl/db_impl.cc:851] 2023-11-15T01:48:45.775 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: ** DB Stats ** 2023-11-15T01:48:45.776 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2023-11-15T01:48:45.776 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-11-15T01:48:45.776 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-11-15T01:48:45.776 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-11-15T01:48:45.776 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-11-15T01:48:45.777 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2023-11-15T01:48:45.777 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-11-15T01:48:45.777 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: 2023-11-15T01:48:45.777 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-11-15T01:48:45.777 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-15T01:48:45.778 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-15T01:48:45.778 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-11-15T01:48:45.778 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-11-15T01:48:45.778 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: 2023-11-15T01:48:45.778 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-11-15T01:48:45.779 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-15T01:48:45.779 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-15T01:48:45.779 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2023-11-15T01:48:45.779 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2023-11-15T01:48:45.779 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2023-11-15T01:48:45.780 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2023-11-15T01:48:45.780 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2023-11-15T01:48:45.780 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2023-11-15T01:48:45.780 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-15T01:48:45.780 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-15T01:48:45.781 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-11-15T01:48:45.781 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: 2023-11-15T01:48:45.781 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2023-11-15T01:48:45.781 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: 2023-11-15T01:48:45.781 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-11-15T01:48:45.782 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-15T01:48:45.782 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-15T01:48:45.782 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-11-15T01:48:45.782 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-11-15T01:48:45.782 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: 2023-11-15T01:48:45.783 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-11-15T01:48:45.783 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-15T01:48:45.783 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-15T01:48:45.783 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2023-11-15T01:48:45.783 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2023-11-15T01:48:45.784 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2023-11-15T01:48:45.784 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2023-11-15T01:48:45.784 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2023-11-15T01:48:45.784 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2023-11-15T01:48:45.784 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-15T01:48:45.785 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-15T01:48:45.785 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-11-15T01:48:45.785 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: 2023-11-15T01:48:45.785 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2023-11-15T01:48:45.785 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: 2023-11-15T01:48:45.786 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.708+0000 7fe5ed66d700 4 rocksdb: [db_impl/db_impl.cc:397] Shutdown: canceling all background work 2023-11-15T01:48:45.786 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.708+0000 7fe5ed66d700 4 rocksdb: [db_impl/db_impl.cc:573] Shutdown complete 2023-11-15T01:48:45.786 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph-mon: debug 2023-11-15T01:48:45.709+0000 7fe5ed66d700 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-smithi055 for mon.smithi055 2023-11-15T01:48:46.058 INFO:teuthology.orchestra.run.smithi055.stderr:create mon.smithi055 on 2023-11-15T01:48:46.059 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: install -d -m0770 -o 167 -g 167 /var/run/ceph/0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 2023-11-15T01:48:46.066 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: systemctl disable ceph.target 2023-11-15T01:48:46.075 INFO:teuthology.orchestra.run.smithi055.stderr:systemctl: Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2023-11-15T01:48:46.173 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: systemctl enable ceph.target 2023-11-15T01:48:46.181 INFO:teuthology.orchestra.run.smithi055.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2023-11-15T01:48:46.273 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: systemctl start ceph.target 2023-11-15T01:48:46.281 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: systemctl enable ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7.target 2023-11-15T01:48:46.287 INFO:teuthology.orchestra.run.smithi055.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7.target → /etc/systemd/system/ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7.target. 2023-11-15T01:48:46.287 INFO:teuthology.orchestra.run.smithi055.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7.target → /etc/systemd/system/ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7.target. 2023-11-15T01:48:46.377 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: systemctl start ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7.target 2023-11-15T01:48:46.385 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: systemctl daemon-reload 2023-11-15T01:48:46.483 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: systemctl stop ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7@mon.smithi055 2023-11-15T01:48:46.522 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: systemctl reset-failed ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7@mon.smithi055 2023-11-15T01:48:46.527 INFO:teuthology.orchestra.run.smithi055.stderr:systemctl: Failed to reset failed state of unit ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7@mon.smithi055.service: Unit ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7@mon.smithi055.service not loaded. 2023-11-15T01:48:46.528 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: systemctl enable ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7@mon.smithi055 2023-11-15T01:48:46.535 INFO:teuthology.orchestra.run.smithi055.stderr:systemctl: Created symlink /etc/systemd/system/ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7.target.wants/ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7@mon.smithi055.service → /etc/systemd/system/ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7@.service. 2023-11-15T01:48:46.633 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: systemctl start ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7@mon.smithi055 2023-11-15T01:48:46.959 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: systemctl is-enabled firewalld.service 2023-11-15T01:48:46.969 INFO:teuthology.orchestra.run.smithi055.stderr:systemctl: disabled 2023-11-15T01:48:46.969 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: systemctl is-active firewalld.service 2023-11-15T01:48:46.978 INFO:teuthology.orchestra.run.smithi055.stderr:systemctl: inactive 2023-11-15T01:48:46.979 INFO:teuthology.orchestra.run.smithi055.stderr:firewalld.service is not enabled 2023-11-15T01:48:46.979 INFO:teuthology.orchestra.run.smithi055.stderr:Not possible to enable service . firewalld.service is not available 2023-11-15T01:48:46.979 INFO:teuthology.orchestra.run.smithi055.stderr:Not possible to open ports <[]>. firewalld.service is not available 2023-11-15T01:48:46.980 INFO:teuthology.orchestra.run.smithi055.stderr:Waiting for mon to start... 2023-11-15T01:48:46.980 INFO:teuthology.orchestra.run.smithi055.stderr:Waiting for mon... 2023-11-15T01:48:46.980 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi055 -e CEPH_USE_RANDOM_NONCE=1 -v /var/lib/ceph/0cb4fc0c-8359-11ee-8dbe-212e2dc638e7/mon.smithi055:/var/lib/ceph/mon/ceph-smithi055:z -v /tmp/ceph-tmpc1yi9jse:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpra3bp1py:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status 2023-11-15T01:48:47.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:47 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141164]: cluster 2023-11-15T01:48:46.974407+0000 mon.smithi055 (mon.0) 1 : cluster [INF] mon.smithi055 is new leader, mons smithi055 in quorum (ranks 0) 2023-11-15T01:48:47.424 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:47.423+0000 7f90c6f43700 1 Processor -- start 2023-11-15T01:48:47.425 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:47.423+0000 7f90c6f43700 1 -- start start 2023-11-15T01:48:47.425 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:47.423+0000 7f90c6f43700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f90c00fab80 0x7f90c00fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T01:48:47.425 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:47.423+0000 7f90c6f43700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90c00fb490 con 0x7f90c00fab80 2023-11-15T01:48:47.428 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:47.423+0000 7f90c4cdf700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f90c00fab80 0x7f90c00fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:48:47.429 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:47.424+0000 7f90c4cdf700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f90c00fab80 0x7f90c00fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:43352/0 (socket says 172.21.15.55:43352) 2023-11-15T01:48:47.429 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:47.424+0000 7f90c4cdf700 1 -- 172.21.15.55:0/2854026485 learned_addr learned my addr 172.21.15.55:0/2854026485 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-11-15T01:48:47.429 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:47.424+0000 7f90c4cdf700 1 -- 172.21.15.55:0/2854026485 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90c00fb5d0 con 0x7f90c00fab80 2023-11-15T01:48:47.429 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:47.424+0000 7f90c4cdf700 1 --2- 172.21.15.55:0/2854026485 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f90c00fab80 0x7f90c00fafa0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f90bc00b5f0 tx=0x7f90bc006ee0).ready entity=mon.0 client_cookie=e96ba0328c19e7aa server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:48:47.430 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:47.424+0000 7f90b77fe700 1 -- 172.21.15.55:0/2854026485 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f90bc00d600 con 0x7f90c00fab80 2023-11-15T01:48:47.430 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:47.424+0000 7f90b77fe700 1 -- 172.21.15.55:0/2854026485 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90bc00d760 con 0x7f90c00fab80 2023-11-15T01:48:47.430 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:47.424+0000 7f90b77fe700 1 -- 172.21.15.55:0/2854026485 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f90bc00da30 con 0x7f90c00fab80 2023-11-15T01:48:47.430 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:47.425+0000 7f90c6f43700 1 -- 172.21.15.55:0/2854026485 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f90c00fab80 msgr2=0x7f90c00fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:48:47.430 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:47.425+0000 7f90c6f43700 1 --2- 172.21.15.55:0/2854026485 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f90c00fab80 0x7f90c00fafa0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f90bc00b5f0 tx=0x7f90bc006ee0).stop 2023-11-15T01:48:47.431 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:47.425+0000 7f90c6f43700 1 -- 172.21.15.55:0/2854026485 shutdown_connections 2023-11-15T01:48:47.431 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:47.425+0000 7f90c6f43700 1 --2- 172.21.15.55:0/2854026485 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f90c00fab80 0x7f90c00fafa0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:48:47.431 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:47.425+0000 7f90c6f43700 1 -- 172.21.15.55:0/2854026485 >> 172.21.15.55:0/2854026485 conn(0x7f90c00f63c0 msgr2=0x7f90c00f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:48:47.431 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:47.425+0000 7f90c6f43700 1 -- 172.21.15.55:0/2854026485 shutdown_connections 2023-11-15T01:48:47.431 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:47.425+0000 7f90c6f43700 1 -- 172.21.15.55:0/2854026485 wait complete. 2023-11-15T01:48:47.432 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:47.426+0000 7f90c6f43700 1 Processor -- start 2023-11-15T01:48:47.432 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:47.426+0000 7f90c6f43700 1 -- start start 2023-11-15T01:48:47.432 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:47.426+0000 7f90c6f43700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f90c00fab80 0x7f90c010a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:48:47.432 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:47.426+0000 7f90c6f43700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90c010a740 con 0x7f90c00fab80 2023-11-15T01:48:47.432 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:47.426+0000 7f90c4cdf700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f90c00fab80 0x7f90c010a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:48:47.433 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:47.426+0000 7f90c4cdf700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f90c00fab80 0x7f90c010a250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:43362/0 (socket says 172.21.15.55:43362) 2023-11-15T01:48:47.433 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:47.426+0000 7f90c4cdf700 1 -- 172.21.15.55:0/3092214590 learned_addr learned my addr 172.21.15.55:0/3092214590 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-11-15T01:48:47.433 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:47.427+0000 7f90c4cdf700 1 -- 172.21.15.55:0/3092214590 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90bc00b200 con 0x7f90c00fab80 2023-11-15T01:48:47.433 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:47.427+0000 7f90c4cdf700 1 --2- 172.21.15.55:0/3092214590 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f90c00fab80 0x7f90c010a250 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f90bc006c70 tx=0x7f90bc016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:48:47.433 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:47.427+0000 7f90b57fa700 1 -- 172.21.15.55:0/3092214590 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f90bc011b40 con 0x7f90c00fab80 2023-11-15T01:48:47.434 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:47.427+0000 7f90b57fa700 1 -- 172.21.15.55:0/3092214590 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90bc011ca0 con 0x7f90c00fab80 2023-11-15T01:48:47.434 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:47.427+0000 7f90c6f43700 1 -- 172.21.15.55:0/3092214590 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90c010a940 con 0x7f90c00fab80 2023-11-15T01:48:47.434 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:47.427+0000 7f90b57fa700 1 -- 172.21.15.55:0/3092214590 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f90bc01d430 con 0x7f90c00fab80 2023-11-15T01:48:47.434 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:47.427+0000 7f90c6f43700 1 -- 172.21.15.55:0/3092214590 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90c010ad50 con 0x7f90c00fab80 2023-11-15T01:48:47.434 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:47.427+0000 7f90b57fa700 1 -- 172.21.15.55:0/3092214590 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f90bc00f070 con 0x7f90c00fab80 2023-11-15T01:48:47.435 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:47.427+0000 7f90b57fa700 1 -- 172.21.15.55:0/3092214590 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f90bc027460 con 0x7f90c00fab80 2023-11-15T01:48:47.435 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:47.428+0000 7f90c6f43700 1 -- 172.21.15.55:0/3092214590 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90c0045ad0 con 0x7f90c00fab80 2023-11-15T01:48:47.435 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:47.430+0000 7f90b57fa700 1 -- 172.21.15.55:0/3092214590 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f90bc016070 con 0x7f90c00fab80 2023-11-15T01:48:47.504 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:47.503+0000 7f90c6f43700 1 -- 172.21.15.55:0/3092214590 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f90c0103c50 con 0x7f90c00fab80 2023-11-15T01:48:47.504 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:47.503+0000 7f90b57fa700 1 -- 172.21.15.55:0/3092214590 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7f90bc01d640 con 0x7f90c00fab80 2023-11-15T01:48:47.505 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: cluster: 2023-11-15T01:48:47.505 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: id: 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 2023-11-15T01:48:47.505 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: health: HEALTH_OK 2023-11-15T01:48:47.505 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:47.505 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: services: 2023-11-15T01:48:47.506 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: mon: 1 daemons, quorum smithi055 (age 0.509036s) 2023-11-15T01:48:47.506 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: mgr: no daemons active 2023-11-15T01:48:47.508 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2023-11-15T01:48:47.508 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:47.509 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: data: 2023-11-15T01:48:47.509 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2023-11-15T01:48:47.509 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2023-11-15T01:48:47.509 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2023-11-15T01:48:47.509 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: pgs: 2023-11-15T01:48:47.510 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:47.510 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:47.504+0000 7f90c6f43700 1 -- 172.21.15.55:0/3092214590 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f90c00fab80 msgr2=0x7f90c010a250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:48:47.511 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:47.504+0000 7f90c6f43700 1 --2- 172.21.15.55:0/3092214590 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f90c00fab80 0x7f90c010a250 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f90bc006c70 tx=0x7f90bc016000).stop 2023-11-15T01:48:47.511 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:47.504+0000 7f90c6f43700 1 -- 172.21.15.55:0/3092214590 shutdown_connections 2023-11-15T01:48:47.511 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:47.504+0000 7f90c6f43700 1 --2- 172.21.15.55:0/3092214590 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f90c00fab80 0x7f90c010a250 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:48:47.512 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:47.504+0000 7f90c6f43700 1 -- 172.21.15.55:0/3092214590 >> 172.21.15.55:0/3092214590 conn(0x7f90c00f63c0 msgr2=0x7f90c006ffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:48:47.512 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:47.504+0000 7f90c6f43700 1 -- 172.21.15.55:0/3092214590 shutdown_connections 2023-11-15T01:48:47.512 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:47.504+0000 7f90c6f43700 1 -- 172.21.15.55:0/3092214590 wait complete. 2023-11-15T01:48:47.857 INFO:teuthology.orchestra.run.smithi055.stderr:mon is available 2023-11-15T01:48:47.857 INFO:teuthology.orchestra.run.smithi055.stderr:Assimilating anything we can from ceph.conf... 2023-11-15T01:48:47.858 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi055 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/0cb4fc0c-8359-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpc1yi9jse:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpra3bp1py:/etc/ceph/ceph.conf:z -v /var/lib/ceph/0cb4fc0c-8359-11ee-8dbe-212e2dc638e7/mon.smithi055:/var/lib/ceph/mon/ceph-smithi055:z docker.io/ceph/ceph:v16.2.4 config assimilate-conf -i /var/lib/ceph/mon/ceph-smithi055/config 2023-11-15T01:48:48.066 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:48 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141164]: cluster 2023-11-15T01:48:46.995142+0000 mon.smithi055 (mon.0) 2 : cluster [INF] mon.smithi055 is new leader, mons smithi055 in quorum (ranks 0) 2023-11-15T01:48:48.067 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:48 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141164]: cluster 2023-11-15T01:48:46.995355+0000 mon.smithi055 (mon.0) 3 : cluster [DBG] monmap e1: 1 mons at {smithi055=[v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0]} 2023-11-15T01:48:48.067 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:48 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141164]: cluster 2023-11-15T01:48:46.997862+0000 mon.smithi055 (mon.0) 4 : cluster [DBG] fsmap 2023-11-15T01:48:48.067 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:48 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141164]: cluster 2023-11-15T01:48:47.004837+0000 mon.smithi055 (mon.0) 5 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2023-11-15T01:48:48.067 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:48 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141164]: cluster 2023-11-15T01:48:47.005467+0000 mon.smithi055 (mon.0) 6 : cluster [DBG] mgrmap e1: no daemons active 2023-11-15T01:48:48.068 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:48 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141164]: audit 2023-11-15T01:48:47.504139+0000 mon.smithi055 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.55:0/3092214590' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2023-11-15T01:48:48.269 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:48.267+0000 7f9b52f74700 1 Processor -- start 2023-11-15T01:48:48.269 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:48.267+0000 7f9b52f74700 1 -- start start 2023-11-15T01:48:48.270 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:48.267+0000 7f9b52f74700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9b4c0703f0 0x7f9b4c0fe920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T01:48:48.270 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:48.267+0000 7f9b52f74700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b4c0fee10 con 0x7f9b4c0703f0 2023-11-15T01:48:48.270 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:48.267+0000 7f9b50d10700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9b4c0703f0 0x7f9b4c0fe920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:48:48.270 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:48.268+0000 7f9b50d10700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9b4c0703f0 0x7f9b4c0fe920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:43378/0 (socket says 172.21.15.55:43378) 2023-11-15T01:48:48.270 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:48.268+0000 7f9b50d10700 1 -- 172.21.15.55:0/3811961338 learned_addr learned my addr 172.21.15.55:0/3811961338 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-11-15T01:48:48.271 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:48.268+0000 7f9b50d10700 1 -- 172.21.15.55:0/3811961338 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b4c0fef50 con 0x7f9b4c0703f0 2023-11-15T01:48:48.271 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:48.268+0000 7f9b50d10700 1 --2- 172.21.15.55:0/3811961338 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9b4c0703f0 0x7f9b4c0fe920 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f9b4000fa00 tx=0x7f9b40006ee0).ready entity=mon.0 client_cookie=cc27cc03e0ea670f server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:48:48.273 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:48.268+0000 7f9b4b7fe700 1 -- 172.21.15.55:0/3811961338 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b400099a0 con 0x7f9b4c0703f0 2023-11-15T01:48:48.274 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:48.268+0000 7f9b4b7fe700 1 -- 172.21.15.55:0/3811961338 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b40009b00 con 0x7f9b4c0703f0 2023-11-15T01:48:48.274 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:48.268+0000 7f9b4b7fe700 1 -- 172.21.15.55:0/3811961338 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b40009dd0 con 0x7f9b4c0703f0 2023-11-15T01:48:48.275 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:48.269+0000 7f9b52f74700 1 -- 172.21.15.55:0/3811961338 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9b4c0703f0 msgr2=0x7f9b4c0fe920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:48:48.275 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:48.269+0000 7f9b52f74700 1 --2- 172.21.15.55:0/3811961338 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9b4c0703f0 0x7f9b4c0fe920 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f9b4000fa00 tx=0x7f9b40006ee0).stop 2023-11-15T01:48:48.275 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:48.269+0000 7f9b52f74700 1 -- 172.21.15.55:0/3811961338 shutdown_connections 2023-11-15T01:48:48.275 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:48.269+0000 7f9b52f74700 1 --2- 172.21.15.55:0/3811961338 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9b4c0703f0 0x7f9b4c0fe920 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:48:48.275 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:48.269+0000 7f9b52f74700 1 -- 172.21.15.55:0/3811961338 >> 172.21.15.55:0/3811961338 conn(0x7f9b4c0f63c0 msgr2=0x7f9b4c0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:48:48.276 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:48.269+0000 7f9b52f74700 1 -- 172.21.15.55:0/3811961338 shutdown_connections 2023-11-15T01:48:48.276 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:48.269+0000 7f9b52f74700 1 -- 172.21.15.55:0/3811961338 wait complete. 2023-11-15T01:48:48.276 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:48.269+0000 7f9b52f74700 1 Processor -- start 2023-11-15T01:48:48.276 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:48.269+0000 7f9b52f74700 1 -- start start 2023-11-15T01:48:48.277 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:48.270+0000 7f9b52f74700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9b4c0703f0 0x7f9b4c107ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:48:48.277 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:48.270+0000 7f9b52f74700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b4c1084e0 con 0x7f9b4c0703f0 2023-11-15T01:48:48.277 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:48.270+0000 7f9b50d10700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9b4c0703f0 0x7f9b4c107ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:48:48.277 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:48.270+0000 7f9b50d10700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9b4c0703f0 0x7f9b4c107ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:43394/0 (socket says 172.21.15.55:43394) 2023-11-15T01:48:48.277 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:48.270+0000 7f9b50d10700 1 -- 172.21.15.55:0/3137190208 learned_addr learned my addr 172.21.15.55:0/3137190208 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-11-15T01:48:48.278 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:48.270+0000 7f9b50d10700 1 -- 172.21.15.55:0/3137190208 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b4000f610 con 0x7f9b4c0703f0 2023-11-15T01:48:48.278 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:48.270+0000 7f9b50d10700 1 --2- 172.21.15.55:0/3137190208 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9b4c0703f0 0x7f9b4c107ff0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f9b40006f20 tx=0x7f9b40019000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:48:48.278 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:48.271+0000 7f9b497fa700 1 -- 172.21.15.55:0/3137190208 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b40018ba0 con 0x7f9b4c0703f0 2023-11-15T01:48:48.278 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:48.271+0000 7f9b497fa700 1 -- 172.21.15.55:0/3137190208 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b40018d00 con 0x7f9b4c0703f0 2023-11-15T01:48:48.278 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:48.271+0000 7f9b497fa700 1 -- 172.21.15.55:0/3137190208 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b40022470 con 0x7f9b4c0703f0 2023-11-15T01:48:48.278 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:48.271+0000 7f9b52f74700 1 -- 172.21.15.55:0/3137190208 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b4c1086e0 con 0x7f9b4c0703f0 2023-11-15T01:48:48.279 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:48.271+0000 7f9b52f74700 1 -- 172.21.15.55:0/3137190208 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b4c108af0 con 0x7f9b4c0703f0 2023-11-15T01:48:48.279 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:48.271+0000 7f9b497fa700 1 -- 172.21.15.55:0/3137190208 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f9b40022880 con 0x7f9b4c0703f0 2023-11-15T01:48:48.279 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:48.271+0000 7f9b497fa700 1 -- 172.21.15.55:0/3137190208 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f9b4002c8c0 con 0x7f9b4c0703f0 2023-11-15T01:48:48.279 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:48.273+0000 7f9b52f74700 1 -- 172.21.15.55:0/3137190208 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b4c045ad0 con 0x7f9b4c0703f0 2023-11-15T01:48:48.280 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:48.275+0000 7f9b497fa700 1 -- 172.21.15.55:0/3137190208 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f9b40032060 con 0x7f9b4c0703f0 2023-11-15T01:48:48.346 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:48.345+0000 7f9b52f74700 1 -- 172.21.15.55:0/3137190208 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f9b4c0fa7c0 con 0x7f9b4c0703f0 2023-11-15T01:48:48.356 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:48.351+0000 7f9b497fa700 1 -- 172.21.15.55:0/3137190208 <== mon.0 v2:172.21.15.55:3300/0 7 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f9b400173e0 con 0x7f9b4c0703f0 2023-11-15T01:48:48.356 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:48.351+0000 7f9b497fa700 1 -- 172.21.15.55:0/3137190208 <== mon.0 v2:172.21.15.55:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+464 (secure 0 0 0) 0x7f9b40016450 con 0x7f9b4c0703f0 2023-11-15T01:48:48.357 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:48.357 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: [global] 2023-11-15T01:48:48.357 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: fsid = 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 2023-11-15T01:48:48.357 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.55:3300,v1:172.21.15.55:6789] 2023-11-15T01:48:48.358 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2023-11-15T01:48:48.358 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2023-11-15T01:48:48.358 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2023-11-15T01:48:48.359 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2023-11-15T01:48:48.359 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:48.360 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: [mgr] 2023-11-15T01:48:48.360 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2023-11-15T01:48:48.361 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:48.361 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: [osd] 2023-11-15T01:48:48.361 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: mon_osd_backfillfull_ratio = 0.85 2023-11-15T01:48:48.362 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: mon_osd_full_ratio = 0.9 2023-11-15T01:48:48.362 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: mon_osd_nearfull_ratio = 0.8 2023-11-15T01:48:48.363 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2023-11-15T01:48:48.363 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: osd_memory_target_autotune = true 2023-11-15T01:48:48.363 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2023-11-15T01:48:48.364 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:48.352+0000 7f9b52f74700 1 -- 172.21.15.55:0/3137190208 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9b4c0703f0 msgr2=0x7f9b4c107ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:48:48.364 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:48.352+0000 7f9b52f74700 1 --2- 172.21.15.55:0/3137190208 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9b4c0703f0 0x7f9b4c107ff0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f9b40006f20 tx=0x7f9b40019000).stop 2023-11-15T01:48:48.364 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:48.352+0000 7f9b52f74700 1 -- 172.21.15.55:0/3137190208 shutdown_connections 2023-11-15T01:48:48.364 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:48.352+0000 7f9b52f74700 1 --2- 172.21.15.55:0/3137190208 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9b4c0703f0 0x7f9b4c107ff0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:48:48.365 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:48.352+0000 7f9b52f74700 1 -- 172.21.15.55:0/3137190208 >> 172.21.15.55:0/3137190208 conn(0x7f9b4c0f63c0 msgr2=0x7f9b4c0fc4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:48:48.365 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:48.352+0000 7f9b52f74700 1 -- 172.21.15.55:0/3137190208 shutdown_connections 2023-11-15T01:48:48.365 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:48.352+0000 7f9b52f74700 1 -- 172.21.15.55:0/3137190208 wait complete. 2023-11-15T01:48:48.699 INFO:teuthology.orchestra.run.smithi055.stderr:Generating new minimal ceph.conf... 2023-11-15T01:48:48.699 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi055 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/0cb4fc0c-8359-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpc1yi9jse:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpra3bp1py:/etc/ceph/ceph.conf:z -v /var/lib/ceph/0cb4fc0c-8359-11ee-8dbe-212e2dc638e7/mon.smithi055:/var/lib/ceph/mon/ceph-smithi055:z docker.io/ceph/ceph:v16.2.4 config generate-minimal-conf -o /var/lib/ceph/mon/ceph-smithi055/config 2023-11-15T01:48:49.163 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:49.161+0000 7f5bc8e18700 1 Processor -- start 2023-11-15T01:48:49.163 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:49.161+0000 7f5bc8e18700 1 -- start start 2023-11-15T01:48:49.163 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:49.161+0000 7f5bc8e18700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5bc40fbaf0 0x7f5bc40fbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T01:48:49.163 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:49.161+0000 7f5bc8e18700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bc40fc400 con 0x7f5bc40fbaf0 2023-11-15T01:48:49.163 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:49.161+0000 7f5bc37fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5bc40fbaf0 0x7f5bc40fbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:48:49.164 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:49.162+0000 7f5bc37fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5bc40fbaf0 0x7f5bc40fbf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:43402/0 (socket says 172.21.15.55:43402) 2023-11-15T01:48:49.164 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:49.162+0000 7f5bc37fe700 1 -- 172.21.15.55:0/2530966945 learned_addr learned my addr 172.21.15.55:0/2530966945 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-11-15T01:48:49.167 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:49.162+0000 7f5bc37fe700 1 -- 172.21.15.55:0/2530966945 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bc40fc540 con 0x7f5bc40fbaf0 2023-11-15T01:48:49.167 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:49.162+0000 7f5bc37fe700 1 --2- 172.21.15.55:0/2530966945 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5bc40fbaf0 0x7f5bc40fbf10 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f5bb400b5f0 tx=0x7f5bb4006ee0).ready entity=mon.0 client_cookie=2205a209b78c5971 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:48:49.168 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:49.162+0000 7f5bc27fc700 1 -- 172.21.15.55:0/2530966945 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5bb400d600 con 0x7f5bc40fbaf0 2023-11-15T01:48:49.168 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:49.162+0000 7f5bc27fc700 1 -- 172.21.15.55:0/2530966945 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f5bb400dbd0 con 0x7f5bc40fbaf0 2023-11-15T01:48:49.168 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:49.162+0000 7f5bc27fc700 1 -- 172.21.15.55:0/2530966945 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5bb4011c00 con 0x7f5bc40fbaf0 2023-11-15T01:48:49.168 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:49.163+0000 7f5bc8e18700 1 -- 172.21.15.55:0/2530966945 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5bc40fbaf0 msgr2=0x7f5bc40fbf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:48:49.168 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:49.163+0000 7f5bc8e18700 1 --2- 172.21.15.55:0/2530966945 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5bc40fbaf0 0x7f5bc40fbf10 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f5bb400b5f0 tx=0x7f5bb4006ee0).stop 2023-11-15T01:48:49.169 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:49.163+0000 7f5bc8e18700 1 -- 172.21.15.55:0/2530966945 shutdown_connections 2023-11-15T01:48:49.169 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:49.163+0000 7f5bc8e18700 1 --2- 172.21.15.55:0/2530966945 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5bc40fbaf0 0x7f5bc40fbf10 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:48:49.169 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:49.163+0000 7f5bc8e18700 1 -- 172.21.15.55:0/2530966945 >> 172.21.15.55:0/2530966945 conn(0x7f5bc40f7350 msgr2=0x7f5bc40f9770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:48:49.169 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:49.163+0000 7f5bc8e18700 1 -- 172.21.15.55:0/2530966945 shutdown_connections 2023-11-15T01:48:49.169 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:49.163+0000 7f5bc8e18700 1 -- 172.21.15.55:0/2530966945 wait complete. 2023-11-15T01:48:49.170 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:49.164+0000 7f5bc8e18700 1 Processor -- start 2023-11-15T01:48:49.170 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:49.164+0000 7f5bc8e18700 1 -- start start 2023-11-15T01:48:49.170 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:49.165+0000 7f5bc8e18700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5bc40fbaf0 0x7f5bc40725e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:48:49.170 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:49.165+0000 7f5bc8e18700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bc4070a60 con 0x7f5bc40fbaf0 2023-11-15T01:48:49.171 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:49.165+0000 7f5bc37fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5bc40fbaf0 0x7f5bc40725e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:48:49.171 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:49.165+0000 7f5bc37fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5bc40fbaf0 0x7f5bc40725e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:43416/0 (socket says 172.21.15.55:43416) 2023-11-15T01:48:49.171 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:49.165+0000 7f5bc37fe700 1 -- 172.21.15.55:0/1304862842 learned_addr learned my addr 172.21.15.55:0/1304862842 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-11-15T01:48:49.171 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:49.165+0000 7f5bc37fe700 1 -- 172.21.15.55:0/1304862842 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bb400b200 con 0x7f5bc40fbaf0 2023-11-15T01:48:49.171 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:49.165+0000 7f5bc37fe700 1 --2- 172.21.15.55:0/1304862842 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5bc40fbaf0 0x7f5bc40725e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f5bb400e040 tx=0x7f5bb4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:48:49.172 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:49.165+0000 7f5babfff700 1 -- 172.21.15.55:0/1304862842 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5bb401b410 con 0x7f5bc40fbaf0 2023-11-15T01:48:49.172 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:49.165+0000 7f5babfff700 1 -- 172.21.15.55:0/1304862842 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f5bb401b9e0 con 0x7f5bc40fbaf0 2023-11-15T01:48:49.172 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:49.165+0000 7f5babfff700 1 -- 172.21.15.55:0/1304862842 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5bb401a770 con 0x7f5bc40fbaf0 2023-11-15T01:48:49.172 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:49.165+0000 7f5bc8e18700 1 -- 172.21.15.55:0/1304862842 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bc4070c00 con 0x7f5bc40fbaf0 2023-11-15T01:48:49.173 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:49.165+0000 7f5bc8e18700 1 -- 172.21.15.55:0/1304862842 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bc4071010 con 0x7f5bc40fbaf0 2023-11-15T01:48:49.173 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:49.166+0000 7f5babfff700 1 -- 172.21.15.55:0/1304862842 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f5bb400f070 con 0x7f5bc40fbaf0 2023-11-15T01:48:49.173 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:49.166+0000 7f5babfff700 1 -- 172.21.15.55:0/1304862842 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f5bb4024c00 con 0x7f5bc40fbaf0 2023-11-15T01:48:49.173 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:49.167+0000 7f5bc8e18700 1 -- 172.21.15.55:0/1304862842 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bc4045ad0 con 0x7f5bc40fbaf0 2023-11-15T01:48:49.174 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:49.169+0000 7f5babfff700 1 -- 172.21.15.55:0/1304862842 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f5bb4016070 con 0x7f5bc40fbaf0 2023-11-15T01:48:49.240 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:49.239+0000 7f5bc8e18700 1 -- 172.21.15.55:0/1304862842 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f5bc4104740 con 0x7f5bc40fbaf0 2023-11-15T01:48:49.240 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:49.240+0000 7f5babfff700 1 -- 172.21.15.55:0/1304862842 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7f5bb4024e40 con 0x7f5bc40fbaf0 2023-11-15T01:48:49.242 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:49.240+0000 7f5bc8e18700 1 -- 172.21.15.55:0/1304862842 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5bc40fbaf0 msgr2=0x7f5bc40725e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:48:49.242 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:49.240+0000 7f5bc8e18700 1 --2- 172.21.15.55:0/1304862842 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5bc40fbaf0 0x7f5bc40725e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f5bb400e040 tx=0x7f5bb4006ee0).stop 2023-11-15T01:48:49.242 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:49.240+0000 7f5bc8e18700 1 -- 172.21.15.55:0/1304862842 shutdown_connections 2023-11-15T01:48:49.243 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:49.240+0000 7f5bc8e18700 1 --2- 172.21.15.55:0/1304862842 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5bc40fbaf0 0x7f5bc40725e0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:48:49.243 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:49.240+0000 7f5bc8e18700 1 -- 172.21.15.55:0/1304862842 >> 172.21.15.55:0/1304862842 conn(0x7f5bc40f7350 msgr2=0x7f5bc40f8e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:48:49.243 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:49.241+0000 7f5bc8e18700 1 -- 172.21.15.55:0/1304862842 shutdown_connections 2023-11-15T01:48:49.243 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:49.241+0000 7f5bc8e18700 1 -- 172.21.15.55:0/1304862842 wait complete. 2023-11-15T01:48:49.266 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:49 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141164]: audit 2023-11-15T01:48:48.346475+0000 mon.smithi055 (mon.0) 8 : audit [INF] from='client.? 172.21.15.55:0/3137190208' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-11-15T01:48:49.267 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:49 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141164]: audit 2023-11-15T01:48:48.351928+0000 mon.smithi055 (mon.0) 9 : audit [INF] from='client.? 172.21.15.55:0/3137190208' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2023-11-15T01:48:49.599 INFO:teuthology.orchestra.run.smithi055.stderr:Restarting the monitor... 2023-11-15T01:48:49.599 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: systemctl restart ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7@mon.smithi055 2023-11-15T01:48:49.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:49 smithi055 systemd[1]: Stopping Ceph mon.smithi055 for 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7... 2023-11-15T01:48:49.829 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:49 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141164]: debug 2023-11-15T01:48:49.697+0000 7f91cdc92700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi055 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2023-11-15T01:48:49.829 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:49 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141164]: debug 2023-11-15T01:48:49.697+0000 7f91cdc92700 -1 mon.smithi055@0(leader) e1 *** Got Signal Terminated *** 2023-11-15T01:48:50.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 podman[141488]: ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055 2023-11-15T01:48:50.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 systemd[1]: ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7@mon.smithi055.service: Succeeded. 2023-11-15T01:48:50.329 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 systemd[1]: Stopped Ceph mon.smithi055 for 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7. 2023-11-15T01:48:50.329 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 systemd[1]: Starting Ceph mon.smithi055 for 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7... 2023-11-15T01:48:50.425 INFO:teuthology.orchestra.run.smithi055.stderr:Setting mon public_network to 172.21.0.0/20 2023-11-15T01:48:50.425 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi055 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/0cb4fc0c-8359-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpc1yi9jse:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpra3bp1py:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mon public_network 172.21.0.0/20 2023-11-15T01:48:50.819 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:50.816+0000 7f29cd8e8700 1 Processor -- start 2023-11-15T01:48:50.819 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:50.816+0000 7f29cd8e8700 1 -- start start 2023-11-15T01:48:50.819 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:50.816+0000 7f29cd8e8700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f29c8072540 0x7f29c8070ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T01:48:50.820 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:50.816+0000 7f29cd8e8700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29c8072960 con 0x7f29c8072540 2023-11-15T01:48:50.820 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:50.817+0000 7f29c6ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f29c8072540 0x7f29c8070ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:48:50.820 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:50.817+0000 7f29c6ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f29c8072540 0x7f29c8070ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:33846/0 (socket says 172.21.15.55:33846) 2023-11-15T01:48:50.820 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:50.817+0000 7f29c6ffd700 1 -- 172.21.15.55:0/2369356815 learned_addr learned my addr 172.21.15.55:0/2369356815 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-11-15T01:48:50.820 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:50.817+0000 7f29c6ffd700 1 -- 172.21.15.55:0/2369356815 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29c8070fd0 con 0x7f29c8072540 2023-11-15T01:48:50.820 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:50.817+0000 7f29c6ffd700 1 --2- 172.21.15.55:0/2369356815 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f29c8072540 0x7f29c8070ae0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f29b000b5f0 tx=0x7f29b0006ee0).ready entity=mon.0 client_cookie=d13c0d193612fb56 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:48:50.821 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:50.817+0000 7f29c5ffb700 1 -- 172.21.15.55:0/2369356815 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f29b000d600 con 0x7f29c8072540 2023-11-15T01:48:50.821 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:50.818+0000 7f29c5ffb700 1 -- 172.21.15.55:0/2369356815 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f29b000dbd0 con 0x7f29c8072540 2023-11-15T01:48:50.821 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:50.818+0000 7f29c5ffb700 1 -- 172.21.15.55:0/2369356815 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f29b0011c00 con 0x7f29c8072540 2023-11-15T01:48:50.821 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:50.818+0000 7f29cd8e8700 1 -- 172.21.15.55:0/2369356815 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f29c8072540 msgr2=0x7f29c8070ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:48:50.824 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:50.818+0000 7f29cd8e8700 1 --2- 172.21.15.55:0/2369356815 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f29c8072540 0x7f29c8070ae0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f29b000b5f0 tx=0x7f29b0006ee0).stop 2023-11-15T01:48:50.824 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:50.818+0000 7f29cd8e8700 1 -- 172.21.15.55:0/2369356815 shutdown_connections 2023-11-15T01:48:50.824 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:50.818+0000 7f29cd8e8700 1 --2- 172.21.15.55:0/2369356815 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f29c8072540 0x7f29c8070ae0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:48:50.824 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:50.818+0000 7f29cd8e8700 1 -- 172.21.15.55:0/2369356815 >> 172.21.15.55:0/2369356815 conn(0x7f29c80f74b0 msgr2=0x7f29c80f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:48:50.824 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:50.818+0000 7f29cd8e8700 1 -- 172.21.15.55:0/2369356815 shutdown_connections 2023-11-15T01:48:50.825 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:50.818+0000 7f29cd8e8700 1 -- 172.21.15.55:0/2369356815 wait complete. 2023-11-15T01:48:50.825 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:50.819+0000 7f29cd8e8700 1 Processor -- start 2023-11-15T01:48:50.825 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:50.819+0000 7f29cd8e8700 1 -- start start 2023-11-15T01:48:50.825 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:50.819+0000 7f29cd8e8700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f29c8072540 0x7f29c810ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:48:50.825 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:50.819+0000 7f29cd8e8700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29c810d1e0 con 0x7f29c8072540 2023-11-15T01:48:50.826 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:50.819+0000 7f29c6ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f29c8072540 0x7f29c810ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:48:50.826 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:50.819+0000 7f29c6ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f29c8072540 0x7f29c810ccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:33856/0 (socket says 172.21.15.55:33856) 2023-11-15T01:48:50.826 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:50.819+0000 7f29c6ffd700 1 -- 172.21.15.55:0/2323281156 learned_addr learned my addr 172.21.15.55:0/2323281156 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-11-15T01:48:50.826 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:50.820+0000 7f29c6ffd700 1 -- 172.21.15.55:0/2323281156 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29b000b200 con 0x7f29c8072540 2023-11-15T01:48:50.826 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:50.820+0000 7f29c6ffd700 1 --2- 172.21.15.55:0/2323281156 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f29c8072540 0x7f29c810ccf0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f29b0006c70 tx=0x7f29b0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:48:50.827 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:50.820+0000 7f29bf7fe700 1 -- 172.21.15.55:0/2323281156 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f29b000da30 con 0x7f29c8072540 2023-11-15T01:48:50.827 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:50.820+0000 7f29bf7fe700 1 -- 172.21.15.55:0/2323281156 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f29b001a9f0 con 0x7f29c8072540 2023-11-15T01:48:50.827 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:50.820+0000 7f29cd8e8700 1 -- 172.21.15.55:0/2323281156 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29c810d3e0 con 0x7f29c8072540 2023-11-15T01:48:50.827 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:50.820+0000 7f29bf7fe700 1 -- 172.21.15.55:0/2323281156 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f29b001b890 con 0x7f29c8072540 2023-11-15T01:48:50.827 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:50.820+0000 7f29cd8e8700 1 -- 172.21.15.55:0/2323281156 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29c810d7f0 con 0x7f29c8072540 2023-11-15T01:48:50.828 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:50.821+0000 7f29bf7fe700 1 -- 172.21.15.55:0/2323281156 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f29b000f070 con 0x7f29c8072540 2023-11-15T01:48:50.828 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:50.821+0000 7f29bf7fe700 1 -- 172.21.15.55:0/2323281156 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f29b0025af0 con 0x7f29c8072540 2023-11-15T01:48:50.828 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:50.821+0000 7f29cd8e8700 1 -- 172.21.15.55:0/2323281156 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29c8045ad0 con 0x7f29c8072540 2023-11-15T01:48:50.828 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:50.824+0000 7f29bf7fe700 1 -- 172.21.15.55:0/2323281156 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f29b0016070 con 0x7f29c8072540 2023-11-15T01:48:50.830 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 bash[141560]: 24091846f7802d72711ab3cd08f4f8bcb0adf2d58bc63d12a5d94066350f4fb4 2023-11-15T01:48:50.830 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.420+0000 7fe89944a700 0 set uid:gid to 167:167 (ceph:ceph) 2023-11-15T01:48:50.830 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.420+0000 7fe89944a700 0 ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable), process ceph-mon, pid 7 2023-11-15T01:48:50.831 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.420+0000 7fe89944a700 0 pidfile_write: ignore empty --pid-file 2023-11-15T01:48:50.831 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 systemd[1]: Started Ceph mon.smithi055 for 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7. 2023-11-15T01:48:50.831 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.426+0000 7fe89944a700 0 load: jerasure load: lrc load: isa 2023-11-15T01:48:50.831 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: RocksDB version: 6.8.1 2023-11-15T01:48:50.831 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: 2023-11-15T01:48:50.832 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2023-11-15T01:48:50.832 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Compile date May 13 2021 2023-11-15T01:48:50.832 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: DB SUMMARY 2023-11-15T01:48:50.832 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: 2023-11-15T01:48:50.832 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: CURRENT file: CURRENT 2023-11-15T01:48:50.833 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: 2023-11-15T01:48:50.833 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: IDENTITY file: IDENTITY 2023-11-15T01:48:50.833 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: 2023-11-15T01:48:50.833 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 131 Bytes 2023-11-15T01:48:50.833 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: 2023-11-15T01:48:50.833 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi055/store.db dir, Total Num: 1, files: 000004.sst 2023-11-15T01:48:50.834 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: 2023-11-15T01:48:50.834 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi055/store.db: 000006.log size: 92918 ; 2023-11-15T01:48:50.834 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: 2023-11-15T01:48:50.834 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.error_if_exists: 0 2023-11-15T01:48:50.834 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.create_if_missing: 0 2023-11-15T01:48:50.835 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.paranoid_checks: 1 2023-11-15T01:48:50.835 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.env: 0x5638c454b1c0 2023-11-15T01:48:50.835 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.fs: Posix File System 2023-11-15T01:48:50.835 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.info_log: 0x5638c5289960 2023-11-15T01:48:50.835 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.max_file_opening_threads: 16 2023-11-15T01:48:50.836 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.statistics: (nil) 2023-11-15T01:48:50.836 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.use_fsync: 0 2023-11-15T01:48:50.836 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.max_log_file_size: 0 2023-11-15T01:48:50.836 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2023-11-15T01:48:50.836 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.log_file_time_to_roll: 0 2023-11-15T01:48:50.836 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.keep_log_file_num: 1000 2023-11-15T01:48:50.837 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.recycle_log_file_num: 0 2023-11-15T01:48:50.837 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.allow_fallocate: 1 2023-11-15T01:48:50.837 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.allow_mmap_reads: 0 2023-11-15T01:48:50.837 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.allow_mmap_writes: 0 2023-11-15T01:48:50.837 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.use_direct_reads: 0 2023-11-15T01:48:50.838 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-11-15T01:48:50.838 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.create_missing_column_families: 0 2023-11-15T01:48:50.838 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.db_log_dir: 2023-11-15T01:48:50.838 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi055/store.db 2023-11-15T01:48:50.838 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.table_cache_numshardbits: 6 2023-11-15T01:48:50.838 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.max_subcompactions: 1 2023-11-15T01:48:50.839 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.max_background_flushes: -1 2023-11-15T01:48:50.839 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.WAL_ttl_seconds: 0 2023-11-15T01:48:50.839 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.WAL_size_limit_MB: 0 2023-11-15T01:48:50.839 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-11-15T01:48:50.839 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2023-11-15T01:48:50.840 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.is_fd_close_on_exec: 1 2023-11-15T01:48:50.840 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.advise_random_on_open: 1 2023-11-15T01:48:50.840 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.db_write_buffer_size: 0 2023-11-15T01:48:50.840 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.write_buffer_manager: 0x5638c5efe030 2023-11-15T01:48:50.840 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2023-11-15T01:48:50.841 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2023-11-15T01:48:50.841 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2023-11-15T01:48:50.841 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.use_adaptive_mutex: 0 2023-11-15T01:48:50.841 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.rate_limiter: (nil) 2023-11-15T01:48:50.841 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-11-15T01:48:50.841 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.wal_recovery_mode: 2 2023-11-15T01:48:50.842 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.enable_thread_tracking: 0 2023-11-15T01:48:50.842 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.enable_pipelined_write: 0 2023-11-15T01:48:50.842 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.unordered_write: 0 2023-11-15T01:48:50.842 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2023-11-15T01:48:50.842 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-11-15T01:48:50.843 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2023-11-15T01:48:50.843 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2023-11-15T01:48:50.843 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.row_cache: None 2023-11-15T01:48:50.843 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.wal_filter: None 2023-11-15T01:48:50.843 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2023-11-15T01:48:50.843 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.allow_ingest_behind: 0 2023-11-15T01:48:50.844 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.preserve_deletes: 0 2023-11-15T01:48:50.844 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.two_write_queues: 0 2023-11-15T01:48:50.844 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.manual_wal_flush: 0 2023-11-15T01:48:50.844 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.atomic_flush: 0 2023-11-15T01:48:50.844 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-11-15T01:48:50.845 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.persist_stats_to_disk: 0 2023-11-15T01:48:50.845 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.write_dbid_to_manifest: 0 2023-11-15T01:48:50.845 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.log_readahead_size: 0 2023-11-15T01:48:50.845 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2023-11-15T01:48:50.845 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.max_background_jobs: 2 2023-11-15T01:48:50.845 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.max_background_compactions: -1 2023-11-15T01:48:50.846 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2023-11-15T01:48:50.846 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-11-15T01:48:50.846 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.delayed_write_rate : 16777216 2023-11-15T01:48:50.846 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.max_total_wal_size: 0 2023-11-15T01:48:50.846 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-11-15T01:48:50.847 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.stats_dump_period_sec: 600 2023-11-15T01:48:50.847 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.stats_persist_period_sec: 600 2023-11-15T01:48:50.847 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2023-11-15T01:48:50.847 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.max_open_files: -1 2023-11-15T01:48:50.847 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.bytes_per_sync: 0 2023-11-15T01:48:50.848 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.wal_bytes_per_sync: 0 2023-11-15T01:48:50.848 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.strict_bytes_per_sync: 0 2023-11-15T01:48:50.848 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.compaction_readahead_size: 0 2023-11-15T01:48:50.848 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Compression algorithms supported: 2023-11-15T01:48:50.848 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2023-11-15T01:48:50.848 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: kZSTD supported: 0 2023-11-15T01:48:50.849 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: kXpressCompression supported: 0 2023-11-15T01:48:50.849 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: kLZ4HCCompression supported: 1 2023-11-15T01:48:50.849 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: kLZ4Compression supported: 1 2023-11-15T01:48:50.849 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: kBZip2Compression supported: 0 2023-11-15T01:48:50.849 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: kZlibCompression supported: 1 2023-11-15T01:48:50.850 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: kSnappyCompression supported: 1 2023-11-15T01:48:50.850 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Fast CRC32 supported: Supported on x86 2023-11-15T01:48:50.850 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi055/store.db/MANIFEST-000005 2023-11-15T01:48:50.850 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: 2023-11-15T01:48:50.850 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2023-11-15T01:48:50.853 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: 2023-11-15T01:48:50.853 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-11-15T01:48:50.854 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.merge_operator: 2023-11-15T01:48:50.854 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.compaction_filter: None 2023-11-15T01:48:50.854 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.compaction_filter_factory: None 2023-11-15T01:48:50.854 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.memtable_factory: SkipListFactory 2023-11-15T01:48:50.854 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.table_factory: BlockBasedTable 2023-11-15T01:48:50.855 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5638c51a0128) 2023-11-15T01:48:50.855 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cache_index_and_filter_blocks: 1 2023-11-15T01:48:50.855 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cache_index_and_filter_blocks_with_high_priority: 0 2023-11-15T01:48:50.855 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: pin_l0_filter_and_index_blocks_in_cache: 0 2023-11-15T01:48:50.855 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: pin_top_level_index_and_filter: 1 2023-11-15T01:48:50.856 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: index_type: 0 2023-11-15T01:48:50.856 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: data_block_index_type: 0 2023-11-15T01:48:50.856 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: index_shortening: 1 2023-11-15T01:48:50.856 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: data_block_hash_table_util_ratio: 0.750000 2023-11-15T01:48:50.856 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: hash_index_allow_collision: 1 2023-11-15T01:48:50.856 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: checksum: 1 2023-11-15T01:48:50.857 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: no_block_cache: 0 2023-11-15T01:48:50.857 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: block_cache: 0x5638c51d6f10 2023-11-15T01:48:50.857 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: block_cache_name: BinnedLRUCache 2023-11-15T01:48:50.857 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: block_cache_options: 2023-11-15T01:48:50.857 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: capacity : 536870912 2023-11-15T01:48:50.858 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: num_shard_bits : 4 2023-11-15T01:48:50.858 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: strict_capacity_limit : 0 2023-11-15T01:48:50.858 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: high_pri_pool_ratio: 0.000 2023-11-15T01:48:50.858 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: block_cache_compressed: (nil) 2023-11-15T01:48:50.858 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: persistent_cache: (nil) 2023-11-15T01:48:50.859 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: block_size: 4096 2023-11-15T01:48:50.859 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: block_size_deviation: 10 2023-11-15T01:48:50.859 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: block_restart_interval: 16 2023-11-15T01:48:50.859 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: index_block_restart_interval: 1 2023-11-15T01:48:50.859 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: metadata_block_size: 4096 2023-11-15T01:48:50.859 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: partition_filters: 0 2023-11-15T01:48:50.860 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: use_delta_encoding: 1 2023-11-15T01:48:50.860 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: filter_policy: rocksdb.BuiltinBloomFilter 2023-11-15T01:48:50.860 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: whole_key_filtering: 1 2023-11-15T01:48:50.860 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: verify_compression: 0 2023-11-15T01:48:50.860 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: read_amp_bytes_per_bit: 0 2023-11-15T01:48:50.861 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: format_version: 2 2023-11-15T01:48:50.861 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: enable_index_compression: 1 2023-11-15T01:48:50.861 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: block_align: 0 2023-11-15T01:48:50.861 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: 2023-11-15T01:48:50.861 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.write_buffer_size: 33554432 2023-11-15T01:48:50.861 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.max_write_buffer_number: 2 2023-11-15T01:48:50.862 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.compression: NoCompression 2023-11-15T01:48:50.862 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.bottommost_compression: Disabled 2023-11-15T01:48:50.862 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.prefix_extractor: nullptr 2023-11-15T01:48:50.862 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-11-15T01:48:50.862 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.num_levels: 7 2023-11-15T01:48:50.863 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-11-15T01:48:50.863 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-11-15T01:48:50.863 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-11-15T01:48:50.863 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-11-15T01:48:50.863 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2023-11-15T01:48:50.864 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-11-15T01:48:50.864 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-11-15T01:48:50.864 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-11-15T01:48:50.864 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2023-11-15T01:48:50.864 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.compression_opts.window_bits: -14 2023-11-15T01:48:50.864 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.compression_opts.level: 32767 2023-11-15T01:48:50.865 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.compression_opts.strategy: 0 2023-11-15T01:48:50.865 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-11-15T01:48:50.865 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-11-15T01:48:50.865 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.compression_opts.enabled: false 2023-11-15T01:48:50.865 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-11-15T01:48:50.866 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-11-15T01:48:50.866 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2023-11-15T01:48:50.866 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.target_file_size_base: 67108864 2023-11-15T01:48:50.866 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.target_file_size_multiplier: 1 2023-11-15T01:48:50.866 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2023-11-15T01:48:50.867 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-11-15T01:48:50.867 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-11-15T01:48:50.867 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-11-15T01:48:50.867 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-11-15T01:48:50.867 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-11-15T01:48:50.867 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-11-15T01:48:50.868 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-11-15T01:48:50.868 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-11-15T01:48:50.868 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-11-15T01:48:50.868 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-11-15T01:48:50.868 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2023-11-15T01:48:50.869 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.arena_block_size: 4194304 2023-11-15T01:48:50.869 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-11-15T01:48:50.869 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-11-15T01:48:50.869 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2023-11-15T01:48:50.869 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.disable_auto_compactions: 0 2023-11-15T01:48:50.869 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-11-15T01:48:50.870 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-11-15T01:48:50.870 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-11-15T01:48:50.870 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-11-15T01:48:50.870 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-11-15T01:48:50.870 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-11-15T01:48:50.871 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-11-15T01:48:50.871 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-11-15T01:48:50.871 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-11-15T01:48:50.871 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-11-15T01:48:50.871 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.table_properties_collectors: 2023-11-15T01:48:50.872 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.inplace_update_support: 0 2023-11-15T01:48:50.872 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.inplace_update_num_locks: 10000 2023-11-15T01:48:50.872 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-11-15T01:48:50.872 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2023-11-15T01:48:50.872 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.memtable_huge_page_size: 0 2023-11-15T01:48:50.872 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.bloom_locality: 0 2023-11-15T01:48:50.873 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.max_successive_merges: 0 2023-11-15T01:48:50.873 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.optimize_filters_for_hits: 0 2023-11-15T01:48:50.873 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.paranoid_file_checks: 0 2023-11-15T01:48:50.873 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.force_consistency_checks: 0 2023-11-15T01:48:50.873 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.report_bg_io_stats: 0 2023-11-15T01:48:50.874 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.ttl: 2592000 2023-11-15T01:48:50.874 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.427+0000 7fe89944a700 4 rocksdb: Options.periodic_compaction_seconds: 0 2023-11-15T01:48:50.874 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.429+0000 7fe89944a700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi055/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 5, log_number is 4,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-11-15T01:48:50.874 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: 2023-11-15T01:48:50.874 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.429+0000 7fe89944a700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 4 2023-11-15T01:48:50.875 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: 2023-11-15T01:48:50.875 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.429+0000 7fe89944a700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700012930429900, "job": 1, "event": "recovery_started", "log_files": [6]} 2023-11-15T01:48:50.875 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.429+0000 7fe89944a700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #6 mode 2 2023-11-15T01:48:50.875 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.429+0000 7fe89944a700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2023-11-15T01:48:50.875 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.430+0000 7fe89944a700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700012930431058, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 7, "file_size": 88698, "table_properties": {"data_size": 86813, "index_size": 362, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 709, "raw_key_size": 12384, "raw_average_key_size": 49, "raw_value_size": 79954, "raw_average_value_size": 321, "num_data_blocks": 11, "num_entries": 249, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1700012930, "oldest_key_time": 3, "file_creation_time": 0}} 2023-11-15T01:48:50.876 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.430+0000 7fe89944a700 4 rocksdb: [version_set.cc:3826] Creating manifest 8 2023-11-15T01:48:50.876 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: 2023-11-15T01:48:50.876 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.431+0000 7fe89944a700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700012930431777, "job": 1, "event": "recovery_finished"} 2023-11-15T01:48:50.876 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.432+0000 7fe89944a700 4 rocksdb: DB pointer 0x5638c529b800 2023-11-15T01:48:50.876 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.432+0000 7fe880262700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2023-11-15T01:48:50.876 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.432+0000 7fe880262700 4 rocksdb: [db_impl/db_impl.cc:851] 2023-11-15T01:48:50.877 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: ** DB Stats ** 2023-11-15T01:48:50.877 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: Uptime(secs): 0.0 total, 0.0 interval 2023-11-15T01:48:50.877 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-11-15T01:48:50.877 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-11-15T01:48:50.877 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-11-15T01:48:50.878 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-11-15T01:48:50.878 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2023-11-15T01:48:50.878 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-11-15T01:48:50.878 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: 2023-11-15T01:48:50.878 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: ** Compaction Stats [default] ** 2023-11-15T01:48:50.878 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-15T01:48:50.879 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-15T01:48:50.879 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: L0 2/0 88.31 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 107.1 0.00 0.00 1 0.001 0 0 2023-11-15T01:48:50.879 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: Sum 2/0 88.31 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 107.1 0.00 0.00 1 0.001 0 0 2023-11-15T01:48:50.879 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 107.1 0.00 0.00 1 0.001 0 0 2023-11-15T01:48:50.879 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: 2023-11-15T01:48:50.880 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: ** Compaction Stats [default] ** 2023-11-15T01:48:50.880 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-15T01:48:50.880 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-15T01:48:50.880 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 107.1 0.00 0.00 1 0.001 0 0 2023-11-15T01:48:50.880 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: Uptime(secs): 0.0 total, 0.0 interval 2023-11-15T01:48:50.881 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: Flush(GB): cumulative 0.000, interval 0.000 2023-11-15T01:48:50.881 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: AddFile(GB): cumulative 0.000, interval 0.000 2023-11-15T01:48:50.881 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: AddFile(Total Files): cumulative 0, interval 0 2023-11-15T01:48:50.882 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: AddFile(L0 Files): cumulative 0, interval 0 2023-11-15T01:48:50.882 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: AddFile(Keys): cumulative 0, interval 0 2023-11-15T01:48:50.882 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: Cumulative compaction: 0.00 GB write, 17.81 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-15T01:48:50.882 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: Interval compaction: 0.00 GB write, 17.81 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-15T01:48:50.883 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-11-15T01:48:50.883 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: 2023-11-15T01:48:50.883 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: ** File Read Latency Histogram By Level [default] ** 2023-11-15T01:48:50.883 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: 2023-11-15T01:48:50.884 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: ** Compaction Stats [default] ** 2023-11-15T01:48:50.884 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-15T01:48:50.884 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-15T01:48:50.884 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: L0 2/0 88.31 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 107.1 0.00 0.00 1 0.001 0 0 2023-11-15T01:48:50.884 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: Sum 2/0 88.31 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 107.1 0.00 0.00 1 0.001 0 0 2023-11-15T01:48:50.885 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-11-15T01:48:50.885 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: 2023-11-15T01:48:50.885 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: ** Compaction Stats [default] ** 2023-11-15T01:48:50.885 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-15T01:48:50.885 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-15T01:48:50.885 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 107.1 0.00 0.00 1 0.001 0 0 2023-11-15T01:48:50.886 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: Uptime(secs): 0.0 total, 0.0 interval 2023-11-15T01:48:50.886 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: Flush(GB): cumulative 0.000, interval 0.000 2023-11-15T01:48:50.886 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: AddFile(GB): cumulative 0.000, interval 0.000 2023-11-15T01:48:50.886 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: AddFile(Total Files): cumulative 0, interval 0 2023-11-15T01:48:50.886 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: AddFile(L0 Files): cumulative 0, interval 0 2023-11-15T01:48:50.887 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: AddFile(Keys): cumulative 0, interval 0 2023-11-15T01:48:50.887 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: Cumulative compaction: 0.00 GB write, 17.68 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-15T01:48:50.887 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-15T01:48:50.887 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-11-15T01:48:50.887 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: 2023-11-15T01:48:50.887 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: ** File Read Latency Histogram By Level [default] ** 2023-11-15T01:48:50.888 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: 2023-11-15T01:48:50.888 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.433+0000 7fe89944a700 0 starting mon.smithi055 rank 0 at public addrs [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] at bind addrs [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi055 fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 2023-11-15T01:48:50.888 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.434+0000 7fe89944a700 1 mon.smithi055@-1(???) e1 preinit fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 2023-11-15T01:48:50.888 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.434+0000 7fe89944a700 0 mon.smithi055@-1(???).mds e1 new map 2023-11-15T01:48:50.888 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.434+0000 7fe89944a700 0 mon.smithi055@-1(???).mds e1 print_map 2023-11-15T01:48:50.889 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: e1 2023-11-15T01:48:50.889 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: enable_multiple, ever_enabled_multiple: 1,1 2023-11-15T01:48:50.889 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-11-15T01:48:50.889 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: legacy client fscid: -1 2023-11-15T01:48:50.889 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: 2023-11-15T01:48:50.890 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: No filesystems configured 2023-11-15T01:48:50.890 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: 2023-11-15T01:48:50.890 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.434+0000 7fe89944a700 0 mon.smithi055@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2023-11-15T01:48:50.890 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.434+0000 7fe89944a700 0 mon.smithi055@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-11-15T01:48:50.890 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.434+0000 7fe89944a700 0 mon.smithi055@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-11-15T01:48:50.890 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.434+0000 7fe89944a700 0 mon.smithi055@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-11-15T01:48:50.891 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:48:48.346475+0000 mon.smithi055 (mon.0) 8 : audit [INF] from='client.? 172.21.15.55:0/3137190208' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-11-15T01:48:50.891 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:48:48.351928+0000 mon.smithi055 (mon.0) 9 : audit [INF] from='client.? 172.21.15.55:0/3137190208' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2023-11-15T01:48:50.891 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.434+0000 7fe89944a700 1 mon.smithi055@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2023-11-15T01:48:50.891 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.436+0000 7fe89944a700 20 expand_channel_meta expand map: {default=false} 2023-11-15T01:48:50.891 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.436+0000 7fe89944a700 20 expand_channel_meta from 'false' to 'false' 2023-11-15T01:48:50.892 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.436+0000 7fe89944a700 20 expand_channel_meta expanded map: {default=false} 2023-11-15T01:48:50.892 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.436+0000 7fe89944a700 20 expand_channel_meta expand map: {default=info} 2023-11-15T01:48:50.892 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.436+0000 7fe89944a700 20 expand_channel_meta from 'info' to 'info' 2023-11-15T01:48:50.892 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.436+0000 7fe89944a700 20 expand_channel_meta expanded map: {default=info} 2023-11-15T01:48:50.892 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.436+0000 7fe89944a700 20 expand_channel_meta expand map: {default=daemon} 2023-11-15T01:48:50.892 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.436+0000 7fe89944a700 20 expand_channel_meta from 'daemon' to 'daemon' 2023-11-15T01:48:50.893 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.436+0000 7fe89944a700 20 expand_channel_meta expanded map: {default=daemon} 2023-11-15T01:48:50.893 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.436+0000 7fe89944a700 20 expand_channel_meta expand map: {default=debug} 2023-11-15T01:48:50.893 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.436+0000 7fe89944a700 20 expand_channel_meta from 'debug' to 'debug' 2023-11-15T01:48:50.893 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:48:50.436+0000 7fe89944a700 20 expand_channel_meta expanded map: {default=debug} 2023-11-15T01:48:50.893 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:48:50.451505+0000 mon.smithi055 (mon.0) 1 : cluster [INF] mon.smithi055 is new leader, mons smithi055 in quorum (ranks 0) 2023-11-15T01:48:50.894 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:48:50.451607+0000 mon.smithi055 (mon.0) 2 : cluster [DBG] monmap e1: 1 mons at {smithi055=[v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0]} 2023-11-15T01:48:50.894 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:48:50.451671+0000 mon.smithi055 (mon.0) 3 : cluster [DBG] fsmap 2023-11-15T01:48:50.894 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:48:50.451737+0000 mon.smithi055 (mon.0) 4 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2023-11-15T01:48:50.894 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:48:50.452373+0000 mon.smithi055 (mon.0) 5 : cluster [DBG] mgrmap e1: no daemons active 2023-11-15T01:48:50.902 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:50.901+0000 7f29cd8e8700 1 -- 172.21.15.55:0/2323281156 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f29c810e6f0 con 0x7f29c8072540 2023-11-15T01:48:50.906 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:50.906+0000 7f29bf7fe700 1 -- 172.21.15.55:0/2323281156 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f29b001bd90 con 0x7f29c8072540 2023-11-15T01:48:50.907 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:50.906+0000 7f29cd8e8700 1 -- 172.21.15.55:0/2323281156 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f29c8072540 msgr2=0x7f29c810ccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:48:50.907 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:50.906+0000 7f29cd8e8700 1 --2- 172.21.15.55:0/2323281156 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f29c8072540 0x7f29c810ccf0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f29b0006c70 tx=0x7f29b0006ee0).stop 2023-11-15T01:48:50.908 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:50.907+0000 7f29cd8e8700 1 -- 172.21.15.55:0/2323281156 shutdown_connections 2023-11-15T01:48:50.908 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:50.907+0000 7f29cd8e8700 1 --2- 172.21.15.55:0/2323281156 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f29c8072540 0x7f29c810ccf0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:48:50.909 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:50.907+0000 7f29cd8e8700 1 -- 172.21.15.55:0/2323281156 >> 172.21.15.55:0/2323281156 conn(0x7f29c80f74b0 msgr2=0x7f29c80f8fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:48:50.909 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:50.907+0000 7f29cd8e8700 1 -- 172.21.15.55:0/2323281156 shutdown_connections 2023-11-15T01:48:50.909 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:50.907+0000 7f29cd8e8700 1 -- 172.21.15.55:0/2323281156 wait complete. 2023-11-15T01:48:51.256 INFO:teuthology.orchestra.run.smithi055.stderr:Wrote config to /etc/ceph/ceph.conf 2023-11-15T01:48:51.257 INFO:teuthology.orchestra.run.smithi055.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2023-11-15T01:48:51.257 INFO:teuthology.orchestra.run.smithi055.stderr:Creating mgr... 2023-11-15T01:48:51.257 INFO:teuthology.orchestra.run.smithi055.stderr:Verifying port 9283 ... 2023-11-15T01:48:51.260 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: systemctl daemon-reload 2023-11-15T01:48:51.373 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: systemctl stop ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7@mgr.smithi055.szwngw 2023-11-15T01:48:51.401 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: systemctl reset-failed ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7@mgr.smithi055.szwngw 2023-11-15T01:48:51.407 INFO:teuthology.orchestra.run.smithi055.stderr:systemctl: Failed to reset failed state of unit ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7@mgr.smithi055.szwngw.service: Unit ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7@mgr.smithi055.szwngw.service not loaded. 2023-11-15T01:48:51.408 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: systemctl enable ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7@mgr.smithi055.szwngw 2023-11-15T01:48:51.414 INFO:teuthology.orchestra.run.smithi055.stderr:systemctl: Created symlink /etc/systemd/system/ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7.target.wants/ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7@mgr.smithi055.szwngw.service → /etc/systemd/system/ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7@.service. 2023-11-15T01:48:51.508 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: systemctl start ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7@mgr.smithi055.szwngw 2023-11-15T01:48:51.851 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: systemctl is-enabled firewalld.service 2023-11-15T01:48:51.860 INFO:teuthology.orchestra.run.smithi055.stderr:systemctl: disabled 2023-11-15T01:48:51.861 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: systemctl is-active firewalld.service 2023-11-15T01:48:51.869 INFO:teuthology.orchestra.run.smithi055.stderr:systemctl: inactive 2023-11-15T01:48:51.870 INFO:teuthology.orchestra.run.smithi055.stderr:firewalld.service is not enabled 2023-11-15T01:48:51.871 INFO:teuthology.orchestra.run.smithi055.stderr:Not possible to enable service . firewalld.service is not available 2023-11-15T01:48:51.871 INFO:teuthology.orchestra.run.smithi055.stderr:Not possible to open ports <[]>. firewalld.service is not available 2023-11-15T01:48:51.871 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: systemctl is-enabled firewalld.service 2023-11-15T01:48:51.881 INFO:teuthology.orchestra.run.smithi055.stderr:systemctl: disabled 2023-11-15T01:48:51.882 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: systemctl is-active firewalld.service 2023-11-15T01:48:51.889 INFO:teuthology.orchestra.run.smithi055.stderr:systemctl: inactive 2023-11-15T01:48:51.890 INFO:teuthology.orchestra.run.smithi055.stderr:firewalld.service is not enabled 2023-11-15T01:48:51.890 INFO:teuthology.orchestra.run.smithi055.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2023-11-15T01:48:51.891 INFO:teuthology.orchestra.run.smithi055.stderr:Waiting for mgr to start... 2023-11-15T01:48:51.891 INFO:teuthology.orchestra.run.smithi055.stderr:Waiting for mgr... 2023-11-15T01:48:51.891 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi055 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/0cb4fc0c-8359-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpc1yi9jse:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpra3bp1py:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2023-11-15T01:48:52.295 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:52.291+0000 7f7bbf59e700 1 Processor -- start 2023-11-15T01:48:52.295 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:52.291+0000 7f7bbf59e700 1 -- start start 2023-11-15T01:48:52.295 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:52.291+0000 7f7bbf59e700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7bc00fd100 0x7f7bc00fd520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T01:48:52.295 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:52.291+0000 7f7bbf59e700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bc00fda10 con 0x7f7bc00fd100 2023-11-15T01:48:52.296 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:52.292+0000 7f7bbe59c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7bc00fd100 0x7f7bc00fd520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:48:52.296 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:52.292+0000 7f7bbe59c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7bc00fd100 0x7f7bc00fd520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:33876/0 (socket says 172.21.15.55:33876) 2023-11-15T01:48:52.296 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:52.292+0000 7f7bbe59c700 1 -- 172.21.15.55:0/3508436166 learned_addr learned my addr 172.21.15.55:0/3508436166 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-11-15T01:48:52.296 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:52.292+0000 7f7bbe59c700 1 -- 172.21.15.55:0/3508436166 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bc00fdb50 con 0x7f7bc00fd100 2023-11-15T01:48:52.296 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:52.292+0000 7f7bbe59c700 1 --2- 172.21.15.55:0/3508436166 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7bc00fd100 0x7f7bc00fd520 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f7bb000b5f0 tx=0x7f7bb0006ee0).ready entity=mon.0 client_cookie=d8d22c15b9536f41 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:48:52.297 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:52.292+0000 7f7bbd59a700 1 -- 172.21.15.55:0/3508436166 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7bb000d600 con 0x7f7bc00fd100 2023-11-15T01:48:52.297 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:52.292+0000 7f7bbd59a700 1 -- 172.21.15.55:0/3508436166 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f7bb000dbd0 con 0x7f7bc00fd100 2023-11-15T01:48:52.297 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:52.292+0000 7f7bbd59a700 1 -- 172.21.15.55:0/3508436166 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7bb0011c00 con 0x7f7bc00fd100 2023-11-15T01:48:52.297 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:52.293+0000 7f7bbf59e700 1 -- 172.21.15.55:0/3508436166 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7bc00fd100 msgr2=0x7f7bc00fd520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:48:52.300 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:52.293+0000 7f7bbf59e700 1 --2- 172.21.15.55:0/3508436166 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7bc00fd100 0x7f7bc00fd520 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f7bb000b5f0 tx=0x7f7bb0006ee0).stop 2023-11-15T01:48:52.300 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:52.293+0000 7f7bbf59e700 1 -- 172.21.15.55:0/3508436166 shutdown_connections 2023-11-15T01:48:52.300 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:52.293+0000 7f7bbf59e700 1 --2- 172.21.15.55:0/3508436166 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7bc00fd100 0x7f7bc00fd520 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:48:52.300 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:52.293+0000 7f7bbf59e700 1 -- 172.21.15.55:0/3508436166 >> 172.21.15.55:0/3508436166 conn(0x7f7bc00f59a0 msgr2=0x7f7bc00f7dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:48:52.300 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:52.293+0000 7f7bbf59e700 1 -- 172.21.15.55:0/3508436166 shutdown_connections 2023-11-15T01:48:52.301 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:52.293+0000 7f7bbf59e700 1 -- 172.21.15.55:0/3508436166 wait complete. 2023-11-15T01:48:52.301 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:52.294+0000 7f7bbf59e700 1 Processor -- start 2023-11-15T01:48:52.301 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:52.294+0000 7f7bbf59e700 1 -- start start 2023-11-15T01:48:52.301 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:52.294+0000 7f7bbf59e700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7bc00fd100 0x7f7bc00fc9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:48:52.301 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:52.294+0000 7f7bbf59e700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bc00fae50 con 0x7f7bc00fd100 2023-11-15T01:48:52.302 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:52.294+0000 7f7bbe59c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7bc00fd100 0x7f7bc00fc9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:48:52.302 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:52.294+0000 7f7bbe59c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7bc00fd100 0x7f7bc00fc9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:33880/0 (socket says 172.21.15.55:33880) 2023-11-15T01:48:52.302 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:52.294+0000 7f7bbe59c700 1 -- 172.21.15.55:0/426191817 learned_addr learned my addr 172.21.15.55:0/426191817 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-11-15T01:48:52.302 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:52.295+0000 7f7bbe59c700 1 -- 172.21.15.55:0/426191817 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bb000b200 con 0x7f7bc00fd100 2023-11-15T01:48:52.302 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:52.295+0000 7f7bbe59c700 1 --2- 172.21.15.55:0/426191817 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7bc00fd100 0x7f7bc00fc9d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f7bb0006c70 tx=0x7f7bb0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:48:52.303 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:52.295+0000 7f7baeffd700 1 -- 172.21.15.55:0/426191817 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7bb0011690 con 0x7f7bc00fd100 2023-11-15T01:48:52.303 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:52.295+0000 7f7baeffd700 1 -- 172.21.15.55:0/426191817 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f7bb001b880 con 0x7f7bc00fd100 2023-11-15T01:48:52.303 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:52.295+0000 7f7bbf59e700 1 -- 172.21.15.55:0/426191817 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bc00faff0 con 0x7f7bc00fd100 2023-11-15T01:48:52.303 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:52.295+0000 7f7baeffd700 1 -- 172.21.15.55:0/426191817 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7bb001a650 con 0x7f7bc00fd100 2023-11-15T01:48:52.303 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:52.295+0000 7f7bbf59e700 1 -- 172.21.15.55:0/426191817 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bc00fb400 con 0x7f7bc00fd100 2023-11-15T01:48:52.303 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:52.295+0000 7f7baeffd700 1 -- 172.21.15.55:0/426191817 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f7bb000f070 con 0x7f7bc00fd100 2023-11-15T01:48:52.304 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:52.295+0000 7f7baeffd700 1 -- 172.21.15.55:0/426191817 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f7bb00259e0 con 0x7f7bc00fd100 2023-11-15T01:48:52.304 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:52.296+0000 7f7bbf59e700 1 -- 172.21.15.55:0/426191817 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bc0045ad0 con 0x7f7bc00fd100 2023-11-15T01:48:52.304 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:52.298+0000 7f7baeffd700 1 -- 172.21.15.55:0/426191817 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f7bb0016070 con 0x7f7bc00fd100 2023-11-15T01:48:52.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:51 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:48:50.906778+0000 mon.smithi055 (mon.0) 6 : audit [INF] from='client.? 172.21.15.55:0/2323281156' entity='client.admin' 2023-11-15T01:48:52.373 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:52.372+0000 7f7bbf59e700 1 -- 172.21.15.55:0/426191817 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f7bc00008d0 con 0x7f7bc00fd100 2023-11-15T01:48:52.373 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:52.372+0000 7f7baeffd700 1 -- 172.21.15.55:0/426191817 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f7bb001bd60 con 0x7f7bc00fd100 2023-11-15T01:48:52.379 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:52.380 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: { 2023-11-15T01:48:52.380 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "fsid": "0cb4fc0c-8359-11ee-8dbe-212e2dc638e7", 2023-11-15T01:48:52.380 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "health": { 2023-11-15T01:48:52.380 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2023-11-15T01:48:52.380 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "checks": {}, 2023-11-15T01:48:52.381 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "mutes": [] 2023-11-15T01:48:52.381 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: }, 2023-11-15T01:48:52.381 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "election_epoch": 5, 2023-11-15T01:48:52.381 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "quorum": [ 2023-11-15T01:48:52.381 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 0 2023-11-15T01:48:52.382 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: ], 2023-11-15T01:48:52.382 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "quorum_names": [ 2023-11-15T01:48:52.382 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "smithi055" 2023-11-15T01:48:52.382 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: ], 2023-11-15T01:48:52.382 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "quorum_age": 1, 2023-11-15T01:48:52.383 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "monmap": { 2023-11-15T01:48:52.383 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T01:48:52.383 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2023-11-15T01:48:52.383 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "num_mons": 1 2023-11-15T01:48:52.383 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: }, 2023-11-15T01:48:52.383 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "osdmap": { 2023-11-15T01:48:52.384 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T01:48:52.384 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "num_osds": 0, 2023-11-15T01:48:52.384 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "num_up_osds": 0, 2023-11-15T01:48:52.384 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "osd_up_since": 0, 2023-11-15T01:48:52.384 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "num_in_osds": 0, 2023-11-15T01:48:52.385 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "osd_in_since": 0, 2023-11-15T01:48:52.385 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2023-11-15T01:48:52.385 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: }, 2023-11-15T01:48:52.385 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "pgmap": { 2023-11-15T01:48:52.385 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "pgs_by_state": [], 2023-11-15T01:48:52.385 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "num_pgs": 0, 2023-11-15T01:48:52.386 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "num_pools": 0, 2023-11-15T01:48:52.386 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "num_objects": 0, 2023-11-15T01:48:52.386 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "data_bytes": 0, 2023-11-15T01:48:52.386 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "bytes_used": 0, 2023-11-15T01:48:52.386 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "bytes_avail": 0, 2023-11-15T01:48:52.387 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "bytes_total": 0 2023-11-15T01:48:52.387 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: }, 2023-11-15T01:48:52.387 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "fsmap": { 2023-11-15T01:48:52.387 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T01:48:52.387 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "by_rank": [], 2023-11-15T01:48:52.387 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "up:standby": 0 2023-11-15T01:48:52.388 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: }, 2023-11-15T01:48:52.388 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "mgrmap": { 2023-11-15T01:48:52.388 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "available": false, 2023-11-15T01:48:52.388 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "num_standbys": 0, 2023-11-15T01:48:52.388 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "modules": [ 2023-11-15T01:48:52.389 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "iostat", 2023-11-15T01:48:52.389 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "restful" 2023-11-15T01:48:52.389 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: ], 2023-11-15T01:48:52.389 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "services": {} 2023-11-15T01:48:52.389 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: }, 2023-11-15T01:48:52.389 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "servicemap": { 2023-11-15T01:48:52.390 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T01:48:52.390 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "modified": "2023-11-15T01:48:46.996378+0000", 2023-11-15T01:48:52.390 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "services": {} 2023-11-15T01:48:52.390 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: }, 2023-11-15T01:48:52.390 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "progress_events": {} 2023-11-15T01:48:52.391 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: } 2023-11-15T01:48:52.391 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:52.373+0000 7f7bbf59e700 1 -- 172.21.15.55:0/426191817 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7bc00fd100 msgr2=0x7f7bc00fc9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:48:52.391 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:52.373+0000 7f7bbf59e700 1 --2- 172.21.15.55:0/426191817 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7bc00fd100 0x7f7bc00fc9d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f7bb0006c70 tx=0x7f7bb0006ee0).stop 2023-11-15T01:48:52.391 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:52.373+0000 7f7bbf59e700 1 -- 172.21.15.55:0/426191817 shutdown_connections 2023-11-15T01:48:52.391 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:52.373+0000 7f7bbf59e700 1 --2- 172.21.15.55:0/426191817 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7bc00fd100 0x7f7bc00fc9d0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:48:52.392 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:52.373+0000 7f7bbf59e700 1 -- 172.21.15.55:0/426191817 >> 172.21.15.55:0/426191817 conn(0x7f7bc00f59a0 msgr2=0x7f7bc00ff180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:48:52.392 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:52.373+0000 7f7bbf59e700 1 -- 172.21.15.55:0/426191817 shutdown_connections 2023-11-15T01:48:52.392 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:52.373+0000 7f7bbf59e700 1 -- 172.21.15.55:0/426191817 wait complete. 2023-11-15T01:48:52.747 INFO:teuthology.orchestra.run.smithi055.stderr:mgr not available, waiting (1/15)... 2023-11-15T01:48:53.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:52 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:48:52.373068+0000 mon.smithi055 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.55:0/426191817' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-11-15T01:48:54.749 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi055 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/0cb4fc0c-8359-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpc1yi9jse:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpra3bp1py:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2023-11-15T01:48:55.140 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:55.139+0000 7f62c9153700 1 Processor -- start 2023-11-15T01:48:55.141 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:55.139+0000 7f62c9153700 1 -- start start 2023-11-15T01:48:55.141 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:55.139+0000 7f62c9153700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f62c40fab80 0x7f62c40fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T01:48:55.142 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:55.139+0000 7f62c9153700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62c40fb490 con 0x7f62c40fab80 2023-11-15T01:48:55.143 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:55.139+0000 7f62c2d9d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f62c40fab80 0x7f62c40fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:48:55.143 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:55.140+0000 7f62c2d9d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f62c40fab80 0x7f62c40fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:33896/0 (socket says 172.21.15.55:33896) 2023-11-15T01:48:55.143 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:55.140+0000 7f62c2d9d700 1 -- 172.21.15.55:0/3410036605 learned_addr learned my addr 172.21.15.55:0/3410036605 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-11-15T01:48:55.143 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:55.140+0000 7f62c2d9d700 1 -- 172.21.15.55:0/3410036605 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62c40fb5d0 con 0x7f62c40fab80 2023-11-15T01:48:55.143 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:55.140+0000 7f62c2d9d700 1 --2- 172.21.15.55:0/3410036605 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f62c40fab80 0x7f62c40fafa0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f62b400b5f0 tx=0x7f62b4006ee0).ready entity=mon.0 client_cookie=28da43137f6cd6f server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:48:55.144 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:55.140+0000 7f62c1d9b700 1 -- 172.21.15.55:0/3410036605 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f62b400d600 con 0x7f62c40fab80 2023-11-15T01:48:55.144 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:55.140+0000 7f62c1d9b700 1 -- 172.21.15.55:0/3410036605 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f62b400dbd0 con 0x7f62c40fab80 2023-11-15T01:48:55.144 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:55.140+0000 7f62c1d9b700 1 -- 172.21.15.55:0/3410036605 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f62b4011c00 con 0x7f62c40fab80 2023-11-15T01:48:55.144 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:55.141+0000 7f62c9153700 1 -- 172.21.15.55:0/3410036605 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f62c40fab80 msgr2=0x7f62c40fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:48:55.144 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:55.141+0000 7f62c9153700 1 --2- 172.21.15.55:0/3410036605 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f62c40fab80 0x7f62c40fafa0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f62b400b5f0 tx=0x7f62b4006ee0).stop 2023-11-15T01:48:55.145 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:55.141+0000 7f62c9153700 1 -- 172.21.15.55:0/3410036605 shutdown_connections 2023-11-15T01:48:55.145 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:55.141+0000 7f62c9153700 1 --2- 172.21.15.55:0/3410036605 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f62c40fab80 0x7f62c40fafa0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:48:55.145 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:55.141+0000 7f62c9153700 1 -- 172.21.15.55:0/3410036605 >> 172.21.15.55:0/3410036605 conn(0x7f62c40f63c0 msgr2=0x7f62c40f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:48:55.145 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:55.141+0000 7f62c9153700 1 -- 172.21.15.55:0/3410036605 shutdown_connections 2023-11-15T01:48:55.147 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:55.141+0000 7f62c9153700 1 -- 172.21.15.55:0/3410036605 wait complete. 2023-11-15T01:48:55.147 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:55.142+0000 7f62c9153700 1 Processor -- start 2023-11-15T01:48:55.147 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:55.143+0000 7f62c9153700 1 -- start start 2023-11-15T01:48:55.148 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:55.143+0000 7f62c9153700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f62c40fab80 0x7f62c410d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:48:55.148 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:55.143+0000 7f62c9153700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62c4070470 con 0x7f62c40fab80 2023-11-15T01:48:55.148 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:55.143+0000 7f62c2d9d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f62c40fab80 0x7f62c410d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:48:55.148 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:55.143+0000 7f62c2d9d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f62c40fab80 0x7f62c410d700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:33912/0 (socket says 172.21.15.55:33912) 2023-11-15T01:48:55.148 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:55.143+0000 7f62c2d9d700 1 -- 172.21.15.55:0/2232111134 learned_addr learned my addr 172.21.15.55:0/2232111134 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-11-15T01:48:55.149 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:55.143+0000 7f62c2d9d700 1 -- 172.21.15.55:0/2232111134 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62b400b200 con 0x7f62c40fab80 2023-11-15T01:48:55.149 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:55.143+0000 7f62c2d9d700 1 --2- 172.21.15.55:0/2232111134 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f62c40fab80 0x7f62c410d700 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f62b400b1d0 tx=0x7f62b4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:48:55.149 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:55.144+0000 7f62ab7fe700 1 -- 172.21.15.55:0/2232111134 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f62b401b410 con 0x7f62c40fab80 2023-11-15T01:48:55.149 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:55.144+0000 7f62ab7fe700 1 -- 172.21.15.55:0/2232111134 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f62b401b9e0 con 0x7f62c40fab80 2023-11-15T01:48:55.149 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:55.144+0000 7f62ab7fe700 1 -- 172.21.15.55:0/2232111134 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f62b401a840 con 0x7f62c40fab80 2023-11-15T01:48:55.150 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:55.144+0000 7f62c9153700 1 -- 172.21.15.55:0/2232111134 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62c4070670 con 0x7f62c40fab80 2023-11-15T01:48:55.150 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:55.144+0000 7f62c9153700 1 -- 172.21.15.55:0/2232111134 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62c410ddb0 con 0x7f62c40fab80 2023-11-15T01:48:55.150 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:55.144+0000 7f62ab7fe700 1 -- 172.21.15.55:0/2232111134 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f62b400f070 con 0x7f62c40fab80 2023-11-15T01:48:55.150 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:55.144+0000 7f62ab7fe700 1 -- 172.21.15.55:0/2232111134 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f62b4025c90 con 0x7f62c40fab80 2023-11-15T01:48:55.150 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:55.145+0000 7f62c9153700 1 -- 172.21.15.55:0/2232111134 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62c4045ad0 con 0x7f62c40fab80 2023-11-15T01:48:55.151 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:55.149+0000 7f62ab7fe700 1 -- 172.21.15.55:0/2232111134 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f62b4016070 con 0x7f62c40fab80 2023-11-15T01:48:55.223 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:55.222+0000 7f62c9153700 1 -- 172.21.15.55:0/2232111134 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f62c4027a00 con 0x7f62c40fab80 2023-11-15T01:48:55.229 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:55.223+0000 7f62ab7fe700 1 -- 172.21.15.55:0/2232111134 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f62b401aea0 con 0x7f62c40fab80 2023-11-15T01:48:55.230 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:55.230 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: { 2023-11-15T01:48:55.230 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "fsid": "0cb4fc0c-8359-11ee-8dbe-212e2dc638e7", 2023-11-15T01:48:55.230 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "health": { 2023-11-15T01:48:55.230 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2023-11-15T01:48:55.231 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "checks": {}, 2023-11-15T01:48:55.231 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "mutes": [] 2023-11-15T01:48:55.231 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: }, 2023-11-15T01:48:55.231 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "election_epoch": 5, 2023-11-15T01:48:55.231 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "quorum": [ 2023-11-15T01:48:55.232 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 0 2023-11-15T01:48:55.232 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: ], 2023-11-15T01:48:55.232 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "quorum_names": [ 2023-11-15T01:48:55.232 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "smithi055" 2023-11-15T01:48:55.232 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: ], 2023-11-15T01:48:55.232 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "quorum_age": 4, 2023-11-15T01:48:55.233 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "monmap": { 2023-11-15T01:48:55.233 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T01:48:55.233 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2023-11-15T01:48:55.233 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "num_mons": 1 2023-11-15T01:48:55.233 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: }, 2023-11-15T01:48:55.234 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "osdmap": { 2023-11-15T01:48:55.234 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T01:48:55.234 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "num_osds": 0, 2023-11-15T01:48:55.234 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "num_up_osds": 0, 2023-11-15T01:48:55.234 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "osd_up_since": 0, 2023-11-15T01:48:55.234 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "num_in_osds": 0, 2023-11-15T01:48:55.235 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "osd_in_since": 0, 2023-11-15T01:48:55.235 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2023-11-15T01:48:55.235 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: }, 2023-11-15T01:48:55.235 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "pgmap": { 2023-11-15T01:48:55.235 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "pgs_by_state": [], 2023-11-15T01:48:55.236 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "num_pgs": 0, 2023-11-15T01:48:55.236 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "num_pools": 0, 2023-11-15T01:48:55.236 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "num_objects": 0, 2023-11-15T01:48:55.236 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "data_bytes": 0, 2023-11-15T01:48:55.237 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "bytes_used": 0, 2023-11-15T01:48:55.237 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "bytes_avail": 0, 2023-11-15T01:48:55.237 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "bytes_total": 0 2023-11-15T01:48:55.237 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: }, 2023-11-15T01:48:55.237 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "fsmap": { 2023-11-15T01:48:55.238 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T01:48:55.238 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "by_rank": [], 2023-11-15T01:48:55.238 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "up:standby": 0 2023-11-15T01:48:55.238 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: }, 2023-11-15T01:48:55.238 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "mgrmap": { 2023-11-15T01:48:55.239 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "available": false, 2023-11-15T01:48:55.239 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "num_standbys": 0, 2023-11-15T01:48:55.239 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "modules": [ 2023-11-15T01:48:55.239 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "iostat", 2023-11-15T01:48:55.239 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "restful" 2023-11-15T01:48:55.239 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: ], 2023-11-15T01:48:55.240 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "services": {} 2023-11-15T01:48:55.240 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: }, 2023-11-15T01:48:55.240 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "servicemap": { 2023-11-15T01:48:55.240 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T01:48:55.240 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "modified": "2023-11-15T01:48:46.996378+0000", 2023-11-15T01:48:55.241 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "services": {} 2023-11-15T01:48:55.241 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: }, 2023-11-15T01:48:55.241 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "progress_events": {} 2023-11-15T01:48:55.241 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: } 2023-11-15T01:48:55.241 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:55.224+0000 7f62c9153700 1 -- 172.21.15.55:0/2232111134 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f62c40fab80 msgr2=0x7f62c410d700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:48:55.242 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:55.224+0000 7f62c9153700 1 --2- 172.21.15.55:0/2232111134 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f62c40fab80 0x7f62c410d700 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f62b400b1d0 tx=0x7f62b4006ee0).stop 2023-11-15T01:48:55.242 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:55.228+0000 7f62c9153700 1 -- 172.21.15.55:0/2232111134 shutdown_connections 2023-11-15T01:48:55.242 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:55.228+0000 7f62c9153700 1 --2- 172.21.15.55:0/2232111134 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f62c40fab80 0x7f62c410d700 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:48:55.242 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:55.228+0000 7f62c9153700 1 -- 172.21.15.55:0/2232111134 >> 172.21.15.55:0/2232111134 conn(0x7f62c40f63c0 msgr2=0x7f62c40f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:48:55.242 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:55.228+0000 7f62c9153700 1 -- 172.21.15.55:0/2232111134 shutdown_connections 2023-11-15T01:48:55.243 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:55.228+0000 7f62c9153700 1 -- 172.21.15.55:0/2232111134 wait complete. 2023-11-15T01:48:55.541 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:55 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:48:55.223935+0000 mon.smithi055 (mon.0) 8 : audit [DBG] from='client.? 172.21.15.55:0/2232111134' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-11-15T01:48:55.597 INFO:teuthology.orchestra.run.smithi055.stderr:mgr not available, waiting (2/15)... 2023-11-15T01:48:57.600 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi055 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/0cb4fc0c-8359-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpc1yi9jse:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpra3bp1py:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2023-11-15T01:48:57.768 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:57 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:48:57.447836+0000 mon.smithi055 (mon.0) 9 : cluster [INF] Activating manager daemon smithi055.szwngw 2023-11-15T01:48:57.769 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:57 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:48:57.450713+0000 mon.smithi055 (mon.0) 10 : cluster [DBG] mgrmap e2: smithi055.szwngw(active, starting, since 0.00300753s) 2023-11-15T01:48:57.769 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:57 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:48:57.451834+0000 mon.smithi055 (mon.0) 11 : audit [DBG] from='mgr.14100 172.21.15.55:0/2408703319' entity='mgr.smithi055.szwngw' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-11-15T01:48:57.769 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:57 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:48:57.451993+0000 mon.smithi055 (mon.0) 12 : audit [DBG] from='mgr.14100 172.21.15.55:0/2408703319' entity='mgr.smithi055.szwngw' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-11-15T01:48:57.769 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:57 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:48:57.452199+0000 mon.smithi055 (mon.0) 13 : audit [DBG] from='mgr.14100 172.21.15.55:0/2408703319' entity='mgr.smithi055.szwngw' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-11-15T01:48:57.769 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:57 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:48:57.452412+0000 mon.smithi055 (mon.0) 14 : audit [DBG] from='mgr.14100 172.21.15.55:0/2408703319' entity='mgr.smithi055.szwngw' cmd=[{"prefix": "mon metadata", "id": "smithi055"}]: dispatch 2023-11-15T01:48:57.770 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:57 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:48:57.452661+0000 mon.smithi055 (mon.0) 15 : audit [DBG] from='mgr.14100 172.21.15.55:0/2408703319' entity='mgr.smithi055.szwngw' cmd=[{"prefix": "mgr metadata", "who": "smithi055.szwngw", "id": "smithi055.szwngw"}]: dispatch 2023-11-15T01:48:57.770 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:57 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:48:57.466394+0000 mon.smithi055 (mon.0) 16 : cluster [INF] Manager daemon smithi055.szwngw is now available 2023-11-15T01:48:57.770 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:57 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:48:57.477233+0000 mon.smithi055 (mon.0) 17 : audit [INF] from='mgr.14100 172.21.15.55:0/2408703319' entity='mgr.smithi055.szwngw' 2023-11-15T01:48:57.770 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:57 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:48:57.481145+0000 mon.smithi055 (mon.0) 18 : audit [INF] from='mgr.14100 172.21.15.55:0/2408703319' entity='mgr.smithi055.szwngw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi055.szwngw/mirror_snapshot_schedule"}]: dispatch 2023-11-15T01:48:57.771 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:57 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:48:57.482458+0000 mon.smithi055 (mon.0) 19 : audit [INF] from='mgr.14100 172.21.15.55:0/2408703319' entity='mgr.smithi055.szwngw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi055.szwngw/trash_purge_schedule"}]: dispatch 2023-11-15T01:48:57.771 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:57 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:48:57.488004+0000 mon.smithi055 (mon.0) 20 : audit [INF] from='mgr.14100 172.21.15.55:0/2408703319' entity='mgr.smithi055.szwngw' 2023-11-15T01:48:57.771 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:57 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:48:57.490544+0000 mon.smithi055 (mon.0) 21 : audit [INF] from='mgr.14100 172.21.15.55:0/2408703319' entity='mgr.smithi055.szwngw' 2023-11-15T01:48:57.977 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:57.975+0000 7fabe255d700 1 Processor -- start 2023-11-15T01:48:57.978 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:57.975+0000 7fabe255d700 1 -- start start 2023-11-15T01:48:57.978 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:57.976+0000 7fabe255d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fabdc070390 0x7fabdc100d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T01:48:57.978 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:57.976+0000 7fabe255d700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabdc0fc770 con 0x7fabdc070390 2023-11-15T01:48:57.978 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:57.976+0000 7fabdbfff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fabdc070390 0x7fabdc100d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:48:57.979 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:57.976+0000 7fabdbfff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fabdc070390 0x7fabdc100d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:33966/0 (socket says 172.21.15.55:33966) 2023-11-15T01:48:57.979 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:57.976+0000 7fabdbfff700 1 -- 172.21.15.55:0/2176334711 learned_addr learned my addr 172.21.15.55:0/2176334711 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-11-15T01:48:57.979 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:57.976+0000 7fabdbfff700 1 -- 172.21.15.55:0/2176334711 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabdc101270 con 0x7fabdc070390 2023-11-15T01:48:57.979 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:57.976+0000 7fabdbfff700 1 --2- 172.21.15.55:0/2176334711 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fabdc070390 0x7fabdc100d80 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fabcc00b5f0 tx=0x7fabcc006ee0).ready entity=mon.0 client_cookie=39c728651d3e5c68 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:48:57.979 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:57.977+0000 7fabdaffd700 1 -- 172.21.15.55:0/2176334711 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fabcc00d600 con 0x7fabdc070390 2023-11-15T01:48:57.980 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:57.977+0000 7fabdaffd700 1 -- 172.21.15.55:0/2176334711 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fabcc00dbd0 con 0x7fabdc070390 2023-11-15T01:48:57.980 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:57.977+0000 7fabdaffd700 1 -- 172.21.15.55:0/2176334711 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fabcc011c00 con 0x7fabdc070390 2023-11-15T01:48:57.982 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:57.977+0000 7fabe255d700 1 -- 172.21.15.55:0/2176334711 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fabdc070390 msgr2=0x7fabdc100d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:48:57.983 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:57.977+0000 7fabe255d700 1 --2- 172.21.15.55:0/2176334711 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fabdc070390 0x7fabdc100d80 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fabcc00b5f0 tx=0x7fabcc006ee0).stop 2023-11-15T01:48:57.983 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:57.977+0000 7fabe255d700 1 -- 172.21.15.55:0/2176334711 shutdown_connections 2023-11-15T01:48:57.983 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:57.977+0000 7fabe255d700 1 --2- 172.21.15.55:0/2176334711 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fabdc070390 0x7fabdc100d80 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:48:57.983 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:57.977+0000 7fabe255d700 1 -- 172.21.15.55:0/2176334711 >> 172.21.15.55:0/2176334711 conn(0x7fabdc0f63c0 msgr2=0x7fabdc0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:48:57.983 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:57.977+0000 7fabe255d700 1 -- 172.21.15.55:0/2176334711 shutdown_connections 2023-11-15T01:48:57.984 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:57.977+0000 7fabe255d700 1 -- 172.21.15.55:0/2176334711 wait complete. 2023-11-15T01:48:57.984 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:57.978+0000 7fabe255d700 1 Processor -- start 2023-11-15T01:48:57.984 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:57.978+0000 7fabe255d700 1 -- start start 2023-11-15T01:48:57.984 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:57.978+0000 7fabe255d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fabdc070390 0x7fabdc10cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:48:57.984 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:57.978+0000 7fabe255d700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabdc10d200 con 0x7fabdc070390 2023-11-15T01:48:57.985 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:57.978+0000 7fabdbfff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fabdc070390 0x7fabdc10cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:48:57.985 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:57.978+0000 7fabdbfff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fabdc070390 0x7fabdc10cd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:33974/0 (socket says 172.21.15.55:33974) 2023-11-15T01:48:57.985 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:57.978+0000 7fabdbfff700 1 -- 172.21.15.55:0/2523958344 learned_addr learned my addr 172.21.15.55:0/2523958344 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-11-15T01:48:57.985 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:57.979+0000 7fabdbfff700 1 -- 172.21.15.55:0/2523958344 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabcc00b200 con 0x7fabdc070390 2023-11-15T01:48:57.985 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:57.979+0000 7fabdbfff700 1 --2- 172.21.15.55:0/2523958344 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fabdc070390 0x7fabdc10cd10 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fabcc00e040 tx=0x7fabcc006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:48:57.985 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:57.979+0000 7fabd8ff9700 1 -- 172.21.15.55:0/2523958344 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fabcc00d9f0 con 0x7fabdc070390 2023-11-15T01:48:57.986 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:57.979+0000 7fabe255d700 1 -- 172.21.15.55:0/2523958344 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabdc10d400 con 0x7fabdc070390 2023-11-15T01:48:57.986 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:57.979+0000 7fabd8ff9700 1 -- 172.21.15.55:0/2523958344 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fabcc01a9f0 con 0x7fabdc070390 2023-11-15T01:48:57.986 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:57.979+0000 7fabd8ff9700 1 -- 172.21.15.55:0/2523958344 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fabcc01b8c0 con 0x7fabdc070390 2023-11-15T01:48:57.987 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:57.979+0000 7fabe255d700 1 -- 172.21.15.55:0/2523958344 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabdc10d810 con 0x7fabdc070390 2023-11-15T01:48:57.987 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:57.981+0000 7fabe255d700 1 -- 172.21.15.55:0/2523958344 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabdc045ad0 con 0x7fabdc070390 2023-11-15T01:48:57.987 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:57.981+0000 7fabd8ff9700 1 -- 172.21.15.55:0/2523958344 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 2) v1 ==== 35959+0+0 (secure 0 0 0) 0x7fabcc00f070 con 0x7fabdc070390 2023-11-15T01:48:57.987 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:57.981+0000 7fabd8ff9700 1 -- 172.21.15.55:0/2523958344 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fabcc041830 con 0x7fabdc070390 2023-11-15T01:48:57.988 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:57.983+0000 7fabd8ff9700 1 -- 172.21.15.55:0/2523958344 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fabcc02db20 con 0x7fabdc070390 2023-11-15T01:48:58.064 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:58.056+0000 7fabe255d700 1 -- 172.21.15.55:0/2523958344 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fabdc101d70 con 0x7fabdc070390 2023-11-15T01:48:58.064 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:58.057+0000 7fabd8ff9700 1 -- 172.21.15.55:0/2523958344 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7fabcc016030 con 0x7fabdc070390 2023-11-15T01:48:58.064 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:58.064 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: { 2023-11-15T01:48:58.064 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "fsid": "0cb4fc0c-8359-11ee-8dbe-212e2dc638e7", 2023-11-15T01:48:58.065 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "health": { 2023-11-15T01:48:58.065 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2023-11-15T01:48:58.065 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "checks": {}, 2023-11-15T01:48:58.065 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "mutes": [] 2023-11-15T01:48:58.065 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: }, 2023-11-15T01:48:58.066 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "election_epoch": 5, 2023-11-15T01:48:58.066 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "quorum": [ 2023-11-15T01:48:58.066 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 0 2023-11-15T01:48:58.066 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: ], 2023-11-15T01:48:58.066 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "quorum_names": [ 2023-11-15T01:48:58.067 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "smithi055" 2023-11-15T01:48:58.067 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: ], 2023-11-15T01:48:58.067 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "quorum_age": 7, 2023-11-15T01:48:58.067 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "monmap": { 2023-11-15T01:48:58.067 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T01:48:58.068 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2023-11-15T01:48:58.068 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "num_mons": 1 2023-11-15T01:48:58.068 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: }, 2023-11-15T01:48:58.068 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "osdmap": { 2023-11-15T01:48:58.068 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T01:48:58.069 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "num_osds": 0, 2023-11-15T01:48:58.069 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "num_up_osds": 0, 2023-11-15T01:48:58.069 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "osd_up_since": 0, 2023-11-15T01:48:58.069 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "num_in_osds": 0, 2023-11-15T01:48:58.069 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "osd_in_since": 0, 2023-11-15T01:48:58.069 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2023-11-15T01:48:58.070 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: }, 2023-11-15T01:48:58.070 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "pgmap": { 2023-11-15T01:48:58.070 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "pgs_by_state": [], 2023-11-15T01:48:58.070 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "num_pgs": 0, 2023-11-15T01:48:58.071 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "num_pools": 0, 2023-11-15T01:48:58.071 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "num_objects": 0, 2023-11-15T01:48:58.071 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "data_bytes": 0, 2023-11-15T01:48:58.071 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "bytes_used": 0, 2023-11-15T01:48:58.071 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "bytes_avail": 0, 2023-11-15T01:48:58.071 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "bytes_total": 0 2023-11-15T01:48:58.072 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: }, 2023-11-15T01:48:58.072 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "fsmap": { 2023-11-15T01:48:58.072 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T01:48:58.072 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "by_rank": [], 2023-11-15T01:48:58.072 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "up:standby": 0 2023-11-15T01:48:58.073 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: }, 2023-11-15T01:48:58.073 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "mgrmap": { 2023-11-15T01:48:58.073 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "available": false, 2023-11-15T01:48:58.073 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "num_standbys": 0, 2023-11-15T01:48:58.073 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "modules": [ 2023-11-15T01:48:58.073 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "iostat", 2023-11-15T01:48:58.074 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "restful" 2023-11-15T01:48:58.074 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: ], 2023-11-15T01:48:58.074 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "services": {} 2023-11-15T01:48:58.074 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: }, 2023-11-15T01:48:58.074 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "servicemap": { 2023-11-15T01:48:58.075 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T01:48:58.075 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "modified": "2023-11-15T01:48:46.996378+0000", 2023-11-15T01:48:58.075 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "services": {} 2023-11-15T01:48:58.075 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: }, 2023-11-15T01:48:58.075 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "progress_events": {} 2023-11-15T01:48:58.076 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: } 2023-11-15T01:48:58.076 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:58.057+0000 7fabe255d700 1 -- 172.21.15.55:0/2523958344 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fabdc070390 msgr2=0x7fabdc10cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:48:58.076 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:58.057+0000 7fabe255d700 1 --2- 172.21.15.55:0/2523958344 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fabdc070390 0x7fabdc10cd10 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fabcc00e040 tx=0x7fabcc006ee0).stop 2023-11-15T01:48:58.076 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:58.057+0000 7fabe255d700 1 -- 172.21.15.55:0/2523958344 shutdown_connections 2023-11-15T01:48:58.076 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:58.058+0000 7fabe255d700 1 --2- 172.21.15.55:0/2523958344 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fabdc070390 0x7fabdc10cd10 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:48:58.077 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:58.058+0000 7fabe255d700 1 -- 172.21.15.55:0/2523958344 >> 172.21.15.55:0/2523958344 conn(0x7fabdc0f63c0 msgr2=0x7fabdc0f7f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:48:58.077 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:58.058+0000 7fabe255d700 1 -- 172.21.15.55:0/2523958344 shutdown_connections 2023-11-15T01:48:58.077 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:48:58.058+0000 7fabe255d700 1 -- 172.21.15.55:0/2523958344 wait complete. 2023-11-15T01:48:58.351 INFO:teuthology.orchestra.run.smithi055.stderr:mgr not available, waiting (3/15)... 2023-11-15T01:48:58.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:58 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:48:58.057765+0000 mon.smithi055 (mon.0) 22 : audit [DBG] from='client.? 172.21.15.55:0/2523958344' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-11-15T01:48:58.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:48:58 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:48:58.456580+0000 mon.smithi055 (mon.0) 23 : cluster [DBG] mgrmap e3: smithi055.szwngw(active, since 1.00887s) 2023-11-15T01:49:00.354 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi055 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/0cb4fc0c-8359-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpc1yi9jse:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpra3bp1py:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2023-11-15T01:49:00.727 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:00.725+0000 7fd0ebd19700 1 Processor -- start 2023-11-15T01:49:00.727 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:00.726+0000 7fd0ebd19700 1 -- start start 2023-11-15T01:49:00.727 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:00.726+0000 7fd0ebd19700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd0e4070430 0x7fd0e40fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T01:49:00.727 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:00.726+0000 7fd0ebd19700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0e40fedd0 con 0x7fd0e4070430 2023-11-15T01:49:00.730 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:00.726+0000 7fd0e9ab5700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd0e4070430 0x7fd0e40fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:49:00.731 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:00.726+0000 7fd0e9ab5700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd0e4070430 0x7fd0e40fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:53092/0 (socket says 172.21.15.55:53092) 2023-11-15T01:49:00.731 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:00.726+0000 7fd0e9ab5700 1 -- 172.21.15.55:0/2819956876 learned_addr learned my addr 172.21.15.55:0/2819956876 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-11-15T01:49:00.731 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:00.726+0000 7fd0e9ab5700 1 -- 172.21.15.55:0/2819956876 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0e40fef10 con 0x7fd0e4070430 2023-11-15T01:49:00.731 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:00.727+0000 7fd0e9ab5700 1 --2- 172.21.15.55:0/2819956876 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd0e4070430 0x7fd0e40fe8e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fd0d400b5f0 tx=0x7fd0d4006ee0).ready entity=mon.0 client_cookie=a61dd93ccacd7892 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:49:00.732 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:00.727+0000 7fd0e8ab3700 1 -- 172.21.15.55:0/2819956876 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd0d400d600 con 0x7fd0e4070430 2023-11-15T01:49:00.732 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:00.727+0000 7fd0e8ab3700 1 -- 172.21.15.55:0/2819956876 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fd0d400dbd0 con 0x7fd0e4070430 2023-11-15T01:49:00.732 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:00.727+0000 7fd0e8ab3700 1 -- 172.21.15.55:0/2819956876 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd0d4011c00 con 0x7fd0e4070430 2023-11-15T01:49:00.732 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:00.727+0000 7fd0ebd19700 1 -- 172.21.15.55:0/2819956876 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd0e4070430 msgr2=0x7fd0e40fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:49:00.732 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:00.727+0000 7fd0ebd19700 1 --2- 172.21.15.55:0/2819956876 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd0e4070430 0x7fd0e40fe8e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fd0d400b5f0 tx=0x7fd0d4006ee0).stop 2023-11-15T01:49:00.732 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:00.727+0000 7fd0ebd19700 1 -- 172.21.15.55:0/2819956876 shutdown_connections 2023-11-15T01:49:00.733 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:00.727+0000 7fd0ebd19700 1 --2- 172.21.15.55:0/2819956876 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd0e4070430 0x7fd0e40fe8e0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:49:00.733 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:00.727+0000 7fd0ebd19700 1 -- 172.21.15.55:0/2819956876 >> 172.21.15.55:0/2819956876 conn(0x7fd0e40f63c0 msgr2=0x7fd0e40f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:49:00.733 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:00.727+0000 7fd0ebd19700 1 -- 172.21.15.55:0/2819956876 shutdown_connections 2023-11-15T01:49:00.733 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:00.727+0000 7fd0ebd19700 1 -- 172.21.15.55:0/2819956876 wait complete. 2023-11-15T01:49:00.733 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:00.728+0000 7fd0ebd19700 1 Processor -- start 2023-11-15T01:49:00.734 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:00.728+0000 7fd0ebd19700 1 -- start start 2023-11-15T01:49:00.734 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:00.728+0000 7fd0ebd19700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd0e4070430 0x7fd0e410cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:49:00.734 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:00.728+0000 7fd0ebd19700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0e410d0d0 con 0x7fd0e4070430 2023-11-15T01:49:00.734 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:00.728+0000 7fd0e9ab5700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd0e4070430 0x7fd0e410cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:49:00.734 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:00.728+0000 7fd0e9ab5700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd0e4070430 0x7fd0e410cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:53102/0 (socket says 172.21.15.55:53102) 2023-11-15T01:49:00.735 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:00.728+0000 7fd0e9ab5700 1 -- 172.21.15.55:0/1238759000 learned_addr learned my addr 172.21.15.55:0/1238759000 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-11-15T01:49:00.735 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:00.729+0000 7fd0e9ab5700 1 -- 172.21.15.55:0/1238759000 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0d400b200 con 0x7fd0e4070430 2023-11-15T01:49:00.735 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:00.729+0000 7fd0e9ab5700 1 --2- 172.21.15.55:0/1238759000 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd0e4070430 0x7fd0e410cbe0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fd0d400b190 tx=0x7fd0d4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:49:00.735 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:00.729+0000 7fd0d27fc700 1 -- 172.21.15.55:0/1238759000 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd0d401b410 con 0x7fd0e4070430 2023-11-15T01:49:00.735 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:00.729+0000 7fd0d27fc700 1 -- 172.21.15.55:0/1238759000 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fd0d401b9e0 con 0x7fd0e4070430 2023-11-15T01:49:00.736 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:00.729+0000 7fd0d27fc700 1 -- 172.21.15.55:0/1238759000 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd0d401a880 con 0x7fd0e4070430 2023-11-15T01:49:00.736 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:00.729+0000 7fd0ebd19700 1 -- 172.21.15.55:0/1238759000 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0e410d2d0 con 0x7fd0e4070430 2023-11-15T01:49:00.736 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:00.729+0000 7fd0ebd19700 1 -- 172.21.15.55:0/1238759000 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0e410d6e0 con 0x7fd0e4070430 2023-11-15T01:49:00.736 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:00.730+0000 7fd0d27fc700 1 -- 172.21.15.55:0/1238759000 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 4) v1 ==== 36109+0+0 (secure 0 0 0) 0x7fd0d400f070 con 0x7fd0e4070430 2023-11-15T01:49:00.737 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:00.730+0000 7fd0d27fc700 1 --2- 172.21.15.55:0/1238759000 >> [v2:172.21.15.55:6800/207668394,v1:172.21.15.55:6801/207668394] conn(0x7fd0cc02fd50 0x7fd0cc032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:49:00.737 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:00.730+0000 7fd0d27fc700 1 -- 172.21.15.55:0/1238759000 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fd0d40411c0 con 0x7fd0e4070430 2023-11-15T01:49:00.738 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:00.730+0000 7fd0e92b4700 1 --2- 172.21.15.55:0/1238759000 >> [v2:172.21.15.55:6800/207668394,v1:172.21.15.55:6801/207668394] conn(0x7fd0cc02fd50 0x7fd0cc032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:49:00.738 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:00.730+0000 7fd0ebd19700 1 -- 172.21.15.55:0/1238759000 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0e4059100 con 0x7fd0e4070430 2023-11-15T01:49:00.738 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:00.731+0000 7fd0e92b4700 1 --2- 172.21.15.55:0/1238759000 >> [v2:172.21.15.55:6800/207668394,v1:172.21.15.55:6801/207668394] conn(0x7fd0cc02fd50 0x7fd0cc032210 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fd0e00096a0 tx=0x7fd0e0006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:49:00.738 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:00.735+0000 7fd0d27fc700 1 -- 172.21.15.55:0/1238759000 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd0d4016070 con 0x7fd0e4070430 2023-11-15T01:49:00.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:00 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:48:59.505471+0000 mon.smithi055 (mon.0) 24 : cluster [DBG] mgrmap e4: smithi055.szwngw(active, since 2s) 2023-11-15T01:49:00.965 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:00.958+0000 7fd0ebd19700 1 -- 172.21.15.55:0/1238759000 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fd0e40ffb90 con 0x7fd0e4070430 2023-11-15T01:49:00.966 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:00.958+0000 7fd0d27fc700 1 -- 172.21.15.55:0/1238759000 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7fd0d4025460 con 0x7fd0e4070430 2023-11-15T01:49:00.966 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:00.966 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: { 2023-11-15T01:49:00.966 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "fsid": "0cb4fc0c-8359-11ee-8dbe-212e2dc638e7", 2023-11-15T01:49:00.966 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "health": { 2023-11-15T01:49:00.967 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2023-11-15T01:49:00.967 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "checks": {}, 2023-11-15T01:49:00.967 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "mutes": [] 2023-11-15T01:49:00.967 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: }, 2023-11-15T01:49:00.967 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "election_epoch": 5, 2023-11-15T01:49:00.968 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "quorum": [ 2023-11-15T01:49:00.968 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 0 2023-11-15T01:49:00.968 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: ], 2023-11-15T01:49:00.968 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "quorum_names": [ 2023-11-15T01:49:00.968 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "smithi055" 2023-11-15T01:49:00.969 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: ], 2023-11-15T01:49:00.969 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "quorum_age": 10, 2023-11-15T01:49:00.969 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "monmap": { 2023-11-15T01:49:00.969 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T01:49:00.969 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2023-11-15T01:49:00.969 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "num_mons": 1 2023-11-15T01:49:00.970 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: }, 2023-11-15T01:49:00.970 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "osdmap": { 2023-11-15T01:49:00.970 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T01:49:00.970 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "num_osds": 0, 2023-11-15T01:49:00.970 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "num_up_osds": 0, 2023-11-15T01:49:00.971 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "osd_up_since": 0, 2023-11-15T01:49:00.971 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "num_in_osds": 0, 2023-11-15T01:49:00.971 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "osd_in_since": 0, 2023-11-15T01:49:00.971 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2023-11-15T01:49:00.971 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: }, 2023-11-15T01:49:00.971 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "pgmap": { 2023-11-15T01:49:00.972 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "pgs_by_state": [], 2023-11-15T01:49:00.972 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "num_pgs": 0, 2023-11-15T01:49:00.972 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "num_pools": 0, 2023-11-15T01:49:00.972 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "num_objects": 0, 2023-11-15T01:49:00.972 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "data_bytes": 0, 2023-11-15T01:49:00.973 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "bytes_used": 0, 2023-11-15T01:49:00.973 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "bytes_avail": 0, 2023-11-15T01:49:00.973 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "bytes_total": 0 2023-11-15T01:49:00.973 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: }, 2023-11-15T01:49:00.973 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "fsmap": { 2023-11-15T01:49:00.974 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T01:49:00.974 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "by_rank": [], 2023-11-15T01:49:00.974 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "up:standby": 0 2023-11-15T01:49:00.974 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: }, 2023-11-15T01:49:00.974 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "mgrmap": { 2023-11-15T01:49:00.975 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "available": true, 2023-11-15T01:49:00.975 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "num_standbys": 0, 2023-11-15T01:49:00.975 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "modules": [ 2023-11-15T01:49:00.975 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "iostat", 2023-11-15T01:49:00.975 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "restful" 2023-11-15T01:49:00.975 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: ], 2023-11-15T01:49:00.976 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "services": {} 2023-11-15T01:49:00.976 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: }, 2023-11-15T01:49:00.976 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "servicemap": { 2023-11-15T01:49:00.976 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T01:49:00.976 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "modified": "2023-11-15T01:48:46.996378+0000", 2023-11-15T01:49:00.977 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "services": {} 2023-11-15T01:49:00.977 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: }, 2023-11-15T01:49:00.977 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "progress_events": {} 2023-11-15T01:49:00.977 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: } 2023-11-15T01:49:00.977 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:00.960+0000 7fd0ebd19700 1 -- 172.21.15.55:0/1238759000 >> [v2:172.21.15.55:6800/207668394,v1:172.21.15.55:6801/207668394] conn(0x7fd0cc02fd50 msgr2=0x7fd0cc032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:49:00.978 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:00.960+0000 7fd0ebd19700 1 --2- 172.21.15.55:0/1238759000 >> [v2:172.21.15.55:6800/207668394,v1:172.21.15.55:6801/207668394] conn(0x7fd0cc02fd50 0x7fd0cc032210 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fd0e00096a0 tx=0x7fd0e0006b40).stop 2023-11-15T01:49:00.978 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:00.960+0000 7fd0ebd19700 1 -- 172.21.15.55:0/1238759000 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd0e4070430 msgr2=0x7fd0e410cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:49:00.978 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:00.960+0000 7fd0ebd19700 1 --2- 172.21.15.55:0/1238759000 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd0e4070430 0x7fd0e410cbe0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fd0d400b190 tx=0x7fd0d4006ee0).stop 2023-11-15T01:49:00.978 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:00.960+0000 7fd0ebd19700 1 -- 172.21.15.55:0/1238759000 shutdown_connections 2023-11-15T01:49:00.978 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:00.960+0000 7fd0ebd19700 1 --2- 172.21.15.55:0/1238759000 >> [v2:172.21.15.55:6800/207668394,v1:172.21.15.55:6801/207668394] conn(0x7fd0cc02fd50 0x7fd0cc032210 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:49:00.979 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:00.960+0000 7fd0ebd19700 1 --2- 172.21.15.55:0/1238759000 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd0e4070430 0x7fd0e410cbe0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:49:00.979 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:00.960+0000 7fd0ebd19700 1 -- 172.21.15.55:0/1238759000 >> 172.21.15.55:0/1238759000 conn(0x7fd0e40f63c0 msgr2=0x7fd0e40f7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:49:00.979 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:00.960+0000 7fd0ebd19700 1 -- 172.21.15.55:0/1238759000 shutdown_connections 2023-11-15T01:49:00.979 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:00.960+0000 7fd0ebd19700 1 -- 172.21.15.55:0/1238759000 wait complete. 2023-11-15T01:49:01.313 INFO:teuthology.orchestra.run.smithi055.stderr:mgr is available 2023-11-15T01:49:01.314 INFO:teuthology.orchestra.run.smithi055.stderr:Enabling cephadm module... 2023-11-15T01:49:01.314 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi055 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/0cb4fc0c-8359-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpc1yi9jse:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpra3bp1py:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable cephadm 2023-11-15T01:49:01.578 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:01 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:49:00.959441+0000 mon.smithi055 (mon.0) 25 : audit [DBG] from='client.? 172.21.15.55:0/1238759000' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-11-15T01:49:01.814 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:01.743+0000 7f5a471c2700 1 Processor -- start 2023-11-15T01:49:01.814 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:01.743+0000 7f5a471c2700 1 -- start start 2023-11-15T01:49:01.814 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:01.743+0000 7f5a471c2700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5a40070430 0x7f5a400fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T01:49:01.815 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:01.743+0000 7f5a471c2700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a400fedd0 con 0x7f5a40070430 2023-11-15T01:49:01.815 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:01.743+0000 7f5a44f5e700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5a40070430 0x7f5a400fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:49:01.815 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:01.743+0000 7f5a44f5e700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5a40070430 0x7f5a400fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:53114/0 (socket says 172.21.15.55:53114) 2023-11-15T01:49:01.815 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:01.743+0000 7f5a44f5e700 1 -- 172.21.15.55:0/872800877 learned_addr learned my addr 172.21.15.55:0/872800877 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-11-15T01:49:01.816 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:01.744+0000 7f5a44f5e700 1 -- 172.21.15.55:0/872800877 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a400fef10 con 0x7f5a40070430 2023-11-15T01:49:01.816 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:01.744+0000 7f5a44f5e700 1 --2- 172.21.15.55:0/872800877 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5a40070430 0x7f5a400fe8e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f5a3400b5f0 tx=0x7f5a34006ee0).ready entity=mon.0 client_cookie=6c865ceda0661cb7 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:49:01.816 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:01.744+0000 7f5a3f7fe700 1 -- 172.21.15.55:0/872800877 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5a3400d600 con 0x7f5a40070430 2023-11-15T01:49:01.816 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:01.744+0000 7f5a3f7fe700 1 -- 172.21.15.55:0/872800877 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f5a3400dbd0 con 0x7f5a40070430 2023-11-15T01:49:01.816 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:01.744+0000 7f5a3f7fe700 1 -- 172.21.15.55:0/872800877 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5a34011c00 con 0x7f5a40070430 2023-11-15T01:49:01.816 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:01.745+0000 7f5a471c2700 1 -- 172.21.15.55:0/872800877 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5a40070430 msgr2=0x7f5a400fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:49:01.817 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:01.745+0000 7f5a471c2700 1 --2- 172.21.15.55:0/872800877 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5a40070430 0x7f5a400fe8e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f5a3400b5f0 tx=0x7f5a34006ee0).stop 2023-11-15T01:49:01.817 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:01.745+0000 7f5a471c2700 1 -- 172.21.15.55:0/872800877 shutdown_connections 2023-11-15T01:49:01.817 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:01.745+0000 7f5a471c2700 1 --2- 172.21.15.55:0/872800877 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5a40070430 0x7f5a400fe8e0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:49:01.817 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:01.745+0000 7f5a471c2700 1 -- 172.21.15.55:0/872800877 >> 172.21.15.55:0/872800877 conn(0x7f5a400f63c0 msgr2=0x7f5a400f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:49:01.817 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:01.745+0000 7f5a471c2700 1 -- 172.21.15.55:0/872800877 shutdown_connections 2023-11-15T01:49:01.818 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:01.745+0000 7f5a471c2700 1 -- 172.21.15.55:0/872800877 wait complete. 2023-11-15T01:49:01.818 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:01.745+0000 7f5a471c2700 1 Processor -- start 2023-11-15T01:49:01.818 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:01.746+0000 7f5a471c2700 1 -- start start 2023-11-15T01:49:01.818 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:01.746+0000 7f5a471c2700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5a40070430 0x7f5a400fc360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:49:01.818 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:01.746+0000 7f5a471c2700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a400fa7e0 con 0x7f5a40070430 2023-11-15T01:49:01.819 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:01.746+0000 7f5a44f5e700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5a40070430 0x7f5a400fc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:49:01.819 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:01.746+0000 7f5a44f5e700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5a40070430 0x7f5a400fc360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:53120/0 (socket says 172.21.15.55:53120) 2023-11-15T01:49:01.819 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:01.746+0000 7f5a44f5e700 1 -- 172.21.15.55:0/548259818 learned_addr learned my addr 172.21.15.55:0/548259818 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-11-15T01:49:01.819 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:01.746+0000 7f5a44f5e700 1 -- 172.21.15.55:0/548259818 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a3400b200 con 0x7f5a40070430 2023-11-15T01:49:01.819 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:01.747+0000 7f5a44f5e700 1 --2- 172.21.15.55:0/548259818 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5a40070430 0x7f5a400fc360 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f5a34002f40 tx=0x7f5a34006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:49:01.820 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:01.747+0000 7f5a3d7fa700 1 -- 172.21.15.55:0/548259818 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5a34011690 con 0x7f5a40070430 2023-11-15T01:49:01.820 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:01.747+0000 7f5a3d7fa700 1 -- 172.21.15.55:0/548259818 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f5a3401b880 con 0x7f5a40070430 2023-11-15T01:49:01.820 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:01.747+0000 7f5a471c2700 1 -- 172.21.15.55:0/548259818 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a400fa980 con 0x7f5a40070430 2023-11-15T01:49:01.820 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:01.747+0000 7f5a3d7fa700 1 -- 172.21.15.55:0/548259818 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5a3401a6b0 con 0x7f5a40070430 2023-11-15T01:49:01.820 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:01.747+0000 7f5a471c2700 1 -- 172.21.15.55:0/548259818 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a400fad90 con 0x7f5a40070430 2023-11-15T01:49:01.821 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:01.747+0000 7f5a3d7fa700 1 -- 172.21.15.55:0/548259818 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 4) v1 ==== 36109+0+0 (secure 0 0 0) 0x7f5a3400f070 con 0x7f5a40070430 2023-11-15T01:49:01.821 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:01.748+0000 7f5a3d7fa700 1 --2- 172.21.15.55:0/548259818 >> [v2:172.21.15.55:6800/207668394,v1:172.21.15.55:6801/207668394] conn(0x7f5a2402fd00 0x7f5a240321c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:49:01.821 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:01.748+0000 7f5a3d7fa700 1 -- 172.21.15.55:0/548259818 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f5a34041360 con 0x7f5a40070430 2023-11-15T01:49:01.821 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:01.748+0000 7f5a3ffff700 1 --2- 172.21.15.55:0/548259818 >> [v2:172.21.15.55:6800/207668394,v1:172.21.15.55:6801/207668394] conn(0x7f5a2402fd00 0x7f5a240321c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:49:01.821 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:01.748+0000 7f5a471c2700 1 -- 172.21.15.55:0/548259818 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a40059100 con 0x7f5a40070430 2023-11-15T01:49:01.822 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:01.748+0000 7f5a3ffff700 1 --2- 172.21.15.55:0/548259818 >> [v2:172.21.15.55:6800/207668394,v1:172.21.15.55:6801/207668394] conn(0x7f5a2402fd00 0x7f5a240321c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f5a2c0096a0 tx=0x7f5a2c006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:49:01.822 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:01.752+0000 7f5a3d7fa700 1 -- 172.21.15.55:0/548259818 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5a34016030 con 0x7f5a40070430 2023-11-15T01:49:01.959 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:01.958+0000 7f5a471c2700 1 -- 172.21.15.55:0/548259818 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f5a40045ad0 con 0x7f5a40070430 2023-11-15T01:49:02.514 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:02.512+0000 7f5a3d7fa700 1 -- 172.21.15.55:0/548259818 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f5a3401a8e0 con 0x7f5a40070430 2023-11-15T01:49:02.515 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:02.514+0000 7f5a3d7fa700 1 -- 172.21.15.55:0/548259818 <== mon.0 v2:172.21.15.55:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f5a3402e7a0 con 0x7f5a40070430 2023-11-15T01:49:02.517 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:02.516+0000 7f5a471c2700 1 -- 172.21.15.55:0/548259818 >> [v2:172.21.15.55:6800/207668394,v1:172.21.15.55:6801/207668394] conn(0x7f5a2402fd00 msgr2=0x7f5a240321c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:49:02.517 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:02.516+0000 7f5a471c2700 1 --2- 172.21.15.55:0/548259818 >> [v2:172.21.15.55:6800/207668394,v1:172.21.15.55:6801/207668394] conn(0x7f5a2402fd00 0x7f5a240321c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f5a2c0096a0 tx=0x7f5a2c006b40).stop 2023-11-15T01:49:02.518 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:02.516+0000 7f5a471c2700 1 -- 172.21.15.55:0/548259818 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5a40070430 msgr2=0x7f5a400fc360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:49:02.519 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:02.516+0000 7f5a471c2700 1 --2- 172.21.15.55:0/548259818 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5a40070430 0x7f5a400fc360 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f5a34002f40 tx=0x7f5a34006ee0).stop 2023-11-15T01:49:02.519 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:02.517+0000 7f5a471c2700 1 -- 172.21.15.55:0/548259818 shutdown_connections 2023-11-15T01:49:02.519 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:02.517+0000 7f5a471c2700 1 --2- 172.21.15.55:0/548259818 >> [v2:172.21.15.55:6800/207668394,v1:172.21.15.55:6801/207668394] conn(0x7f5a2402fd00 0x7f5a240321c0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:49:02.519 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:02.517+0000 7f5a471c2700 1 --2- 172.21.15.55:0/548259818 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5a40070430 0x7f5a400fc360 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:49:02.519 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:02.517+0000 7f5a471c2700 1 -- 172.21.15.55:0/548259818 >> 172.21.15.55:0/548259818 conn(0x7f5a400f63c0 msgr2=0x7f5a400f7fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:49:02.520 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:02.517+0000 7f5a471c2700 1 -- 172.21.15.55:0/548259818 shutdown_connections 2023-11-15T01:49:02.520 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:02.517+0000 7f5a471c2700 1 -- 172.21.15.55:0/548259818 wait complete. 2023-11-15T01:49:02.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:02 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:49:01.959697+0000 mon.smithi055 (mon.0) 26 : audit [INF] from='client.? 172.21.15.55:0/548259818' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2023-11-15T01:49:03.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:03 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:49:02.514932+0000 mon.smithi055 (mon.0) 27 : audit [INF] from='client.? 172.21.15.55:0/548259818' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2023-11-15T01:49:03.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:03 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:49:02.515017+0000 mon.smithi055 (mon.0) 28 : cluster [DBG] mgrmap e5: smithi055.szwngw(active, since 5s) 2023-11-15T01:49:03.828 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi055 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/0cb4fc0c-8359-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpc1yi9jse:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpra3bp1py:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2023-11-15T01:49:04.203 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:04.201+0000 7f649bfff700 1 Processor -- start 2023-11-15T01:49:04.203 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:04.201+0000 7f649bfff700 1 -- start start 2023-11-15T01:49:04.204 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:04.201+0000 7f649bfff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f649c0703d0 0x7f649c100b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T01:49:04.204 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:04.201+0000 7f649bfff700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f649c0fc580 con 0x7f649c0703d0 2023-11-15T01:49:04.204 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:04.202+0000 7f649affd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f649c0703d0 0x7f649c100b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:49:04.204 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:04.202+0000 7f649affd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f649c0703d0 0x7f649c100b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:53136/0 (socket says 172.21.15.55:53136) 2023-11-15T01:49:04.204 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:04.202+0000 7f649affd700 1 -- 172.21.15.55:0/1263997057 learned_addr learned my addr 172.21.15.55:0/1263997057 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-11-15T01:49:04.205 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:04.202+0000 7f649affd700 1 -- 172.21.15.55:0/1263997057 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f649c101080 con 0x7f649c0703d0 2023-11-15T01:49:04.205 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:04.202+0000 7f649affd700 1 --2- 172.21.15.55:0/1263997057 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f649c0703d0 0x7f649c100b90 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f648c00b5f0 tx=0x7f648c006ee0).ready entity=mon.0 client_cookie=3ad62687979f7af7 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:49:04.205 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:04.202+0000 7f6499ffb700 1 -- 172.21.15.55:0/1263997057 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f648c00d600 con 0x7f649c0703d0 2023-11-15T01:49:04.208 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:04.202+0000 7f6499ffb700 1 -- 172.21.15.55:0/1263997057 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f648c00dbd0 con 0x7f649c0703d0 2023-11-15T01:49:04.208 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:04.202+0000 7f6499ffb700 1 -- 172.21.15.55:0/1263997057 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f648c011c00 con 0x7f649c0703d0 2023-11-15T01:49:04.208 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:04.203+0000 7f649bfff700 1 -- 172.21.15.55:0/1263997057 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f649c0703d0 msgr2=0x7f649c100b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:49:04.208 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:04.203+0000 7f649bfff700 1 --2- 172.21.15.55:0/1263997057 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f649c0703d0 0x7f649c100b90 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f648c00b5f0 tx=0x7f648c006ee0).stop 2023-11-15T01:49:04.209 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:04.203+0000 7f649bfff700 1 -- 172.21.15.55:0/1263997057 shutdown_connections 2023-11-15T01:49:04.209 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:04.203+0000 7f649bfff700 1 --2- 172.21.15.55:0/1263997057 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f649c0703d0 0x7f649c100b90 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:49:04.210 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:04.203+0000 7f649bfff700 1 -- 172.21.15.55:0/1263997057 >> 172.21.15.55:0/1263997057 conn(0x7f649c0f6210 msgr2=0x7f649c0f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:49:04.210 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:04.203+0000 7f649bfff700 1 -- 172.21.15.55:0/1263997057 shutdown_connections 2023-11-15T01:49:04.210 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:04.203+0000 7f649bfff700 1 -- 172.21.15.55:0/1263997057 wait complete. 2023-11-15T01:49:04.210 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:04.204+0000 7f649bfff700 1 Processor -- start 2023-11-15T01:49:04.210 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:04.204+0000 7f649bfff700 1 -- start start 2023-11-15T01:49:04.211 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:04.204+0000 7f649bfff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f649c0703d0 0x7f649c10ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:49:04.211 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:04.204+0000 7f649bfff700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f649c10cf60 con 0x7f649c0703d0 2023-11-15T01:49:04.211 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:04.204+0000 7f649affd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f649c0703d0 0x7f649c10ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:49:04.211 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:04.204+0000 7f649affd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f649c0703d0 0x7f649c10ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:53146/0 (socket says 172.21.15.55:53146) 2023-11-15T01:49:04.211 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:04.204+0000 7f649affd700 1 -- 172.21.15.55:0/4288198727 learned_addr learned my addr 172.21.15.55:0/4288198727 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-11-15T01:49:04.212 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:04.205+0000 7f649affd700 1 -- 172.21.15.55:0/4288198727 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f648c00b200 con 0x7f649c0703d0 2023-11-15T01:49:04.212 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:04.205+0000 7f649affd700 1 --2- 172.21.15.55:0/4288198727 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f649c0703d0 0x7f649c10ca70 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f648c00b190 tx=0x7f648c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:49:04.212 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:04.205+0000 7f64837fe700 1 -- 172.21.15.55:0/4288198727 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f648c01b410 con 0x7f649c0703d0 2023-11-15T01:49:04.212 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:04.205+0000 7f649bfff700 1 -- 172.21.15.55:0/4288198727 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f649c10d160 con 0x7f649c0703d0 2023-11-15T01:49:04.212 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:04.205+0000 7f64837fe700 1 -- 172.21.15.55:0/4288198727 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f648c01b9e0 con 0x7f649c0703d0 2023-11-15T01:49:04.213 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:04.205+0000 7f64837fe700 1 -- 172.21.15.55:0/4288198727 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f648c01a880 con 0x7f649c0703d0 2023-11-15T01:49:04.213 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:04.205+0000 7f649bfff700 1 -- 172.21.15.55:0/4288198727 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f649c10d570 con 0x7f649c0703d0 2023-11-15T01:49:04.213 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:04.206+0000 7f64837fe700 1 -- 172.21.15.55:0/4288198727 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f648c00f070 con 0x7f649c0703d0 2023-11-15T01:49:04.213 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:04.206+0000 7f649bfff700 1 -- 172.21.15.55:0/4288198727 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f649c045ad0 con 0x7f649c0703d0 2023-11-15T01:49:04.213 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:04.206+0000 7f64837fe700 1 --2- 172.21.15.55:0/4288198727 >> [v2:172.21.15.55:6800/207668394,v1:172.21.15.55:6801/207668394] conn(0x7f648802fd50 0x7f6488032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:49:04.214 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:04.206+0000 7f64837fe700 1 -- 172.21.15.55:0/4288198727 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f648c0411f0 con 0x7f649c0703d0 2023-11-15T01:49:04.214 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:04.206+0000 7f649a7fc700 1 -- 172.21.15.55:0/4288198727 >> [v2:172.21.15.55:6800/207668394,v1:172.21.15.55:6801/207668394] conn(0x7f648802fd50 msgr2=0x7f6488032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.55:6800/207668394 2023-11-15T01:49:04.214 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:04.206+0000 7f649a7fc700 1 --2- 172.21.15.55:0/4288198727 >> [v2:172.21.15.55:6800/207668394,v1:172.21.15.55:6801/207668394] conn(0x7f648802fd50 0x7f6488032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-11-15T01:49:04.214 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:04.211+0000 7f64837fe700 1 -- 172.21.15.55:0/4288198727 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f648c010360 con 0x7f649c0703d0 2023-11-15T01:49:04.408 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:04.407+0000 7f649a7fc700 1 -- 172.21.15.55:0/4288198727 >> [v2:172.21.15.55:6800/207668394,v1:172.21.15.55:6801/207668394] conn(0x7f648802fd50 msgr2=0x7f6488032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.55:6800/207668394 2023-11-15T01:49:04.408 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:04.407+0000 7f649a7fc700 1 --2- 172.21.15.55:0/4288198727 >> [v2:172.21.15.55:6800/207668394,v1:172.21.15.55:6801/207668394] conn(0x7f648802fd50 0x7f6488032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-11-15T01:49:04.419 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:04.418+0000 7f649bfff700 1 -- 172.21.15.55:0/4288198727 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f649c106940 con 0x7f649c0703d0 2023-11-15T01:49:04.420 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:04.419+0000 7f64837fe700 1 -- 172.21.15.55:0/4288198727 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7f648c016070 con 0x7f649c0703d0 2023-11-15T01:49:04.420 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: { 2023-11-15T01:49:04.421 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "epoch": 5, 2023-11-15T01:49:04.421 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "available": true, 2023-11-15T01:49:04.421 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "active_name": "smithi055.szwngw", 2023-11-15T01:49:04.421 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "num_standby": 0 2023-11-15T01:49:04.421 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: } 2023-11-15T01:49:04.423 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:04.421+0000 7f649bfff700 1 -- 172.21.15.55:0/4288198727 >> [v2:172.21.15.55:6800/207668394,v1:172.21.15.55:6801/207668394] conn(0x7f648802fd50 msgr2=0x7f6488032210 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-15T01:49:04.423 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:04.421+0000 7f649bfff700 1 --2- 172.21.15.55:0/4288198727 >> [v2:172.21.15.55:6800/207668394,v1:172.21.15.55:6801/207668394] conn(0x7f648802fd50 0x7f6488032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-15T01:49:04.423 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:04.421+0000 7f649bfff700 1 -- 172.21.15.55:0/4288198727 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f649c0703d0 msgr2=0x7f649c10ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:49:04.424 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:04.421+0000 7f649bfff700 1 --2- 172.21.15.55:0/4288198727 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f649c0703d0 0x7f649c10ca70 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f648c00b190 tx=0x7f648c006ee0).stop 2023-11-15T01:49:04.424 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:04.421+0000 7f649bfff700 1 -- 172.21.15.55:0/4288198727 shutdown_connections 2023-11-15T01:49:04.424 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:04.421+0000 7f649bfff700 1 --2- 172.21.15.55:0/4288198727 >> [v2:172.21.15.55:6800/207668394,v1:172.21.15.55:6801/207668394] conn(0x7f648802fd50 0x7f6488032210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-15T01:49:04.424 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:04.421+0000 7f649bfff700 1 --2- 172.21.15.55:0/4288198727 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f649c0703d0 0x7f649c10ca70 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:49:04.424 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:04.421+0000 7f649bfff700 1 -- 172.21.15.55:0/4288198727 >> 172.21.15.55:0/4288198727 conn(0x7f649c0f6210 msgr2=0x7f649c0f7ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:49:04.425 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:04.421+0000 7f649bfff700 1 -- 172.21.15.55:0/4288198727 shutdown_connections 2023-11-15T01:49:04.425 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:04.421+0000 7f649bfff700 1 -- 172.21.15.55:0/4288198727 wait complete. 2023-11-15T01:49:04.715 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:04 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:49:04.419681+0000 mon.smithi055 (mon.0) 29 : audit [DBG] from='client.? 172.21.15.55:0/4288198727' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-11-15T01:49:04.779 INFO:teuthology.orchestra.run.smithi055.stderr:Waiting for the mgr to restart... 2023-11-15T01:49:04.780 INFO:teuthology.orchestra.run.smithi055.stderr:Waiting for mgr epoch 5... 2023-11-15T01:49:04.780 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi055 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/0cb4fc0c-8359-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpc1yi9jse:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpra3bp1py:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2023-11-15T01:49:05.166 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:05.164+0000 7f4d779b0700 1 Processor -- start 2023-11-15T01:49:05.166 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:05.164+0000 7f4d779b0700 1 -- start start 2023-11-15T01:49:05.166 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:05.165+0000 7f4d779b0700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4d700724b0 0x7f4d70070a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T01:49:05.167 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:05.165+0000 7f4d779b0700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d700728d0 con 0x7f4d700724b0 2023-11-15T01:49:05.167 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:05.165+0000 7f4d7574c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4d700724b0 0x7f4d70070a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:49:05.167 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:05.165+0000 7f4d7574c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4d700724b0 0x7f4d70070a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:53160/0 (socket says 172.21.15.55:53160) 2023-11-15T01:49:05.169 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:05.165+0000 7f4d7574c700 1 -- 172.21.15.55:0/1383494477 learned_addr learned my addr 172.21.15.55:0/1383494477 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-11-15T01:49:05.169 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:05.165+0000 7f4d7574c700 1 -- 172.21.15.55:0/1383494477 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d70070f40 con 0x7f4d700724b0 2023-11-15T01:49:05.169 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:05.165+0000 7f4d7574c700 1 --2- 172.21.15.55:0/1383494477 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4d700724b0 0x7f4d70070a50 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f4d6c00b5f0 tx=0x7f4d6c006ee0).ready entity=mon.0 client_cookie=fa151fec19751286 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:49:05.169 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:05.165+0000 7f4d5ffff700 1 -- 172.21.15.55:0/1383494477 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d6c00d600 con 0x7f4d700724b0 2023-11-15T01:49:05.170 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:05.166+0000 7f4d5ffff700 1 -- 172.21.15.55:0/1383494477 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f4d6c00dbd0 con 0x7f4d700724b0 2023-11-15T01:49:05.170 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:05.166+0000 7f4d5ffff700 1 -- 172.21.15.55:0/1383494477 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d6c011c00 con 0x7f4d700724b0 2023-11-15T01:49:05.170 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:05.166+0000 7f4d779b0700 1 -- 172.21.15.55:0/1383494477 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4d700724b0 msgr2=0x7f4d70070a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:49:05.170 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:05.166+0000 7f4d779b0700 1 --2- 172.21.15.55:0/1383494477 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4d700724b0 0x7f4d70070a50 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f4d6c00b5f0 tx=0x7f4d6c006ee0).stop 2023-11-15T01:49:05.171 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:05.166+0000 7f4d779b0700 1 -- 172.21.15.55:0/1383494477 shutdown_connections 2023-11-15T01:49:05.171 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:05.166+0000 7f4d779b0700 1 --2- 172.21.15.55:0/1383494477 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4d700724b0 0x7f4d70070a50 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:49:05.171 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:05.166+0000 7f4d779b0700 1 -- 172.21.15.55:0/1383494477 >> 172.21.15.55:0/1383494477 conn(0x7f4d700f74c0 msgr2=0x7f4d700f98e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:49:05.171 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:05.166+0000 7f4d779b0700 1 -- 172.21.15.55:0/1383494477 shutdown_connections 2023-11-15T01:49:05.171 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:05.166+0000 7f4d779b0700 1 -- 172.21.15.55:0/1383494477 wait complete. 2023-11-15T01:49:05.171 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:05.167+0000 7f4d779b0700 1 Processor -- start 2023-11-15T01:49:05.172 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:05.167+0000 7f4d779b0700 1 -- start start 2023-11-15T01:49:05.172 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:05.167+0000 7f4d779b0700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4d700724b0 0x7f4d7010cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:49:05.172 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:05.167+0000 7f4d779b0700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d7010d0e0 con 0x7f4d700724b0 2023-11-15T01:49:05.172 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:05.167+0000 7f4d7574c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4d700724b0 0x7f4d7010cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:49:05.173 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:05.167+0000 7f4d7574c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4d700724b0 0x7f4d7010cbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:53166/0 (socket says 172.21.15.55:53166) 2023-11-15T01:49:05.173 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:05.167+0000 7f4d7574c700 1 -- 172.21.15.55:0/1343990362 learned_addr learned my addr 172.21.15.55:0/1343990362 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-11-15T01:49:05.173 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:05.168+0000 7f4d7574c700 1 -- 172.21.15.55:0/1343990362 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d6c00b200 con 0x7f4d700724b0 2023-11-15T01:49:05.173 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:05.168+0000 7f4d7574c700 1 --2- 172.21.15.55:0/1343990362 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4d700724b0 0x7f4d7010cbf0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f4d6c00b190 tx=0x7f4d6c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:49:05.174 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:05.168+0000 7f4d5dffb700 1 -- 172.21.15.55:0/1343990362 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d6c01b410 con 0x7f4d700724b0 2023-11-15T01:49:05.174 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:05.168+0000 7f4d5dffb700 1 -- 172.21.15.55:0/1343990362 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f4d6c01b9e0 con 0x7f4d700724b0 2023-11-15T01:49:05.174 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:05.168+0000 7f4d5dffb700 1 -- 172.21.15.55:0/1343990362 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d6c01a880 con 0x7f4d700724b0 2023-11-15T01:49:05.174 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:05.168+0000 7f4d779b0700 1 -- 172.21.15.55:0/1343990362 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d7010d2e0 con 0x7f4d700724b0 2023-11-15T01:49:05.174 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:05.168+0000 7f4d779b0700 1 -- 172.21.15.55:0/1343990362 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d7010d6f0 con 0x7f4d700724b0 2023-11-15T01:49:05.175 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:05.169+0000 7f4d5dffb700 1 -- 172.21.15.55:0/1343990362 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f4d6c00f070 con 0x7f4d700724b0 2023-11-15T01:49:05.175 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:05.169+0000 7f4d5dffb700 1 --2- 172.21.15.55:0/1343990362 >> [v2:172.21.15.55:6800/207668394,v1:172.21.15.55:6801/207668394] conn(0x7f4d5802fdb0 0x7f4d58032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:49:05.175 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:05.169+0000 7f4d5dffb700 1 -- 172.21.15.55:0/1343990362 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f4d6c041230 con 0x7f4d700724b0 2023-11-15T01:49:05.175 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:05.169+0000 7f4d74f4b700 1 -- 172.21.15.55:0/1343990362 >> [v2:172.21.15.55:6800/207668394,v1:172.21.15.55:6801/207668394] conn(0x7f4d5802fdb0 msgr2=0x7f4d58032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.55:6800/207668394 2023-11-15T01:49:05.175 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:05.169+0000 7f4d74f4b700 1 --2- 172.21.15.55:0/1343990362 >> [v2:172.21.15.55:6800/207668394,v1:172.21.15.55:6801/207668394] conn(0x7f4d5802fdb0 0x7f4d58032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-11-15T01:49:05.175 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:05.170+0000 7f4d779b0700 1 -- 172.21.15.55:0/1343990362 --> [v2:172.21.15.55:6800/207668394,v1:172.21.15.55:6801/207668394] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4d70059100 con 0x7f4d5802fdb0 2023-11-15T01:49:05.371 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:05.370+0000 7f4d74f4b700 1 -- 172.21.15.55:0/1343990362 >> [v2:172.21.15.55:6800/207668394,v1:172.21.15.55:6801/207668394] conn(0x7f4d5802fdb0 msgr2=0x7f4d58032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.55:6800/207668394 2023-11-15T01:49:05.371 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:05.370+0000 7f4d74f4b700 1 --2- 172.21.15.55:0/1343990362 >> [v2:172.21.15.55:6800/207668394,v1:172.21.15.55:6801/207668394] conn(0x7f4d5802fdb0 0x7f4d58032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-11-15T01:49:05.772 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:05.771+0000 7f4d74f4b700 1 -- 172.21.15.55:0/1343990362 >> [v2:172.21.15.55:6800/207668394,v1:172.21.15.55:6801/207668394] conn(0x7f4d5802fdb0 msgr2=0x7f4d58032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.55:6800/207668394 2023-11-15T01:49:05.772 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:05.771+0000 7f4d74f4b700 1 --2- 172.21.15.55:0/1343990362 >> [v2:172.21.15.55:6800/207668394,v1:172.21.15.55:6801/207668394] conn(0x7f4d5802fdb0 0x7f4d58032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-11-15T01:49:06.573 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:06.571+0000 7f4d74f4b700 1 -- 172.21.15.55:0/1343990362 >> [v2:172.21.15.55:6800/207668394,v1:172.21.15.55:6801/207668394] conn(0x7f4d5802fdb0 msgr2=0x7f4d58032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.55:6800/207668394 2023-11-15T01:49:06.573 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:06.571+0000 7f4d74f4b700 1 --2- 172.21.15.55:0/1343990362 >> [v2:172.21.15.55:6800/207668394,v1:172.21.15.55:6801/207668394] conn(0x7f4d5802fdb0 0x7f4d58032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-11-15T01:49:08.174 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:08.172+0000 7f4d5dffb700 1 -- 172.21.15.55:0/1343990362 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mgrmap(e 6) v1 ==== 35970+0+0 (secure 0 0 0) 0x7f4d6c022d80 con 0x7f4d700724b0 2023-11-15T01:49:08.174 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:08.172+0000 7f4d5dffb700 1 -- 172.21.15.55:0/1343990362 >> [v2:172.21.15.55:6800/207668394,v1:172.21.15.55:6801/207668394] conn(0x7f4d5802fdb0 msgr2=0x7f4d58032270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-15T01:49:08.175 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:08.172+0000 7f4d5dffb700 1 --2- 172.21.15.55:0/1343990362 >> [v2:172.21.15.55:6800/207668394,v1:172.21.15.55:6801/207668394] conn(0x7f4d5802fdb0 0x7f4d58032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-15T01:49:09.178 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:09.173+0000 7f4d5dffb700 1 -- 172.21.15.55:0/1343990362 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mgrmap(e 7) v1 ==== 36040+0+0 (secure 0 0 0) 0x7f4d6c01b740 con 0x7f4d700724b0 2023-11-15T01:49:09.178 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:09.174+0000 7f4d5dffb700 1 --2- 172.21.15.55:0/1343990362 >> [v2:172.21.15.55:6800/1950005071,v1:172.21.15.55:6801/1950005071] conn(0x7f4d580336d0 0x7f4d58035ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:49:09.178 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:09.174+0000 7f4d5dffb700 1 -- 172.21.15.55:0/1343990362 --> [v2:172.21.15.55:6800/1950005071,v1:172.21.15.55:6801/1950005071] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4d70059100 con 0x7f4d580336d0 2023-11-15T01:49:09.178 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:09.174+0000 7f4d74f4b700 1 --2- 172.21.15.55:0/1343990362 >> [v2:172.21.15.55:6800/1950005071,v1:172.21.15.55:6801/1950005071] conn(0x7f4d580336d0 0x7f4d58035ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:49:09.179 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:09.175+0000 7f4d74f4b700 1 --2- 172.21.15.55:0/1343990362 >> [v2:172.21.15.55:6800/1950005071,v1:172.21.15.55:6801/1950005071] conn(0x7f4d580336d0 0x7f4d58035ac0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f4d600036d0 tx=0x7f4d60006d20).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:49:09.179 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:09.176+0000 7f4d5dffb700 1 -- 172.21.15.55:0/1343990362 <== mgr.14116 v2:172.21.15.55:6800/1950005071 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7f4d70059100 con 0x7f4d580336d0 2023-11-15T01:49:09.184 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:09.182+0000 7f4d779b0700 1 -- 172.21.15.55:0/1343990362 --> [v2:172.21.15.55:6800/1950005071,v1:172.21.15.55:6801/1950005071] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f4d70045ad0 con 0x7f4d580336d0 2023-11-15T01:49:09.184 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:09.183+0000 7f4d5dffb700 1 -- 172.21.15.55:0/1343990362 <== mgr.14116 v2:172.21.15.55:6800/1950005071 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f4d70045ad0 con 0x7f4d580336d0 2023-11-15T01:49:09.186 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: { 2023-11-15T01:49:09.187 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2023-11-15T01:49:09.187 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "initialized": true 2023-11-15T01:49:09.187 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: } 2023-11-15T01:49:09.187 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:09.183+0000 7f4d779b0700 1 -- 172.21.15.55:0/1343990362 >> [v2:172.21.15.55:6800/1950005071,v1:172.21.15.55:6801/1950005071] conn(0x7f4d580336d0 msgr2=0x7f4d58035ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:49:09.187 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:09.183+0000 7f4d779b0700 1 --2- 172.21.15.55:0/1343990362 >> [v2:172.21.15.55:6800/1950005071,v1:172.21.15.55:6801/1950005071] conn(0x7f4d580336d0 0x7f4d58035ac0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f4d600036d0 tx=0x7f4d60006d20).stop 2023-11-15T01:49:09.188 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:09.183+0000 7f4d779b0700 1 -- 172.21.15.55:0/1343990362 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4d700724b0 msgr2=0x7f4d7010cbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:49:09.188 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:09.183+0000 7f4d779b0700 1 --2- 172.21.15.55:0/1343990362 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4d700724b0 0x7f4d7010cbf0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f4d6c00b190 tx=0x7f4d6c006ee0).stop 2023-11-15T01:49:09.188 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:09.183+0000 7f4d779b0700 1 -- 172.21.15.55:0/1343990362 shutdown_connections 2023-11-15T01:49:09.188 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:09.183+0000 7f4d779b0700 1 --2- 172.21.15.55:0/1343990362 >> [v2:172.21.15.55:6800/1950005071,v1:172.21.15.55:6801/1950005071] conn(0x7f4d580336d0 0x7f4d58035ac0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:49:09.188 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:09.183+0000 7f4d779b0700 1 --2- 172.21.15.55:0/1343990362 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4d700724b0 0x7f4d7010cbf0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:49:09.189 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:09.183+0000 7f4d779b0700 1 --2- 172.21.15.55:0/1343990362 >> [v2:172.21.15.55:6800/207668394,v1:172.21.15.55:6801/207668394] conn(0x7f4d5802fdb0 0x7f4d58032270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-15T01:49:09.189 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:09.183+0000 7f4d779b0700 1 -- 172.21.15.55:0/1343990362 >> 172.21.15.55:0/1343990362 conn(0x7f4d700f74c0 msgr2=0x7f4d700f8fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:49:09.189 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:09.183+0000 7f4d779b0700 1 -- 172.21.15.55:0/1343990362 shutdown_connections 2023-11-15T01:49:09.189 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:09.183+0000 7f4d779b0700 1 -- 172.21.15.55:0/1343990362 wait complete. 2023-11-15T01:49:09.578 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:09 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:49:08.121728+0000 mon.smithi055 (mon.0) 30 : cluster [INF] Active manager daemon smithi055.szwngw restarted 2023-11-15T01:49:09.579 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:09 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:49:08.122183+0000 mon.smithi055 (mon.0) 31 : cluster [INF] Activating manager daemon smithi055.szwngw 2023-11-15T01:49:09.579 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:09 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:49:08.123981+0000 mon.smithi055 (mon.0) 32 : cluster [DBG] osdmap e2: 0 total, 0 up, 0 in 2023-11-15T01:49:09.579 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:09 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:49:08.175310+0000 mon.smithi055 (mon.0) 33 : cluster [DBG] mgrmap e6: smithi055.szwngw(active, starting, since 0.053202s) 2023-11-15T01:49:09.579 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:09 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:49:08.176231+0000 mon.smithi055 (mon.0) 34 : audit [DBG] from='mgr.14116 172.21.15.55:0/3759764227' entity='mgr.smithi055.szwngw' cmd=[{"prefix": "mon metadata", "id": "smithi055"}]: dispatch 2023-11-15T01:49:09.579 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:09 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:49:08.177123+0000 mon.smithi055 (mon.0) 35 : audit [DBG] from='mgr.14116 172.21.15.55:0/3759764227' entity='mgr.smithi055.szwngw' cmd=[{"prefix": "mgr metadata", "who": "smithi055.szwngw", "id": "smithi055.szwngw"}]: dispatch 2023-11-15T01:49:09.580 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:09 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:49:08.177935+0000 mon.smithi055 (mon.0) 36 : audit [DBG] from='mgr.14116 172.21.15.55:0/3759764227' entity='mgr.smithi055.szwngw' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-11-15T01:49:09.580 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:09 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:49:08.178214+0000 mon.smithi055 (mon.0) 37 : audit [DBG] from='mgr.14116 172.21.15.55:0/3759764227' entity='mgr.smithi055.szwngw' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-11-15T01:49:09.580 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:09 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:49:08.178473+0000 mon.smithi055 (mon.0) 38 : audit [DBG] from='mgr.14116 172.21.15.55:0/3759764227' entity='mgr.smithi055.szwngw' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-11-15T01:49:09.580 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:09 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:49:08.192531+0000 mon.smithi055 (mon.0) 39 : cluster [INF] Manager daemon smithi055.szwngw is now available 2023-11-15T01:49:09.581 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:09 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:49:08.209292+0000 mon.smithi055 (mon.0) 40 : audit [INF] from='mgr.14116 172.21.15.55:0/3759764227' entity='mgr.smithi055.szwngw' 2023-11-15T01:49:09.581 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:09 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:49:08.212469+0000 mon.smithi055 (mon.0) 41 : audit [INF] from='mgr.14116 172.21.15.55:0/3759764227' entity='mgr.smithi055.szwngw' 2023-11-15T01:49:09.581 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:09 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:49:08.215573+0000 mon.smithi055 (mon.0) 42 : audit [INF] from='mgr.14116 172.21.15.55:0/3759764227' entity='mgr.smithi055.szwngw' 2023-11-15T01:49:09.581 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:09 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:49:08.324697+0000 mon.smithi055 (mon.0) 43 : audit [INF] from='mgr.14116 172.21.15.55:0/3759764227' entity='mgr.smithi055.szwngw' 2023-11-15T01:49:09.581 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:09 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:49:08.325433+0000 mon.smithi055 (mon.0) 44 : audit [DBG] from='mgr.14116 172.21.15.55:0/3759764227' entity='mgr.smithi055.szwngw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-15T01:49:09.582 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:09 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:49:08.327173+0000 mon.smithi055 (mon.0) 45 : audit [DBG] from='mgr.14116 172.21.15.55:0/3759764227' entity='mgr.smithi055.szwngw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-15T01:49:09.582 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:09 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:49:08.332556+0000 mon.smithi055 (mon.0) 46 : audit [DBG] from='mgr.14116 172.21.15.55:0/3759764227' entity='mgr.smithi055.szwngw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-15T01:49:09.582 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:09 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:49:08.340327+0000 mon.smithi055 (mon.0) 47 : audit [INF] from='mgr.14116 172.21.15.55:0/3759764227' entity='mgr.smithi055.szwngw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi055.szwngw/mirror_snapshot_schedule"}]: dispatch 2023-11-15T01:49:09.582 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:09 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:49:08.341616+0000 mon.smithi055 (mon.0) 48 : audit [INF] from='mgr.14116 172.21.15.55:0/3759764227' entity='mgr.smithi055.szwngw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi055.szwngw/trash_purge_schedule"}]: dispatch 2023-11-15T01:49:10.578 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:10 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:49:09.177267+0000 mgr.smithi055.szwngw (mgr.14116) 1 : audit [DBG] from='client.14120 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-11-15T01:49:10.578 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:10 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:49:09.178197+0000 mon.smithi055 (mon.0) 49 : cluster [DBG] mgrmap e7: smithi055.szwngw(active, since 1.05608s) 2023-11-15T01:49:10.578 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:10 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:49:09.184735+0000 mgr.smithi055.szwngw (mgr.14116) 2 : audit [DBG] from='client.14120 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-11-15T01:49:10.729 INFO:teuthology.orchestra.run.smithi055.stderr:mgr epoch 5 is available 2023-11-15T01:49:10.730 INFO:teuthology.orchestra.run.smithi055.stderr:Setting orchestrator backend to cephadm... 2023-11-15T01:49:10.730 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi055 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/0cb4fc0c-8359-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpc1yi9jse:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpra3bp1py:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch set backend cephadm 2023-11-15T01:49:11.578 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:11 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:49:10.187584+0000 mon.smithi055 (mon.0) 50 : cluster [DBG] mgrmap e8: smithi055.szwngw(active, since 2s) 2023-11-15T01:49:12.187 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:12.127+0000 7f3b45f4c700 1 Processor -- start 2023-11-15T01:49:12.188 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:12.127+0000 7f3b45f4c700 1 -- start start 2023-11-15T01:49:12.188 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:12.127+0000 7f3b45f4c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3b40070430 0x7f3b400fe730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T01:49:12.188 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:12.127+0000 7f3b45f4c700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b400fec20 con 0x7f3b40070430 2023-11-15T01:49:12.188 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:12.127+0000 7f3b44f4a700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3b40070430 0x7f3b400fe730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:49:12.189 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:12.127+0000 7f3b44f4a700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3b40070430 0x7f3b400fe730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:55628/0 (socket says 172.21.15.55:55628) 2023-11-15T01:49:12.189 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:12.127+0000 7f3b44f4a700 1 -- 172.21.15.55:0/143555786 learned_addr learned my addr 172.21.15.55:0/143555786 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-11-15T01:49:12.189 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:12.127+0000 7f3b44f4a700 1 -- 172.21.15.55:0/143555786 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b400fed60 con 0x7f3b40070430 2023-11-15T01:49:12.189 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:12.127+0000 7f3b44f4a700 1 --2- 172.21.15.55:0/143555786 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3b40070430 0x7f3b400fe730 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f3b3400b5f0 tx=0x7f3b34006ee0).ready entity=mon.0 client_cookie=ba17a84fe5fc707 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:49:12.189 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:12.128+0000 7f3b3f7fe700 1 -- 172.21.15.55:0/143555786 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3b3400d600 con 0x7f3b40070430 2023-11-15T01:49:12.190 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:12.128+0000 7f3b3f7fe700 1 -- 172.21.15.55:0/143555786 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f3b3400dbd0 con 0x7f3b40070430 2023-11-15T01:49:12.190 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:12.128+0000 7f3b3f7fe700 1 -- 172.21.15.55:0/143555786 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3b34011c00 con 0x7f3b40070430 2023-11-15T01:49:12.190 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:12.128+0000 7f3b45f4c700 1 -- 172.21.15.55:0/143555786 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3b40070430 msgr2=0x7f3b400fe730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:49:12.190 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:12.128+0000 7f3b45f4c700 1 --2- 172.21.15.55:0/143555786 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3b40070430 0x7f3b400fe730 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f3b3400b5f0 tx=0x7f3b34006ee0).stop 2023-11-15T01:49:12.190 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:12.128+0000 7f3b45f4c700 1 -- 172.21.15.55:0/143555786 shutdown_connections 2023-11-15T01:49:12.191 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:12.128+0000 7f3b45f4c700 1 --2- 172.21.15.55:0/143555786 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3b40070430 0x7f3b400fe730 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:49:12.191 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:12.128+0000 7f3b45f4c700 1 -- 172.21.15.55:0/143555786 >> 172.21.15.55:0/143555786 conn(0x7f3b400f6210 msgr2=0x7f3b400f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:49:12.191 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:12.128+0000 7f3b45f4c700 1 -- 172.21.15.55:0/143555786 shutdown_connections 2023-11-15T01:49:12.191 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:12.128+0000 7f3b45f4c700 1 -- 172.21.15.55:0/143555786 wait complete. 2023-11-15T01:49:12.191 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:12.129+0000 7f3b45f4c700 1 Processor -- start 2023-11-15T01:49:12.192 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:12.129+0000 7f3b45f4c700 1 -- start start 2023-11-15T01:49:12.192 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:12.129+0000 7f3b45f4c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3b40070430 0x7f3b4010a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:49:12.192 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:12.129+0000 7f3b45f4c700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b4010ad20 con 0x7f3b40070430 2023-11-15T01:49:12.192 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:12.129+0000 7f3b44f4a700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3b40070430 0x7f3b4010a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:49:12.192 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:12.129+0000 7f3b44f4a700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3b40070430 0x7f3b4010a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:55632/0 (socket says 172.21.15.55:55632) 2023-11-15T01:49:12.193 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:12.129+0000 7f3b44f4a700 1 -- 172.21.15.55:0/906840680 learned_addr learned my addr 172.21.15.55:0/906840680 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-11-15T01:49:12.193 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:12.130+0000 7f3b44f4a700 1 -- 172.21.15.55:0/906840680 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b3400b200 con 0x7f3b40070430 2023-11-15T01:49:12.193 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:12.130+0000 7f3b44f4a700 1 --2- 172.21.15.55:0/906840680 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3b40070430 0x7f3b4010a830 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f3b3400b190 tx=0x7f3b34006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:49:12.193 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:12.130+0000 7f3b3d7fa700 1 -- 172.21.15.55:0/906840680 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3b3401b410 con 0x7f3b40070430 2023-11-15T01:49:12.193 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:12.130+0000 7f3b3d7fa700 1 -- 172.21.15.55:0/906840680 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f3b3401b9e0 con 0x7f3b40070430 2023-11-15T01:49:12.193 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:12.130+0000 7f3b3d7fa700 1 -- 172.21.15.55:0/906840680 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3b3401a880 con 0x7f3b40070430 2023-11-15T01:49:12.194 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:12.130+0000 7f3b45f4c700 1 -- 172.21.15.55:0/906840680 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b4010af20 con 0x7f3b40070430 2023-11-15T01:49:12.194 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:12.130+0000 7f3b45f4c700 1 -- 172.21.15.55:0/906840680 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b4010b330 con 0x7f3b40070430 2023-11-15T01:49:12.194 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:12.131+0000 7f3b3d7fa700 1 -- 172.21.15.55:0/906840680 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f3b3400f070 con 0x7f3b40070430 2023-11-15T01:49:12.194 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:12.131+0000 7f3b3d7fa700 1 --2- 172.21.15.55:0/906840680 >> [v2:172.21.15.55:6800/1950005071,v1:172.21.15.55:6801/1950005071] conn(0x7f3b2402fd50 0x7f3b24032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:49:12.194 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:12.131+0000 7f3b3d7fa700 1 -- 172.21.15.55:0/906840680 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f3b34041270 con 0x7f3b40070430 2023-11-15T01:49:12.195 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:12.131+0000 7f3b45f4c700 1 -- 172.21.15.55:0/906840680 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b40045ad0 con 0x7f3b40070430 2023-11-15T01:49:12.195 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:12.131+0000 7f3b3ffff700 1 --2- 172.21.15.55:0/906840680 >> [v2:172.21.15.55:6800/1950005071,v1:172.21.15.55:6801/1950005071] conn(0x7f3b2402fd50 0x7f3b24032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:49:12.195 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:12.131+0000 7f3b3ffff700 1 --2- 172.21.15.55:0/906840680 >> [v2:172.21.15.55:6800/1950005071,v1:172.21.15.55:6801/1950005071] conn(0x7f3b2402fd50 0x7f3b24032210 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f3b2c0096a0 tx=0x7f3b2c006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:49:12.195 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:12.136+0000 7f3b3d7fa700 1 -- 172.21.15.55:0/906840680 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3b34016080 con 0x7f3b40070430 2023-11-15T01:49:12.335 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:12.334+0000 7f3b45f4c700 1 -- 172.21.15.55:0/906840680 --> [v2:172.21.15.55:6800/1950005071,v1:172.21.15.55:6801/1950005071] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f3b401047b0 con 0x7f3b2402fd50 2023-11-15T01:49:12.340 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:12.339+0000 7f3b3d7fa700 1 -- 172.21.15.55:0/906840680 <== mgr.14116 v2:172.21.15.55:6800/1950005071 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f3b401047b0 con 0x7f3b2402fd50 2023-11-15T01:49:12.343 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:12.341+0000 7f3b45f4c700 1 -- 172.21.15.55:0/906840680 >> [v2:172.21.15.55:6800/1950005071,v1:172.21.15.55:6801/1950005071] conn(0x7f3b2402fd50 msgr2=0x7f3b24032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:49:12.344 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:12.341+0000 7f3b45f4c700 1 --2- 172.21.15.55:0/906840680 >> [v2:172.21.15.55:6800/1950005071,v1:172.21.15.55:6801/1950005071] conn(0x7f3b2402fd50 0x7f3b24032210 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f3b2c0096a0 tx=0x7f3b2c006b40).stop 2023-11-15T01:49:12.344 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:12.341+0000 7f3b45f4c700 1 -- 172.21.15.55:0/906840680 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3b40070430 msgr2=0x7f3b4010a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:49:12.344 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:12.341+0000 7f3b45f4c700 1 --2- 172.21.15.55:0/906840680 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3b40070430 0x7f3b4010a830 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f3b3400b190 tx=0x7f3b34006ee0).stop 2023-11-15T01:49:12.344 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:12.341+0000 7f3b45f4c700 1 -- 172.21.15.55:0/906840680 shutdown_connections 2023-11-15T01:49:12.344 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:12.341+0000 7f3b45f4c700 1 --2- 172.21.15.55:0/906840680 >> [v2:172.21.15.55:6800/1950005071,v1:172.21.15.55:6801/1950005071] conn(0x7f3b2402fd50 0x7f3b24032210 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:49:12.345 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:12.341+0000 7f3b45f4c700 1 --2- 172.21.15.55:0/906840680 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3b40070430 0x7f3b4010a830 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:49:12.345 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:12.341+0000 7f3b45f4c700 1 -- 172.21.15.55:0/906840680 >> 172.21.15.55:0/906840680 conn(0x7f3b400f6210 msgr2=0x7f3b400f7cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:49:12.345 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:12.341+0000 7f3b45f4c700 1 -- 172.21.15.55:0/906840680 shutdown_connections 2023-11-15T01:49:12.345 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:12.341+0000 7f3b45f4c700 1 -- 172.21.15.55:0/906840680 wait complete. 2023-11-15T01:49:13.652 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:13 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:49:12.336619+0000 mgr.smithi055.szwngw (mgr.14116) 3 : audit [DBG] from='client.14126 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T01:49:13.652 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:13 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:49:12.340773+0000 mon.smithi055 (mon.0) 51 : audit [INF] from='mgr.14116 172.21.15.55:0/3759764227' entity='mgr.smithi055.szwngw' 2023-11-15T01:49:13.653 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:13 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:49:12.344878+0000 mon.smithi055 (mon.0) 52 : audit [DBG] from='mgr.14116 172.21.15.55:0/3759764227' entity='mgr.smithi055.szwngw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-15T01:49:14.444 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi055 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/0cb4fc0c-8359-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpc1yi9jse:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpra3bp1py:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm set-user root 2023-11-15T01:49:15.701 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:15.674+0000 7f2d9055c700 1 Processor -- start 2023-11-15T01:49:15.705 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:15.674+0000 7f2d9055c700 1 -- start start 2023-11-15T01:49:15.705 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:15.674+0000 7f2d9055c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2d880703b0 0x7f2d88100d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T01:49:15.706 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:15.674+0000 7f2d9055c700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d880fc750 con 0x7f2d880703b0 2023-11-15T01:49:15.706 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:15.674+0000 7f2d8e2f8700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2d880703b0 0x7f2d88100d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:49:15.706 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:15.674+0000 7f2d8e2f8700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2d880703b0 0x7f2d88100d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:55638/0 (socket says 172.21.15.55:55638) 2023-11-15T01:49:15.706 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:15.674+0000 7f2d8e2f8700 1 -- 172.21.15.55:0/3935672489 learned_addr learned my addr 172.21.15.55:0/3935672489 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-11-15T01:49:15.706 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:15.675+0000 7f2d8e2f8700 1 -- 172.21.15.55:0/3935672489 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d88101250 con 0x7f2d880703b0 2023-11-15T01:49:15.707 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:15.675+0000 7f2d8e2f8700 1 --2- 172.21.15.55:0/3935672489 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2d880703b0 0x7f2d88100d60 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f2d8400b5f0 tx=0x7f2d84006ee0).ready entity=mon.0 client_cookie=d8169bbf883f71ff server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:49:15.707 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:15.675+0000 7f2d8d2f6700 1 -- 172.21.15.55:0/3935672489 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d8400d600 con 0x7f2d880703b0 2023-11-15T01:49:15.707 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:15.675+0000 7f2d8d2f6700 1 -- 172.21.15.55:0/3935672489 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f2d8400dbd0 con 0x7f2d880703b0 2023-11-15T01:49:15.707 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:15.675+0000 7f2d8d2f6700 1 -- 172.21.15.55:0/3935672489 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d84011c00 con 0x7f2d880703b0 2023-11-15T01:49:15.708 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:15.676+0000 7f2d9055c700 1 -- 172.21.15.55:0/3935672489 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2d880703b0 msgr2=0x7f2d88100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:49:15.708 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:15.676+0000 7f2d9055c700 1 --2- 172.21.15.55:0/3935672489 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2d880703b0 0x7f2d88100d60 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f2d8400b5f0 tx=0x7f2d84006ee0).stop 2023-11-15T01:49:15.708 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:15.676+0000 7f2d9055c700 1 -- 172.21.15.55:0/3935672489 shutdown_connections 2023-11-15T01:49:15.708 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:15.676+0000 7f2d9055c700 1 --2- 172.21.15.55:0/3935672489 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2d880703b0 0x7f2d88100d60 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:49:15.708 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:15.676+0000 7f2d9055c700 1 -- 172.21.15.55:0/3935672489 >> 172.21.15.55:0/3935672489 conn(0x7f2d880f63c0 msgr2=0x7f2d880f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:49:15.709 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:15.676+0000 7f2d9055c700 1 -- 172.21.15.55:0/3935672489 shutdown_connections 2023-11-15T01:49:15.709 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:15.676+0000 7f2d9055c700 1 -- 172.21.15.55:0/3935672489 wait complete. 2023-11-15T01:49:15.709 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:15.676+0000 7f2d9055c700 1 Processor -- start 2023-11-15T01:49:15.709 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:15.676+0000 7f2d9055c700 1 -- start start 2023-11-15T01:49:15.709 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:15.677+0000 7f2d9055c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2d880703b0 0x7f2d8810cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:49:15.710 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:15.677+0000 7f2d9055c700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d8810d100 con 0x7f2d880703b0 2023-11-15T01:49:15.710 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:15.677+0000 7f2d8e2f8700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2d880703b0 0x7f2d8810cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:49:15.710 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:15.677+0000 7f2d8e2f8700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2d880703b0 0x7f2d8810cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:55642/0 (socket says 172.21.15.55:55642) 2023-11-15T01:49:15.710 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:15.677+0000 7f2d8e2f8700 1 -- 172.21.15.55:0/80980541 learned_addr learned my addr 172.21.15.55:0/80980541 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-11-15T01:49:15.710 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:15.677+0000 7f2d8e2f8700 1 -- 172.21.15.55:0/80980541 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d8400b200 con 0x7f2d880703b0 2023-11-15T01:49:15.711 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:15.677+0000 7f2d8e2f8700 1 --2- 172.21.15.55:0/80980541 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2d880703b0 0x7f2d8810cc10 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f2d8400b190 tx=0x7f2d84006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:49:15.711 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:15.677+0000 7f2d76ffd700 1 -- 172.21.15.55:0/80980541 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d8401b410 con 0x7f2d880703b0 2023-11-15T01:49:15.711 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:15.677+0000 7f2d9055c700 1 -- 172.21.15.55:0/80980541 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d8810d300 con 0x7f2d880703b0 2023-11-15T01:49:15.711 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:15.677+0000 7f2d76ffd700 1 -- 172.21.15.55:0/80980541 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f2d8401b9e0 con 0x7f2d880703b0 2023-11-15T01:49:15.711 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:15.677+0000 7f2d76ffd700 1 -- 172.21.15.55:0/80980541 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d8401a880 con 0x7f2d880703b0 2023-11-15T01:49:15.711 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:15.677+0000 7f2d9055c700 1 -- 172.21.15.55:0/80980541 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d8810d710 con 0x7f2d880703b0 2023-11-15T01:49:15.712 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:15.678+0000 7f2d76ffd700 1 -- 172.21.15.55:0/80980541 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f2d8400f070 con 0x7f2d880703b0 2023-11-15T01:49:15.712 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:15.678+0000 7f2d9055c700 1 -- 172.21.15.55:0/80980541 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d88045ad0 con 0x7f2d880703b0 2023-11-15T01:49:15.712 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:15.678+0000 7f2d76ffd700 1 --2- 172.21.15.55:0/80980541 >> [v2:172.21.15.55:6800/1950005071,v1:172.21.15.55:6801/1950005071] conn(0x7f2d7002fd50 0x7f2d70032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:49:15.712 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:15.678+0000 7f2d76ffd700 1 -- 172.21.15.55:0/80980541 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f2d840412b0 con 0x7f2d880703b0 2023-11-15T01:49:15.712 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:15.679+0000 7f2d8daf7700 1 --2- 172.21.15.55:0/80980541 >> [v2:172.21.15.55:6800/1950005071,v1:172.21.15.55:6801/1950005071] conn(0x7f2d7002fd50 0x7f2d70032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:49:15.713 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:15.679+0000 7f2d8daf7700 1 --2- 172.21.15.55:0/80980541 >> [v2:172.21.15.55:6800/1950005071,v1:172.21.15.55:6801/1950005071] conn(0x7f2d7002fd50 0x7f2d70032210 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f2d780096a0 tx=0x7f2d78006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:49:15.713 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:15.683+0000 7f2d76ffd700 1 -- 172.21.15.55:0/80980541 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2d840224f0 con 0x7f2d880703b0 2023-11-15T01:49:15.854 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:15.852+0000 7f2d9055c700 1 -- 172.21.15.55:0/80980541 --> [v2:172.21.15.55:6800/1950005071,v1:172.21.15.55:6801/1950005071] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f2d8810e050 con 0x7f2d7002fd50 2023-11-15T01:49:15.854 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:15.852+0000 7f2d76ffd700 1 -- 172.21.15.55:0/80980541 <== mgr.14116 v2:172.21.15.55:6800/1950005071 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f2d8810e050 con 0x7f2d7002fd50 2023-11-15T01:49:15.855 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: value unchanged 2023-11-15T01:49:15.856 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:15.854+0000 7f2d9055c700 1 -- 172.21.15.55:0/80980541 >> [v2:172.21.15.55:6800/1950005071,v1:172.21.15.55:6801/1950005071] conn(0x7f2d7002fd50 msgr2=0x7f2d70032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:49:15.856 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:15.854+0000 7f2d9055c700 1 --2- 172.21.15.55:0/80980541 >> [v2:172.21.15.55:6800/1950005071,v1:172.21.15.55:6801/1950005071] conn(0x7f2d7002fd50 0x7f2d70032210 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f2d780096a0 tx=0x7f2d78006b40).stop 2023-11-15T01:49:15.856 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:15.854+0000 7f2d9055c700 1 -- 172.21.15.55:0/80980541 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2d880703b0 msgr2=0x7f2d8810cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:49:15.857 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:15.854+0000 7f2d9055c700 1 --2- 172.21.15.55:0/80980541 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2d880703b0 0x7f2d8810cc10 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f2d8400b190 tx=0x7f2d84006ee0).stop 2023-11-15T01:49:15.857 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:15.854+0000 7f2d9055c700 1 -- 172.21.15.55:0/80980541 shutdown_connections 2023-11-15T01:49:15.858 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:15.854+0000 7f2d9055c700 1 --2- 172.21.15.55:0/80980541 >> [v2:172.21.15.55:6800/1950005071,v1:172.21.15.55:6801/1950005071] conn(0x7f2d7002fd50 0x7f2d70032210 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:49:15.858 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:15.854+0000 7f2d9055c700 1 --2- 172.21.15.55:0/80980541 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2d880703b0 0x7f2d8810cc10 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:49:15.858 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:15.854+0000 7f2d9055c700 1 -- 172.21.15.55:0/80980541 >> 172.21.15.55:0/80980541 conn(0x7f2d880f63c0 msgr2=0x7f2d880f7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:49:15.858 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:15.854+0000 7f2d9055c700 1 -- 172.21.15.55:0/80980541 shutdown_connections 2023-11-15T01:49:15.859 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:15.854+0000 7f2d9055c700 1 -- 172.21.15.55:0/80980541 wait complete. 2023-11-15T01:49:16.977 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:16 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:49:15.854323+0000 mgr.smithi055.szwngw (mgr.14116) 4 : audit [DBG] from='client.14128 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T01:49:17.561 INFO:teuthology.orchestra.run.smithi055.stderr:Generating ssh key... 2023-11-15T01:49:17.562 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi055 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/0cb4fc0c-8359-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpc1yi9jse:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpra3bp1py:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm generate-key 2023-11-15T01:49:17.967 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:17.964+0000 7f34fd317700 1 Processor -- start 2023-11-15T01:49:17.968 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:17.964+0000 7f34fd317700 1 -- start start 2023-11-15T01:49:17.968 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:17.964+0000 7f34fd317700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f34f80703b0 0x7f34f8100d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T01:49:17.968 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:17.965+0000 7f34fd317700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34f80fc750 con 0x7f34f80703b0 2023-11-15T01:49:17.968 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:17.965+0000 7f34f6ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f34f80703b0 0x7f34f8100d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:49:17.968 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:17.965+0000 7f34f6ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f34f80703b0 0x7f34f8100d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:55652/0 (socket says 172.21.15.55:55652) 2023-11-15T01:49:17.969 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:17.965+0000 7f34f6ffd700 1 -- 172.21.15.55:0/322167060 learned_addr learned my addr 172.21.15.55:0/322167060 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-11-15T01:49:17.969 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:17.965+0000 7f34f6ffd700 1 -- 172.21.15.55:0/322167060 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34f8101250 con 0x7f34f80703b0 2023-11-15T01:49:17.969 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:17.965+0000 7f34f6ffd700 1 --2- 172.21.15.55:0/322167060 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f34f80703b0 0x7f34f8100d60 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f34e800b5f0 tx=0x7f34e8006ee0).ready entity=mon.0 client_cookie=dc2c63075b84e900 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:49:17.969 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:17.965+0000 7f34f67fc700 1 -- 172.21.15.55:0/322167060 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f34e800d600 con 0x7f34f80703b0 2023-11-15T01:49:17.969 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:17.965+0000 7f34f67fc700 1 -- 172.21.15.55:0/322167060 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f34e800dbd0 con 0x7f34f80703b0 2023-11-15T01:49:17.969 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:17.965+0000 7f34f67fc700 1 -- 172.21.15.55:0/322167060 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f34e8011c00 con 0x7f34f80703b0 2023-11-15T01:49:17.972 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:17.966+0000 7f34fd317700 1 -- 172.21.15.55:0/322167060 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f34f80703b0 msgr2=0x7f34f8100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:49:17.972 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:17.966+0000 7f34fd317700 1 --2- 172.21.15.55:0/322167060 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f34f80703b0 0x7f34f8100d60 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f34e800b5f0 tx=0x7f34e8006ee0).stop 2023-11-15T01:49:17.972 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:17.966+0000 7f34fd317700 1 -- 172.21.15.55:0/322167060 shutdown_connections 2023-11-15T01:49:17.973 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:17.966+0000 7f34fd317700 1 --2- 172.21.15.55:0/322167060 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f34f80703b0 0x7f34f8100d60 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:49:17.973 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:17.966+0000 7f34fd317700 1 -- 172.21.15.55:0/322167060 >> 172.21.15.55:0/322167060 conn(0x7f34f80f63c0 msgr2=0x7f34f80f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:49:17.973 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:17.966+0000 7f34fd317700 1 -- 172.21.15.55:0/322167060 shutdown_connections 2023-11-15T01:49:17.973 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:17.966+0000 7f34fd317700 1 -- 172.21.15.55:0/322167060 wait complete. 2023-11-15T01:49:17.973 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:17.967+0000 7f34fd317700 1 Processor -- start 2023-11-15T01:49:17.973 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:17.967+0000 7f34fd317700 1 -- start start 2023-11-15T01:49:17.974 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:17.967+0000 7f34fd317700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f34f80703b0 0x7f34f810cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:49:17.974 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:17.967+0000 7f34fd317700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34f810d100 con 0x7f34f80703b0 2023-11-15T01:49:17.974 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:17.967+0000 7f34f6ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f34f80703b0 0x7f34f810cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:49:17.974 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:17.967+0000 7f34f6ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f34f80703b0 0x7f34f810cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:55668/0 (socket says 172.21.15.55:55668) 2023-11-15T01:49:17.974 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:17.967+0000 7f34f6ffd700 1 -- 172.21.15.55:0/2945228095 learned_addr learned my addr 172.21.15.55:0/2945228095 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-11-15T01:49:17.975 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:17.968+0000 7f34f6ffd700 1 -- 172.21.15.55:0/2945228095 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34e800b200 con 0x7f34f80703b0 2023-11-15T01:49:17.975 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:17.968+0000 7f34f6ffd700 1 --2- 172.21.15.55:0/2945228095 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f34f80703b0 0x7f34f810cc10 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f34e800b190 tx=0x7f34e8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:49:17.975 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:17.968+0000 7f34effff700 1 -- 172.21.15.55:0/2945228095 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f34e801b410 con 0x7f34f80703b0 2023-11-15T01:49:17.975 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:17.968+0000 7f34effff700 1 -- 172.21.15.55:0/2945228095 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f34e801b9e0 con 0x7f34f80703b0 2023-11-15T01:49:17.975 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:17.968+0000 7f34fd317700 1 -- 172.21.15.55:0/2945228095 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34f810d300 con 0x7f34f80703b0 2023-11-15T01:49:17.975 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:17.968+0000 7f34effff700 1 -- 172.21.15.55:0/2945228095 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f34e801a880 con 0x7f34f80703b0 2023-11-15T01:49:17.976 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:17.968+0000 7f34fd317700 1 -- 172.21.15.55:0/2945228095 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34f810d710 con 0x7f34f80703b0 2023-11-15T01:49:17.976 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:17.969+0000 7f34effff700 1 -- 172.21.15.55:0/2945228095 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f34e800f070 con 0x7f34f80703b0 2023-11-15T01:49:17.976 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:17.969+0000 7f34effff700 1 --2- 172.21.15.55:0/2945228095 >> [v2:172.21.15.55:6800/1950005071,v1:172.21.15.55:6801/1950005071] conn(0x7f34d802fd50 0x7f34d8032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:49:17.976 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:17.969+0000 7f34effff700 1 -- 172.21.15.55:0/2945228095 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f34e8041290 con 0x7f34f80703b0 2023-11-15T01:49:17.976 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:17.969+0000 7f34fd317700 1 -- 172.21.15.55:0/2945228095 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34f8045ad0 con 0x7f34f80703b0 2023-11-15T01:49:17.976 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:17.970+0000 7f34ee7fc700 1 --2- 172.21.15.55:0/2945228095 >> [v2:172.21.15.55:6800/1950005071,v1:172.21.15.55:6801/1950005071] conn(0x7f34d802fd50 0x7f34d8032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:49:17.977 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:17.970+0000 7f34ee7fc700 1 --2- 172.21.15.55:0/2945228095 >> [v2:172.21.15.55:6800/1950005071,v1:172.21.15.55:6801/1950005071] conn(0x7f34d802fd50 0x7f34d8032210 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f34e00096a0 tx=0x7f34e0006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:49:17.977 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:17.974+0000 7f34effff700 1 -- 172.21.15.55:0/2945228095 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f34e8016080 con 0x7f34f80703b0 2023-11-15T01:49:18.142 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:18.141+0000 7f34fd317700 1 -- 172.21.15.55:0/2945228095 --> [v2:172.21.15.55:6800/1950005071,v1:172.21.15.55:6801/1950005071] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f34f8106ae0 con 0x7f34d802fd50 2023-11-15T01:49:18.227 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:18.226+0000 7f34effff700 1 -- 172.21.15.55:0/2945228095 <== mgr.14116 v2:172.21.15.55:6800/1950005071 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f34f8106ae0 con 0x7f34d802fd50 2023-11-15T01:49:18.230 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:18.227+0000 7f34fd317700 1 -- 172.21.15.55:0/2945228095 >> [v2:172.21.15.55:6800/1950005071,v1:172.21.15.55:6801/1950005071] conn(0x7f34d802fd50 msgr2=0x7f34d8032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:49:18.230 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:18.228+0000 7f34fd317700 1 --2- 172.21.15.55:0/2945228095 >> [v2:172.21.15.55:6800/1950005071,v1:172.21.15.55:6801/1950005071] conn(0x7f34d802fd50 0x7f34d8032210 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f34e00096a0 tx=0x7f34e0006b40).stop 2023-11-15T01:49:18.230 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:18.228+0000 7f34fd317700 1 -- 172.21.15.55:0/2945228095 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f34f80703b0 msgr2=0x7f34f810cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:49:18.230 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:18.228+0000 7f34fd317700 1 --2- 172.21.15.55:0/2945228095 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f34f80703b0 0x7f34f810cc10 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f34e800b190 tx=0x7f34e8006ee0).stop 2023-11-15T01:49:18.231 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:18.228+0000 7f34fd317700 1 -- 172.21.15.55:0/2945228095 shutdown_connections 2023-11-15T01:49:18.231 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:18.228+0000 7f34fd317700 1 --2- 172.21.15.55:0/2945228095 >> [v2:172.21.15.55:6800/1950005071,v1:172.21.15.55:6801/1950005071] conn(0x7f34d802fd50 0x7f34d8032210 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:49:18.231 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:18.228+0000 7f34fd317700 1 --2- 172.21.15.55:0/2945228095 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f34f80703b0 0x7f34f810cc10 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:49:18.231 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:18.228+0000 7f34fd317700 1 -- 172.21.15.55:0/2945228095 >> 172.21.15.55:0/2945228095 conn(0x7f34f80f63c0 msgr2=0x7f34f80f7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:49:18.232 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:18.229+0000 7f34fd317700 1 -- 172.21.15.55:0/2945228095 shutdown_connections 2023-11-15T01:49:18.232 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:18.229+0000 7f34fd317700 1 -- 172.21.15.55:0/2945228095 wait complete. 2023-11-15T01:49:18.594 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi055 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/0cb4fc0c-8359-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpc1yi9jse:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpra3bp1py:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm get-pub-key 2023-11-15T01:49:18.953 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:18.951+0000 7f0ff0c94700 1 Processor -- start 2023-11-15T01:49:18.954 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:18.951+0000 7f0ff0c94700 1 -- start start 2023-11-15T01:49:18.954 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:18.952+0000 7f0ff0c94700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0fec0703b0 0x7f0fec100d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T01:49:18.954 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:18.952+0000 7f0ff0c94700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fec0fc750 con 0x7f0fec0703b0 2023-11-15T01:49:18.957 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:18.952+0000 7f0fea59c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0fec0703b0 0x7f0fec100d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:49:18.957 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:18.952+0000 7f0fea59c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0fec0703b0 0x7f0fec100d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:55684/0 (socket says 172.21.15.55:55684) 2023-11-15T01:49:18.958 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:18.952+0000 7f0fea59c700 1 -- 172.21.15.55:0/2469367918 learned_addr learned my addr 172.21.15.55:0/2469367918 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-11-15T01:49:18.958 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:18.952+0000 7f0fea59c700 1 -- 172.21.15.55:0/2469367918 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fec101250 con 0x7f0fec0703b0 2023-11-15T01:49:18.958 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:18.952+0000 7f0fea59c700 1 --2- 172.21.15.55:0/2469367918 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0fec0703b0 0x7f0fec100d60 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f0fdc00b5f0 tx=0x7f0fdc006ee0).ready entity=mon.0 client_cookie=b6385290fe8ad904 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:49:18.958 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:18.952+0000 7f0fe9d9b700 1 -- 172.21.15.55:0/2469367918 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0fdc00d600 con 0x7f0fec0703b0 2023-11-15T01:49:18.958 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:18.952+0000 7f0fe9d9b700 1 -- 172.21.15.55:0/2469367918 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f0fdc00dbd0 con 0x7f0fec0703b0 2023-11-15T01:49:18.958 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:18.952+0000 7f0fe9d9b700 1 -- 172.21.15.55:0/2469367918 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0fdc011c00 con 0x7f0fec0703b0 2023-11-15T01:49:18.959 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:18.953+0000 7f0ff0c94700 1 -- 172.21.15.55:0/2469367918 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0fec0703b0 msgr2=0x7f0fec100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:49:18.959 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:18.953+0000 7f0ff0c94700 1 --2- 172.21.15.55:0/2469367918 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0fec0703b0 0x7f0fec100d60 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f0fdc00b5f0 tx=0x7f0fdc006ee0).stop 2023-11-15T01:49:18.959 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:18.953+0000 7f0ff0c94700 1 -- 172.21.15.55:0/2469367918 shutdown_connections 2023-11-15T01:49:18.959 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:18.953+0000 7f0ff0c94700 1 --2- 172.21.15.55:0/2469367918 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0fec0703b0 0x7f0fec100d60 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:49:18.959 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:18.953+0000 7f0ff0c94700 1 -- 172.21.15.55:0/2469367918 >> 172.21.15.55:0/2469367918 conn(0x7f0fec0f63c0 msgr2=0x7f0fec0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:49:18.960 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:18.953+0000 7f0ff0c94700 1 -- 172.21.15.55:0/2469367918 shutdown_connections 2023-11-15T01:49:18.960 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:18.953+0000 7f0ff0c94700 1 -- 172.21.15.55:0/2469367918 wait complete. 2023-11-15T01:49:18.960 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:18.954+0000 7f0ff0c94700 1 Processor -- start 2023-11-15T01:49:18.960 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:18.954+0000 7f0ff0c94700 1 -- start start 2023-11-15T01:49:18.960 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:18.954+0000 7f0ff0c94700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0fec0703b0 0x7f0fec10cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:49:18.960 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:18.954+0000 7f0ff0c94700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fec10d100 con 0x7f0fec0703b0 2023-11-15T01:49:18.961 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:18.954+0000 7f0fea59c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0fec0703b0 0x7f0fec10cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:49:18.961 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:18.954+0000 7f0fea59c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0fec0703b0 0x7f0fec10cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:55694/0 (socket says 172.21.15.55:55694) 2023-11-15T01:49:18.961 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:18.954+0000 7f0fea59c700 1 -- 172.21.15.55:0/3741808087 learned_addr learned my addr 172.21.15.55:0/3741808087 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-11-15T01:49:18.961 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:18.955+0000 7f0fea59c700 1 -- 172.21.15.55:0/3741808087 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fdc00b200 con 0x7f0fec0703b0 2023-11-15T01:49:18.961 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:18.955+0000 7f0fea59c700 1 --2- 172.21.15.55:0/3741808087 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0fec0703b0 0x7f0fec10cc10 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f0fdc00b190 tx=0x7f0fdc006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:49:18.962 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:18.955+0000 7f0fe37fe700 1 -- 172.21.15.55:0/3741808087 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0fdc01b410 con 0x7f0fec0703b0 2023-11-15T01:49:18.962 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:18.955+0000 7f0fe37fe700 1 -- 172.21.15.55:0/3741808087 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f0fdc01b9e0 con 0x7f0fec0703b0 2023-11-15T01:49:18.962 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:18.955+0000 7f0ff0c94700 1 -- 172.21.15.55:0/3741808087 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fec10d300 con 0x7f0fec0703b0 2023-11-15T01:49:18.962 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:18.955+0000 7f0fe37fe700 1 -- 172.21.15.55:0/3741808087 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0fdc01a880 con 0x7f0fec0703b0 2023-11-15T01:49:18.962 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:18.955+0000 7f0ff0c94700 1 -- 172.21.15.55:0/3741808087 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fec10d710 con 0x7f0fec0703b0 2023-11-15T01:49:18.963 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:18.956+0000 7f0ff0c94700 1 -- 172.21.15.55:0/3741808087 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fec045ad0 con 0x7f0fec0703b0 2023-11-15T01:49:18.963 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:18.957+0000 7f0fe37fe700 1 -- 172.21.15.55:0/3741808087 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f0fdc00f070 con 0x7f0fec0703b0 2023-11-15T01:49:18.964 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:18.957+0000 7f0fe37fe700 1 --2- 172.21.15.55:0/3741808087 >> [v2:172.21.15.55:6800/1950005071,v1:172.21.15.55:6801/1950005071] conn(0x7f0fcc02fdb0 0x7f0fcc032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:49:18.964 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:18.957+0000 7f0fe37fe700 1 -- 172.21.15.55:0/3741808087 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f0fdc041cd0 con 0x7f0fec0703b0 2023-11-15T01:49:18.964 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:18.957+0000 7f0fe1d9b700 1 --2- 172.21.15.55:0/3741808087 >> [v2:172.21.15.55:6800/1950005071,v1:172.21.15.55:6801/1950005071] conn(0x7f0fcc02fdb0 0x7f0fcc032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:49:18.964 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:18.957+0000 7f0fe1d9b700 1 --2- 172.21.15.55:0/3741808087 >> [v2:172.21.15.55:6800/1950005071,v1:172.21.15.55:6801/1950005071] conn(0x7f0fcc02fdb0 0x7f0fcc032270 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f0fd40096a0 tx=0x7f0fd4006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:49:18.964 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:18.961+0000 7f0fe37fe700 1 -- 172.21.15.55:0/3741808087 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0fdc02db70 con 0x7f0fec0703b0 2023-11-15T01:49:19.129 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:19.127+0000 7f0ff0c94700 1 -- 172.21.15.55:0/3741808087 --> [v2:172.21.15.55:6800/1950005071,v1:172.21.15.55:6801/1950005071] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f0fec10e050 con 0x7f0fcc02fdb0 2023-11-15T01:49:19.130 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:19.127+0000 7f0fe37fe700 1 -- 172.21.15.55:0/3741808087 <== mgr.14116 v2:172.21.15.55:6800/1950005071 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f0fec10e050 con 0x7f0fcc02fdb0 2023-11-15T01:49:19.130 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCxLJt0GaYl6FwRC+2jujyNfcjnFqDoIxZQt7xyvegv6X8cybULkGSYLvK2zO64L7UGRiP49oFJW7nNqZAhOpR5bTmfY1tfgvHj1I9aS541zHVvC1lr206T7ujPgW3N8/P0usLiIVeb3VwxdxL8LBQvx1YoULbTGxIzaxDCg9pRgf4pDURazpQxmxZ1XcbjkhKesOBEBAqPfibyGsLpAZTgkw4JGXS86QoKGCgEQ8/A/eN0h7UlAZoly3kFEqX5Owfb55y+YVEvZJ0TSRfxwzhHbwOmZy8cCVWFnsvj7rnzqIPNmcVwLz1jr9mXN/+wiaPm1r9XGg2OPRnEyd4YX7JreBaUSwX5vFW5zzvYLpuPHh3+yGTvzYZ/7VZ44LO1EFApLv229iASMFSG6kThktQM6Uk3/gZMKhDTOX6FHnG8RK5KtO0mF2B9wcj36FafX+DUkyVahKm6IsamAsiC29trMQrU8VQKuCduq8SRObbLbz1L1+2vXAOJoYgXXLAl8zE= ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 2023-11-15T01:49:19.132 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:19.129+0000 7f0ff0c94700 1 -- 172.21.15.55:0/3741808087 >> [v2:172.21.15.55:6800/1950005071,v1:172.21.15.55:6801/1950005071] conn(0x7f0fcc02fdb0 msgr2=0x7f0fcc032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:49:19.132 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:19.129+0000 7f0ff0c94700 1 --2- 172.21.15.55:0/3741808087 >> [v2:172.21.15.55:6800/1950005071,v1:172.21.15.55:6801/1950005071] conn(0x7f0fcc02fdb0 0x7f0fcc032270 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f0fd40096a0 tx=0x7f0fd4006b40).stop 2023-11-15T01:49:19.132 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:19.129+0000 7f0ff0c94700 1 -- 172.21.15.55:0/3741808087 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0fec0703b0 msgr2=0x7f0fec10cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:49:19.132 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:19.129+0000 7f0ff0c94700 1 --2- 172.21.15.55:0/3741808087 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0fec0703b0 0x7f0fec10cc10 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f0fdc00b190 tx=0x7f0fdc006ee0).stop 2023-11-15T01:49:19.133 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:19.129+0000 7f0ff0c94700 1 -- 172.21.15.55:0/3741808087 shutdown_connections 2023-11-15T01:49:19.133 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:19.129+0000 7f0ff0c94700 1 --2- 172.21.15.55:0/3741808087 >> [v2:172.21.15.55:6800/1950005071,v1:172.21.15.55:6801/1950005071] conn(0x7f0fcc02fdb0 0x7f0fcc032270 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:49:19.133 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:19.129+0000 7f0ff0c94700 1 --2- 172.21.15.55:0/3741808087 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0fec0703b0 0x7f0fec10cc10 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:49:19.133 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:19.129+0000 7f0ff0c94700 1 -- 172.21.15.55:0/3741808087 >> 172.21.15.55:0/3741808087 conn(0x7f0fec0f63c0 msgr2=0x7f0fec0f7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:49:19.134 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:19.129+0000 7f0ff0c94700 1 -- 172.21.15.55:0/3741808087 shutdown_connections 2023-11-15T01:49:19.134 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:19.129+0000 7f0ff0c94700 1 -- 172.21.15.55:0/3741808087 wait complete. 2023-11-15T01:49:19.422 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:19 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cephadm 2023-11-15T01:49:18.143362+0000 mgr.smithi055.szwngw (mgr.14116) 6 : cephadm [INF] Generating ssh key... 2023-11-15T01:49:19.423 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:19 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:49:18.222463+0000 mon.smithi055 (mon.0) 53 : audit [INF] from='mgr.14116 172.21.15.55:0/3759764227' entity='mgr.smithi055.szwngw' 2023-11-15T01:49:19.423 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:19 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:49:18.226827+0000 mon.smithi055 (mon.0) 54 : audit [INF] from='mgr.14116 172.21.15.55:0/3759764227' entity='mgr.smithi055.szwngw' 2023-11-15T01:49:19.478 INFO:teuthology.orchestra.run.smithi055.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2023-11-15T01:49:19.478 INFO:teuthology.orchestra.run.smithi055.stderr:Adding key to root@localhost authorized_keys... 2023-11-15T01:49:19.479 INFO:teuthology.orchestra.run.smithi055.stderr:Adding host smithi055... 2023-11-15T01:49:19.479 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi055 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/0cb4fc0c-8359-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpc1yi9jse:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpra3bp1py:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch host add smithi055 172.21.15.55 2023-11-15T01:49:19.862 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:19.860+0000 7f1afb4ea700 1 Processor -- start 2023-11-15T01:49:19.863 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:19.860+0000 7f1afb4ea700 1 -- start start 2023-11-15T01:49:19.863 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:19.860+0000 7f1afb4ea700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1af40703b0 0x7f1af4100bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T01:49:19.863 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:19.860+0000 7f1afb4ea700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1af40fc5a0 con 0x7f1af40703b0 2023-11-15T01:49:19.863 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:19.860+0000 7f1afa4e8700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1af40703b0 0x7f1af4100bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:49:19.864 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:19.860+0000 7f1afa4e8700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1af40703b0 0x7f1af4100bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:53116/0 (socket says 172.21.15.55:53116) 2023-11-15T01:49:19.864 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:19.860+0000 7f1afa4e8700 1 -- 172.21.15.55:0/1488206249 learned_addr learned my addr 172.21.15.55:0/1488206249 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-11-15T01:49:19.865 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:19.861+0000 7f1afa4e8700 1 -- 172.21.15.55:0/1488206249 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1af41010a0 con 0x7f1af40703b0 2023-11-15T01:49:19.866 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:19.861+0000 7f1afa4e8700 1 --2- 172.21.15.55:0/1488206249 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1af40703b0 0x7f1af4100bb0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f1af000b5f0 tx=0x7f1af0006ee0).ready entity=mon.0 client_cookie=679bfb5ac4b7aa2e server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:49:19.866 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:19.861+0000 7f1af94e6700 1 -- 172.21.15.55:0/1488206249 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1af000d600 con 0x7f1af40703b0 2023-11-15T01:49:19.866 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:19.861+0000 7f1af94e6700 1 -- 172.21.15.55:0/1488206249 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f1af000dbd0 con 0x7f1af40703b0 2023-11-15T01:49:19.866 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:19.861+0000 7f1af94e6700 1 -- 172.21.15.55:0/1488206249 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1af0011c00 con 0x7f1af40703b0 2023-11-15T01:49:19.866 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:19.862+0000 7f1afb4ea700 1 -- 172.21.15.55:0/1488206249 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1af40703b0 msgr2=0x7f1af4100bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:49:19.867 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:19.862+0000 7f1afb4ea700 1 --2- 172.21.15.55:0/1488206249 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1af40703b0 0x7f1af4100bb0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f1af000b5f0 tx=0x7f1af0006ee0).stop 2023-11-15T01:49:19.867 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:19.862+0000 7f1afb4ea700 1 -- 172.21.15.55:0/1488206249 shutdown_connections 2023-11-15T01:49:19.867 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:19.862+0000 7f1afb4ea700 1 --2- 172.21.15.55:0/1488206249 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1af40703b0 0x7f1af4100bb0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:49:19.867 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:19.862+0000 7f1afb4ea700 1 -- 172.21.15.55:0/1488206249 >> 172.21.15.55:0/1488206249 conn(0x7f1af40f6210 msgr2=0x7f1af40f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:49:19.868 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:19.862+0000 7f1afb4ea700 1 -- 172.21.15.55:0/1488206249 shutdown_connections 2023-11-15T01:49:19.868 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:19.862+0000 7f1afb4ea700 1 -- 172.21.15.55:0/1488206249 wait complete. 2023-11-15T01:49:19.868 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:19.863+0000 7f1afb4ea700 1 Processor -- start 2023-11-15T01:49:19.868 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:19.863+0000 7f1afb4ea700 1 -- start start 2023-11-15T01:49:19.868 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:19.863+0000 7f1afb4ea700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1af40703b0 0x7f1af410ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:49:19.870 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:19.863+0000 7f1afb4ea700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1af410cf60 con 0x7f1af40703b0 2023-11-15T01:49:19.870 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:19.863+0000 7f1afa4e8700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1af40703b0 0x7f1af410ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:49:19.870 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:19.863+0000 7f1afa4e8700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1af40703b0 0x7f1af410ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:53124/0 (socket says 172.21.15.55:53124) 2023-11-15T01:49:19.870 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:19.863+0000 7f1afa4e8700 1 -- 172.21.15.55:0/1481403661 learned_addr learned my addr 172.21.15.55:0/1481403661 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-11-15T01:49:19.871 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:19.863+0000 7f1afa4e8700 1 -- 172.21.15.55:0/1481403661 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1af000b200 con 0x7f1af40703b0 2023-11-15T01:49:19.871 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:19.863+0000 7f1afa4e8700 1 --2- 172.21.15.55:0/1481403661 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1af40703b0 0x7f1af410ca70 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f1af000e040 tx=0x7f1af0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:49:19.871 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:19.864+0000 7f1ae2ffd700 1 -- 172.21.15.55:0/1481403661 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1af000d9f0 con 0x7f1af40703b0 2023-11-15T01:49:19.871 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:19.864+0000 7f1ae2ffd700 1 -- 172.21.15.55:0/1481403661 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f1af001a9f0 con 0x7f1af40703b0 2023-11-15T01:49:19.871 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:19.864+0000 7f1afb4ea700 1 -- 172.21.15.55:0/1481403661 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1af410d160 con 0x7f1af40703b0 2023-11-15T01:49:19.872 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:19.864+0000 7f1ae2ffd700 1 -- 172.21.15.55:0/1481403661 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1af001b8c0 con 0x7f1af40703b0 2023-11-15T01:49:19.872 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:19.864+0000 7f1afb4ea700 1 -- 172.21.15.55:0/1481403661 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1af410d570 con 0x7f1af40703b0 2023-11-15T01:49:19.872 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:19.865+0000 7f1ae2ffd700 1 -- 172.21.15.55:0/1481403661 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f1af000f070 con 0x7f1af40703b0 2023-11-15T01:49:19.872 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:19.865+0000 7f1ae2ffd700 1 --2- 172.21.15.55:0/1481403661 >> [v2:172.21.15.55:6800/1950005071,v1:172.21.15.55:6801/1950005071] conn(0x7f1adc02fdb0 0x7f1adc032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:49:19.872 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:19.865+0000 7f1ae2ffd700 1 -- 172.21.15.55:0/1481403661 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f1af00412c0 con 0x7f1af40703b0 2023-11-15T01:49:19.873 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:19.865+0000 7f1af9ce7700 1 --2- 172.21.15.55:0/1481403661 >> [v2:172.21.15.55:6800/1950005071,v1:172.21.15.55:6801/1950005071] conn(0x7f1adc02fdb0 0x7f1adc032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:49:19.873 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:19.865+0000 7f1afb4ea700 1 -- 172.21.15.55:0/1481403661 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1af4059100 con 0x7f1af40703b0 2023-11-15T01:49:19.873 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:19.865+0000 7f1af9ce7700 1 --2- 172.21.15.55:0/1481403661 >> [v2:172.21.15.55:6800/1950005071,v1:172.21.15.55:6801/1950005071] conn(0x7f1adc02fdb0 0x7f1adc032270 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f1ae40096a0 tx=0x7f1ae4006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:49:19.874 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:19.869+0000 7f1ae2ffd700 1 -- 172.21.15.55:0/1481403661 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1af0016070 con 0x7f1af40703b0 2023-11-15T01:49:20.060 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:20.059+0000 7f1afb4ea700 1 -- 172.21.15.55:0/1481403661 --> [v2:172.21.15.55:6800/1950005071,v1:172.21.15.55:6801/1950005071] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi055", "addr": "172.21.15.55", "target": ["mon-mgr", ""]}) v1 -- 0x7f1af41068c0 con 0x7f1adc02fdb0 2023-11-15T01:49:20.578 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:20 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:49:19.129336+0000 mgr.smithi055.szwngw (mgr.14116) 7 : audit [DBG] from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T01:49:20.714 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: Added host 'smithi055' 2023-11-15T01:49:20.715 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:20.712+0000 7f1ae2ffd700 1 -- 172.21.15.55:0/1481403661 <== mgr.14116 v2:172.21.15.55:6800/1950005071 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7f1af41068c0 con 0x7f1adc02fdb0 2023-11-15T01:49:20.716 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:20.714+0000 7f1afb4ea700 1 -- 172.21.15.55:0/1481403661 >> [v2:172.21.15.55:6800/1950005071,v1:172.21.15.55:6801/1950005071] conn(0x7f1adc02fdb0 msgr2=0x7f1adc032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:49:20.717 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:20.714+0000 7f1afb4ea700 1 --2- 172.21.15.55:0/1481403661 >> [v2:172.21.15.55:6800/1950005071,v1:172.21.15.55:6801/1950005071] conn(0x7f1adc02fdb0 0x7f1adc032270 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f1ae40096a0 tx=0x7f1ae4006b40).stop 2023-11-15T01:49:20.717 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:20.714+0000 7f1afb4ea700 1 -- 172.21.15.55:0/1481403661 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1af40703b0 msgr2=0x7f1af410ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:49:20.717 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:20.714+0000 7f1afb4ea700 1 --2- 172.21.15.55:0/1481403661 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1af40703b0 0x7f1af410ca70 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f1af000e040 tx=0x7f1af0006ee0).stop 2023-11-15T01:49:20.717 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:20.714+0000 7f1afb4ea700 1 -- 172.21.15.55:0/1481403661 shutdown_connections 2023-11-15T01:49:20.718 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:20.714+0000 7f1afb4ea700 1 --2- 172.21.15.55:0/1481403661 >> [v2:172.21.15.55:6800/1950005071,v1:172.21.15.55:6801/1950005071] conn(0x7f1adc02fdb0 0x7f1adc032270 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:49:20.718 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:20.714+0000 7f1afb4ea700 1 --2- 172.21.15.55:0/1481403661 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1af40703b0 0x7f1af410ca70 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:49:20.718 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:20.714+0000 7f1afb4ea700 1 -- 172.21.15.55:0/1481403661 >> 172.21.15.55:0/1481403661 conn(0x7f1af40f6210 msgr2=0x7f1af40f7d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:49:20.718 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:20.715+0000 7f1afb4ea700 1 -- 172.21.15.55:0/1481403661 shutdown_connections 2023-11-15T01:49:20.718 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:20.715+0000 7f1afb4ea700 1 -- 172.21.15.55:0/1481403661 wait complete. 2023-11-15T01:49:21.027 INFO:teuthology.orchestra.run.smithi055.stderr:Deploying mon service with default placement... 2023-11-15T01:49:21.027 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi055 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/0cb4fc0c-8359-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpc1yi9jse:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpra3bp1py:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mon 2023-11-15T01:49:21.384 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:21.381+0000 7f490635d700 1 Processor -- start 2023-11-15T01:49:21.385 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:21.381+0000 7f490635d700 1 -- start start 2023-11-15T01:49:21.385 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:21.382+0000 7f490635d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f49000703d0 0x7f49000fe940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T01:49:21.385 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:21.382+0000 7f490635d700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49000fee30 con 0x7f49000703d0 2023-11-15T01:49:21.385 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:21.382+0000 7f48fffff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f49000703d0 0x7f49000fe940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:49:21.385 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:21.382+0000 7f48fffff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f49000703d0 0x7f49000fe940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:53128/0 (socket says 172.21.15.55:53128) 2023-11-15T01:49:21.386 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:21.382+0000 7f48fffff700 1 -- 172.21.15.55:0/2861356484 learned_addr learned my addr 172.21.15.55:0/2861356484 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-11-15T01:49:21.386 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:21.382+0000 7f48fffff700 1 -- 172.21.15.55:0/2861356484 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49000fef70 con 0x7f49000703d0 2023-11-15T01:49:21.389 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:21.382+0000 7f48fffff700 1 --2- 172.21.15.55:0/2861356484 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f49000703d0 0x7f49000fe940 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f48f000b5f0 tx=0x7f48f0006ee0).ready entity=mon.0 client_cookie=7ae43106cb0e8753 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:49:21.389 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:21.383+0000 7f48feffd700 1 -- 172.21.15.55:0/2861356484 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f48f000d600 con 0x7f49000703d0 2023-11-15T01:49:21.389 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:21.383+0000 7f48feffd700 1 -- 172.21.15.55:0/2861356484 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f48f000dbd0 con 0x7f49000703d0 2023-11-15T01:49:21.389 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:21.383+0000 7f48feffd700 1 -- 172.21.15.55:0/2861356484 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f48f0011c00 con 0x7f49000703d0 2023-11-15T01:49:21.390 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:21.383+0000 7f490635d700 1 -- 172.21.15.55:0/2861356484 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f49000703d0 msgr2=0x7f49000fe940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:49:21.390 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:21.383+0000 7f490635d700 1 --2- 172.21.15.55:0/2861356484 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f49000703d0 0x7f49000fe940 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f48f000b5f0 tx=0x7f48f0006ee0).stop 2023-11-15T01:49:21.390 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:21.383+0000 7f490635d700 1 -- 172.21.15.55:0/2861356484 shutdown_connections 2023-11-15T01:49:21.390 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:21.383+0000 7f490635d700 1 --2- 172.21.15.55:0/2861356484 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f49000703d0 0x7f49000fe940 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:49:21.390 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:21.383+0000 7f490635d700 1 -- 172.21.15.55:0/2861356484 >> 172.21.15.55:0/2861356484 conn(0x7f49000f63c0 msgr2=0x7f49000f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:49:21.391 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:21.383+0000 7f490635d700 1 -- 172.21.15.55:0/2861356484 shutdown_connections 2023-11-15T01:49:21.391 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:21.383+0000 7f490635d700 1 -- 172.21.15.55:0/2861356484 wait complete. 2023-11-15T01:49:21.391 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:21.384+0000 7f490635d700 1 Processor -- start 2023-11-15T01:49:21.391 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:21.384+0000 7f490635d700 1 -- start start 2023-11-15T01:49:21.391 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:21.384+0000 7f490635d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f49000703d0 0x7f490010aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:49:21.392 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:21.384+0000 7f490635d700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f490010af20 con 0x7f49000703d0 2023-11-15T01:49:21.392 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:21.385+0000 7f48fffff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f49000703d0 0x7f490010aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:49:21.392 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:21.385+0000 7f48fffff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f49000703d0 0x7f490010aa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:53132/0 (socket says 172.21.15.55:53132) 2023-11-15T01:49:21.392 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:21.385+0000 7f48fffff700 1 -- 172.21.15.55:0/3213616498 learned_addr learned my addr 172.21.15.55:0/3213616498 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-11-15T01:49:21.392 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:21.385+0000 7f48fffff700 1 -- 172.21.15.55:0/3213616498 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48f000b200 con 0x7f49000703d0 2023-11-15T01:49:21.392 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:21.385+0000 7f48fffff700 1 --2- 172.21.15.55:0/3213616498 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f49000703d0 0x7f490010aa30 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f48f000e040 tx=0x7f48f0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:49:21.393 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:21.385+0000 7f48fcff9700 1 -- 172.21.15.55:0/3213616498 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f48f000d9f0 con 0x7f49000703d0 2023-11-15T01:49:21.393 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:21.385+0000 7f48fcff9700 1 -- 172.21.15.55:0/3213616498 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f48f001a9f0 con 0x7f49000703d0 2023-11-15T01:49:21.393 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:21.385+0000 7f48fcff9700 1 -- 172.21.15.55:0/3213616498 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f48f001b8c0 con 0x7f49000703d0 2023-11-15T01:49:21.393 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:21.385+0000 7f490635d700 1 -- 172.21.15.55:0/3213616498 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f490010b120 con 0x7f49000703d0 2023-11-15T01:49:21.393 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:21.385+0000 7f490635d700 1 -- 172.21.15.55:0/3213616498 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f490010b530 con 0x7f49000703d0 2023-11-15T01:49:21.394 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:21.386+0000 7f48fcff9700 1 -- 172.21.15.55:0/3213616498 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f48f000f070 con 0x7f49000703d0 2023-11-15T01:49:21.394 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:21.386+0000 7f48fcff9700 1 --2- 172.21.15.55:0/3213616498 >> [v2:172.21.15.55:6800/1950005071,v1:172.21.15.55:6801/1950005071] conn(0x7f48ec02fd50 0x7f48ec032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:49:21.394 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:21.386+0000 7f490635d700 1 -- 172.21.15.55:0/3213616498 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4900045ad0 con 0x7f49000703d0 2023-11-15T01:49:21.394 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:21.387+0000 7f48fcff9700 1 -- 172.21.15.55:0/3213616498 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f48f00413d0 con 0x7f49000703d0 2023-11-15T01:49:21.394 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:21.387+0000 7f48ff7fe700 1 --2- 172.21.15.55:0/3213616498 >> [v2:172.21.15.55:6800/1950005071,v1:172.21.15.55:6801/1950005071] conn(0x7f48ec02fd50 0x7f48ec032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:49:21.394 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:21.387+0000 7f48ff7fe700 1 --2- 172.21.15.55:0/3213616498 >> [v2:172.21.15.55:6800/1950005071,v1:172.21.15.55:6801/1950005071] conn(0x7f48ec02fd50 0x7f48ec032210 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f48f40096a0 tx=0x7f48f4006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:49:21.395 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:21.391+0000 7f48fcff9700 1 -- 172.21.15.55:0/3213616498 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f48f0016070 con 0x7f49000703d0 2023-11-15T01:49:21.590 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:21.589+0000 7f490635d700 1 -- 172.21.15.55:0/3213616498 --> [v2:172.21.15.55:6800/1950005071,v1:172.21.15.55:6801/1950005071] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7f490010be10 con 0x7f48ec02fd50 2023-11-15T01:49:21.595 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:21.593+0000 7f48fcff9700 1 -- 172.21.15.55:0/3213616498 <== mgr.14116 v2:172.21.15.55:6800/1950005071 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f490010be10 con 0x7f48ec02fd50 2023-11-15T01:49:21.595 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: Scheduled mon update... 2023-11-15T01:49:21.598 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:21.595+0000 7f490635d700 1 -- 172.21.15.55:0/3213616498 >> [v2:172.21.15.55:6800/1950005071,v1:172.21.15.55:6801/1950005071] conn(0x7f48ec02fd50 msgr2=0x7f48ec032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:49:21.598 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:21.595+0000 7f490635d700 1 --2- 172.21.15.55:0/3213616498 >> [v2:172.21.15.55:6800/1950005071,v1:172.21.15.55:6801/1950005071] conn(0x7f48ec02fd50 0x7f48ec032210 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f48f40096a0 tx=0x7f48f4006b40).stop 2023-11-15T01:49:21.598 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:21.595+0000 7f490635d700 1 -- 172.21.15.55:0/3213616498 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f49000703d0 msgr2=0x7f490010aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:49:21.599 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:21.595+0000 7f490635d700 1 --2- 172.21.15.55:0/3213616498 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f49000703d0 0x7f490010aa30 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f48f000e040 tx=0x7f48f0006ee0).stop 2023-11-15T01:49:21.599 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:21.595+0000 7f490635d700 1 -- 172.21.15.55:0/3213616498 shutdown_connections 2023-11-15T01:49:21.599 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:21.595+0000 7f490635d700 1 --2- 172.21.15.55:0/3213616498 >> [v2:172.21.15.55:6800/1950005071,v1:172.21.15.55:6801/1950005071] conn(0x7f48ec02fd50 0x7f48ec032210 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:49:21.599 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:21.595+0000 7f490635d700 1 --2- 172.21.15.55:0/3213616498 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f49000703d0 0x7f490010aa30 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:49:21.599 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:21.595+0000 7f490635d700 1 -- 172.21.15.55:0/3213616498 >> 172.21.15.55:0/3213616498 conn(0x7f49000f63c0 msgr2=0x7f49000f7f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:49:21.600 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:21.596+0000 7f490635d700 1 -- 172.21.15.55:0/3213616498 shutdown_connections 2023-11-15T01:49:21.600 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:21.596+0000 7f490635d700 1 -- 172.21.15.55:0/3213616498 wait complete. 2023-11-15T01:49:21.776 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:21 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:49:20.061431+0000 mgr.smithi055.szwngw (mgr.14116) 8 : audit [DBG] from='client.14134 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi055", "addr": "172.21.15.55", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T01:49:21.777 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:21 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:49:20.712941+0000 mon.smithi055 (mon.0) 55 : audit [INF] from='mgr.14116 172.21.15.55:0/3759764227' entity='mgr.smithi055.szwngw' 2023-11-15T01:49:21.777 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:21 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:49:20.713572+0000 mon.smithi055 (mon.0) 56 : audit [DBG] from='mgr.14116 172.21.15.55:0/3759764227' entity='mgr.smithi055.szwngw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-15T01:49:21.928 INFO:teuthology.orchestra.run.smithi055.stderr:Deploying mgr service with default placement... 2023-11-15T01:49:21.928 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi055 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/0cb4fc0c-8359-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpc1yi9jse:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpra3bp1py:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mgr 2023-11-15T01:49:22.323 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:22.320+0000 7f05ab59e700 1 Processor -- start 2023-11-15T01:49:22.323 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:22.320+0000 7f05ab59e700 1 -- start start 2023-11-15T01:49:22.324 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:22.320+0000 7f05ab59e700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f05ac0fbaf0 0x7f05ac0fbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T01:49:22.324 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:22.320+0000 7f05ab59e700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05ac0fc400 con 0x7f05ac0fbaf0 2023-11-15T01:49:22.324 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:22.321+0000 7f05aa59c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f05ac0fbaf0 0x7f05ac0fbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:49:22.324 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:22.321+0000 7f05aa59c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f05ac0fbaf0 0x7f05ac0fbf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:53148/0 (socket says 172.21.15.55:53148) 2023-11-15T01:49:22.324 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:22.321+0000 7f05aa59c700 1 -- 172.21.15.55:0/882390465 learned_addr learned my addr 172.21.15.55:0/882390465 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-11-15T01:49:22.324 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:22.321+0000 7f05aa59c700 1 -- 172.21.15.55:0/882390465 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05ac0fc540 con 0x7f05ac0fbaf0 2023-11-15T01:49:22.325 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:22.321+0000 7f05aa59c700 1 --2- 172.21.15.55:0/882390465 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f05ac0fbaf0 0x7f05ac0fbf10 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f059400b5f0 tx=0x7f0594006ee0).ready entity=mon.0 client_cookie=1f1b1acf5c8e500a server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:49:22.325 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:22.321+0000 7f05a959a700 1 -- 172.21.15.55:0/882390465 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f059400d600 con 0x7f05ac0fbaf0 2023-11-15T01:49:22.325 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:22.321+0000 7f05a959a700 1 -- 172.21.15.55:0/882390465 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f059400dbd0 con 0x7f05ac0fbaf0 2023-11-15T01:49:22.325 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:22.321+0000 7f05a959a700 1 -- 172.21.15.55:0/882390465 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0594011c00 con 0x7f05ac0fbaf0 2023-11-15T01:49:22.325 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:22.322+0000 7f05ab59e700 1 -- 172.21.15.55:0/882390465 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f05ac0fbaf0 msgr2=0x7f05ac0fbf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:49:22.328 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:22.322+0000 7f05ab59e700 1 --2- 172.21.15.55:0/882390465 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f05ac0fbaf0 0x7f05ac0fbf10 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f059400b5f0 tx=0x7f0594006ee0).stop 2023-11-15T01:49:22.328 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:22.322+0000 7f05ab59e700 1 -- 172.21.15.55:0/882390465 shutdown_connections 2023-11-15T01:49:22.328 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:22.322+0000 7f05ab59e700 1 --2- 172.21.15.55:0/882390465 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f05ac0fbaf0 0x7f05ac0fbf10 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:49:22.328 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:22.322+0000 7f05ab59e700 1 -- 172.21.15.55:0/882390465 >> 172.21.15.55:0/882390465 conn(0x7f05ac0f7350 msgr2=0x7f05ac0f9770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:49:22.329 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:22.322+0000 7f05ab59e700 1 -- 172.21.15.55:0/882390465 shutdown_connections 2023-11-15T01:49:22.329 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:22.322+0000 7f05ab59e700 1 -- 172.21.15.55:0/882390465 wait complete. 2023-11-15T01:49:22.329 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:22.323+0000 7f05ab59e700 1 Processor -- start 2023-11-15T01:49:22.329 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:22.323+0000 7f05ab59e700 1 -- start start 2023-11-15T01:49:22.329 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:22.323+0000 7f05ab59e700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f05ac0fbaf0 0x7f05ac10a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:49:22.330 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:22.323+0000 7f05ab59e700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05ac10adf0 con 0x7f05ac0fbaf0 2023-11-15T01:49:22.330 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:22.323+0000 7f05aa59c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f05ac0fbaf0 0x7f05ac10a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:49:22.330 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:22.323+0000 7f05aa59c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f05ac0fbaf0 0x7f05ac10a900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:53158/0 (socket says 172.21.15.55:53158) 2023-11-15T01:49:22.330 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:22.323+0000 7f05aa59c700 1 -- 172.21.15.55:0/160026744 learned_addr learned my addr 172.21.15.55:0/160026744 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-11-15T01:49:22.330 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:22.323+0000 7f05aa59c700 1 -- 172.21.15.55:0/160026744 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f059400b200 con 0x7f05ac0fbaf0 2023-11-15T01:49:22.331 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:22.324+0000 7f05aa59c700 1 --2- 172.21.15.55:0/160026744 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f05ac0fbaf0 0x7f05ac10a900 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f0594000c00 tx=0x7f0594006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:49:22.331 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:22.324+0000 7f05a2ffd700 1 -- 172.21.15.55:0/160026744 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0594011690 con 0x7f05ac0fbaf0 2023-11-15T01:49:22.331 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:22.324+0000 7f05a2ffd700 1 -- 172.21.15.55:0/160026744 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f059400d9f0 con 0x7f05ac0fbaf0 2023-11-15T01:49:22.331 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:22.324+0000 7f05ab59e700 1 -- 172.21.15.55:0/160026744 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05ac10aff0 con 0x7f05ac0fbaf0 2023-11-15T01:49:22.331 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:22.324+0000 7f05a2ffd700 1 -- 172.21.15.55:0/160026744 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f059401b710 con 0x7f05ac0fbaf0 2023-11-15T01:49:22.332 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:22.324+0000 7f05ab59e700 1 -- 172.21.15.55:0/160026744 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05ac10b400 con 0x7f05ac0fbaf0 2023-11-15T01:49:22.332 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:22.325+0000 7f05a2ffd700 1 -- 172.21.15.55:0/160026744 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f059400f070 con 0x7f05ac0fbaf0 2023-11-15T01:49:22.332 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:22.325+0000 7f05ab59e700 1 -- 172.21.15.55:0/160026744 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05ac045ad0 con 0x7f05ac0fbaf0 2023-11-15T01:49:22.332 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:22.325+0000 7f05a2ffd700 1 --2- 172.21.15.55:0/160026744 >> [v2:172.21.15.55:6800/1950005071,v1:172.21.15.55:6801/1950005071] conn(0x7f058c02fdb0 0x7f058c032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:49:22.332 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:22.325+0000 7f05a2ffd700 1 -- 172.21.15.55:0/160026744 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f0594041180 con 0x7f05ac0fbaf0 2023-11-15T01:49:22.332 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:22.325+0000 7f05a9d9b700 1 --2- 172.21.15.55:0/160026744 >> [v2:172.21.15.55:6800/1950005071,v1:172.21.15.55:6801/1950005071] conn(0x7f058c02fdb0 0x7f058c032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:49:22.333 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:22.325+0000 7f05a9d9b700 1 --2- 172.21.15.55:0/160026744 >> [v2:172.21.15.55:6800/1950005071,v1:172.21.15.55:6801/1950005071] conn(0x7f058c02fdb0 0x7f058c032270 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f059c0096a0 tx=0x7f059c006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:49:22.333 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:22.330+0000 7f05a2ffd700 1 -- 172.21.15.55:0/160026744 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0594022900 con 0x7f05ac0fbaf0 2023-11-15T01:49:22.526 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:22.524+0000 7f05ab59e700 1 -- 172.21.15.55:0/160026744 --> [v2:172.21.15.55:6800/1950005071,v1:172.21.15.55:6801/1950005071] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f05ac10bce0 con 0x7f058c02fdb0 2023-11-15T01:49:22.530 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:22.528+0000 7f05a2ffd700 1 -- 172.21.15.55:0/160026744 <== mgr.14116 v2:172.21.15.55:6800/1950005071 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f05ac10bce0 con 0x7f058c02fdb0 2023-11-15T01:49:22.530 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: Scheduled mgr update... 2023-11-15T01:49:22.533 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:22.530+0000 7f05ab59e700 1 -- 172.21.15.55:0/160026744 >> [v2:172.21.15.55:6800/1950005071,v1:172.21.15.55:6801/1950005071] conn(0x7f058c02fdb0 msgr2=0x7f058c032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:49:22.533 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:22.530+0000 7f05ab59e700 1 --2- 172.21.15.55:0/160026744 >> [v2:172.21.15.55:6800/1950005071,v1:172.21.15.55:6801/1950005071] conn(0x7f058c02fdb0 0x7f058c032270 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f059c0096a0 tx=0x7f059c006b40).stop 2023-11-15T01:49:22.533 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:22.530+0000 7f05ab59e700 1 -- 172.21.15.55:0/160026744 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f05ac0fbaf0 msgr2=0x7f05ac10a900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:49:22.533 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:22.530+0000 7f05ab59e700 1 --2- 172.21.15.55:0/160026744 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f05ac0fbaf0 0x7f05ac10a900 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f0594000c00 tx=0x7f0594006ee0).stop 2023-11-15T01:49:22.534 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:22.532+0000 7f05ab59e700 1 -- 172.21.15.55:0/160026744 shutdown_connections 2023-11-15T01:49:22.534 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:22.532+0000 7f05ab59e700 1 --2- 172.21.15.55:0/160026744 >> [v2:172.21.15.55:6800/1950005071,v1:172.21.15.55:6801/1950005071] conn(0x7f058c02fdb0 0x7f058c032270 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:49:22.534 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:22.532+0000 7f05ab59e700 1 --2- 172.21.15.55:0/160026744 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f05ac0fbaf0 0x7f05ac10a900 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:49:22.534 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:22.532+0000 7f05ab59e700 1 -- 172.21.15.55:0/160026744 >> 172.21.15.55:0/160026744 conn(0x7f05ac0f7350 msgr2=0x7f05ac0f8ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:49:22.535 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:22.532+0000 7f05ab59e700 1 -- 172.21.15.55:0/160026744 shutdown_connections 2023-11-15T01:49:22.535 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:22.532+0000 7f05ab59e700 1 -- 172.21.15.55:0/160026744 wait complete. 2023-11-15T01:49:22.751 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:22 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cephadm 2023-11-15T01:49:20.713292+0000 mgr.smithi055.szwngw (mgr.14116) 9 : cephadm [INF] Added host smithi055 2023-11-15T01:49:22.752 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:22 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:49:21.595131+0000 mon.smithi055 (mon.0) 57 : audit [INF] from='mgr.14116 172.21.15.55:0/3759764227' entity='mgr.smithi055.szwngw' 2023-11-15T01:49:22.936 INFO:teuthology.orchestra.run.smithi055.stderr:Deploying crash service with default placement... 2023-11-15T01:49:22.936 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi055 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/0cb4fc0c-8359-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpc1yi9jse:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpra3bp1py:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply crash 2023-11-15T01:49:23.289 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:23.287+0000 7fa900866700 1 Processor -- start 2023-11-15T01:49:23.289 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:23.287+0000 7fa900866700 1 -- start start 2023-11-15T01:49:23.289 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:23.287+0000 7fa900866700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa8fc0703f0 0x7fa8fc0fe770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T01:49:23.290 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:23.287+0000 7fa900866700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8fc0fec60 con 0x7fa8fc0703f0 2023-11-15T01:49:23.290 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:23.287+0000 7fa8fb7fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa8fc0703f0 0x7fa8fc0fe770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:49:23.290 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:23.288+0000 7fa8fb7fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa8fc0703f0 0x7fa8fc0fe770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:53168/0 (socket says 172.21.15.55:53168) 2023-11-15T01:49:23.290 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:23.288+0000 7fa8fb7fe700 1 -- 172.21.15.55:0/938892723 learned_addr learned my addr 172.21.15.55:0/938892723 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-11-15T01:49:23.290 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:23.288+0000 7fa8fb7fe700 1 -- 172.21.15.55:0/938892723 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8fc0feda0 con 0x7fa8fc0703f0 2023-11-15T01:49:23.291 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:23.288+0000 7fa8fb7fe700 1 --2- 172.21.15.55:0/938892723 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa8fc0703f0 0x7fa8fc0fe770 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7fa8ec00b5f0 tx=0x7fa8ec006ee0).ready entity=mon.0 client_cookie=977dc63da6cf6edc server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:49:23.291 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:23.288+0000 7fa8fa7fc700 1 -- 172.21.15.55:0/938892723 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa8ec00d600 con 0x7fa8fc0703f0 2023-11-15T01:49:23.293 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:23.288+0000 7fa8fa7fc700 1 -- 172.21.15.55:0/938892723 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fa8ec00dbd0 con 0x7fa8fc0703f0 2023-11-15T01:49:23.293 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:23.288+0000 7fa8fa7fc700 1 -- 172.21.15.55:0/938892723 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa8ec011c00 con 0x7fa8fc0703f0 2023-11-15T01:49:23.293 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:23.289+0000 7fa900866700 1 -- 172.21.15.55:0/938892723 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa8fc0703f0 msgr2=0x7fa8fc0fe770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:49:23.293 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:23.289+0000 7fa900866700 1 --2- 172.21.15.55:0/938892723 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa8fc0703f0 0x7fa8fc0fe770 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7fa8ec00b5f0 tx=0x7fa8ec006ee0).stop 2023-11-15T01:49:23.294 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:23.289+0000 7fa900866700 1 -- 172.21.15.55:0/938892723 shutdown_connections 2023-11-15T01:49:23.294 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:23.289+0000 7fa900866700 1 --2- 172.21.15.55:0/938892723 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa8fc0703f0 0x7fa8fc0fe770 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:49:23.294 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:23.289+0000 7fa900866700 1 -- 172.21.15.55:0/938892723 >> 172.21.15.55:0/938892723 conn(0x7fa8fc0f6210 msgr2=0x7fa8fc0f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:49:23.294 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:23.289+0000 7fa900866700 1 -- 172.21.15.55:0/938892723 shutdown_connections 2023-11-15T01:49:23.295 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:23.289+0000 7fa900866700 1 -- 172.21.15.55:0/938892723 wait complete. 2023-11-15T01:49:23.295 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:23.289+0000 7fa900866700 1 Processor -- start 2023-11-15T01:49:23.295 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:23.290+0000 7fa900866700 1 -- start start 2023-11-15T01:49:23.295 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:23.290+0000 7fa900866700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa8fc0703f0 0x7fa8fc0fc1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:49:23.295 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:23.290+0000 7fa900866700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8fc0fa670 con 0x7fa8fc0703f0 2023-11-15T01:49:23.296 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:23.290+0000 7fa8fb7fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa8fc0703f0 0x7fa8fc0fc1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:49:23.296 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:23.290+0000 7fa8fb7fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa8fc0703f0 0x7fa8fc0fc1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:53172/0 (socket says 172.21.15.55:53172) 2023-11-15T01:49:23.296 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:23.290+0000 7fa8fb7fe700 1 -- 172.21.15.55:0/3535011269 learned_addr learned my addr 172.21.15.55:0/3535011269 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-11-15T01:49:23.296 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:23.290+0000 7fa8fb7fe700 1 -- 172.21.15.55:0/3535011269 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8ec00b200 con 0x7fa8fc0703f0 2023-11-15T01:49:23.296 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:23.290+0000 7fa8fb7fe700 1 --2- 172.21.15.55:0/3535011269 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa8fc0703f0 0x7fa8fc0fc1f0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7fa8ec00e040 tx=0x7fa8ec006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:49:23.296 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:23.291+0000 7fa8e3fff700 1 -- 172.21.15.55:0/3535011269 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa8ec00d9f0 con 0x7fa8fc0703f0 2023-11-15T01:49:23.297 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:23.291+0000 7fa8e3fff700 1 -- 172.21.15.55:0/3535011269 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fa8ec01a9f0 con 0x7fa8fc0703f0 2023-11-15T01:49:23.297 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:23.291+0000 7fa8e3fff700 1 -- 172.21.15.55:0/3535011269 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa8ec01b8c0 con 0x7fa8fc0703f0 2023-11-15T01:49:23.297 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:23.291+0000 7fa900866700 1 -- 172.21.15.55:0/3535011269 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8fc0fa810 con 0x7fa8fc0703f0 2023-11-15T01:49:23.297 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:23.291+0000 7fa900866700 1 -- 172.21.15.55:0/3535011269 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8fc0fac20 con 0x7fa8fc0703f0 2023-11-15T01:49:23.297 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:23.292+0000 7fa900866700 1 -- 172.21.15.55:0/3535011269 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8fc045ad0 con 0x7fa8fc0703f0 2023-11-15T01:49:23.298 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:23.292+0000 7fa8e3fff700 1 -- 172.21.15.55:0/3535011269 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fa8ec00f070 con 0x7fa8fc0703f0 2023-11-15T01:49:23.299 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:23.293+0000 7fa8e3fff700 1 --2- 172.21.15.55:0/3535011269 >> [v2:172.21.15.55:6800/1950005071,v1:172.21.15.55:6801/1950005071] conn(0x7fa8e802fdb0 0x7fa8e8032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:49:23.299 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:23.293+0000 7fa8e3fff700 1 -- 172.21.15.55:0/3535011269 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fa8ec041980 con 0x7fa8fc0703f0 2023-11-15T01:49:23.299 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:23.293+0000 7fa8faffd700 1 --2- 172.21.15.55:0/3535011269 >> [v2:172.21.15.55:6800/1950005071,v1:172.21.15.55:6801/1950005071] conn(0x7fa8e802fdb0 0x7fa8e8032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:49:23.299 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:23.293+0000 7fa8faffd700 1 --2- 172.21.15.55:0/3535011269 >> [v2:172.21.15.55:6800/1950005071,v1:172.21.15.55:6801/1950005071] conn(0x7fa8e802fdb0 0x7fa8e8032270 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7fa8f00096a0 tx=0x7fa8f0006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:49:23.299 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:23.297+0000 7fa8e3fff700 1 -- 172.21.15.55:0/3535011269 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa8ec02d7a0 con 0x7fa8fc0703f0 2023-11-15T01:49:23.493 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:23.492+0000 7fa900866700 1 -- 172.21.15.55:0/3535011269 --> [v2:172.21.15.55:6800/1950005071,v1:172.21.15.55:6801/1950005071] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7fa8fc0fae60 con 0x7fa8e802fdb0 2023-11-15T01:49:23.502 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:23.501+0000 7fa8e3fff700 1 -- 172.21.15.55:0/3535011269 <== mgr.14116 v2:172.21.15.55:6800/1950005071 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7fa8fc0fae60 con 0x7fa8e802fdb0 2023-11-15T01:49:23.502 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: Scheduled crash update... 2023-11-15T01:49:23.505 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:23.503+0000 7fa900866700 1 -- 172.21.15.55:0/3535011269 >> [v2:172.21.15.55:6800/1950005071,v1:172.21.15.55:6801/1950005071] conn(0x7fa8e802fdb0 msgr2=0x7fa8e8032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:49:23.505 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:23.503+0000 7fa900866700 1 --2- 172.21.15.55:0/3535011269 >> [v2:172.21.15.55:6800/1950005071,v1:172.21.15.55:6801/1950005071] conn(0x7fa8e802fdb0 0x7fa8e8032270 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7fa8f00096a0 tx=0x7fa8f0006b40).stop 2023-11-15T01:49:23.505 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:23.503+0000 7fa900866700 1 -- 172.21.15.55:0/3535011269 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa8fc0703f0 msgr2=0x7fa8fc0fc1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:49:23.505 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:23.503+0000 7fa900866700 1 --2- 172.21.15.55:0/3535011269 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa8fc0703f0 0x7fa8fc0fc1f0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7fa8ec00e040 tx=0x7fa8ec006ee0).stop 2023-11-15T01:49:23.505 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:23.503+0000 7fa900866700 1 -- 172.21.15.55:0/3535011269 shutdown_connections 2023-11-15T01:49:23.506 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:23.503+0000 7fa900866700 1 --2- 172.21.15.55:0/3535011269 >> [v2:172.21.15.55:6800/1950005071,v1:172.21.15.55:6801/1950005071] conn(0x7fa8e802fdb0 0x7fa8e8032270 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:49:23.506 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:23.503+0000 7fa900866700 1 --2- 172.21.15.55:0/3535011269 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa8fc0703f0 0x7fa8fc0fc1f0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:49:23.506 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:23.503+0000 7fa900866700 1 -- 172.21.15.55:0/3535011269 >> 172.21.15.55:0/3535011269 conn(0x7fa8fc0f6210 msgr2=0x7fa8fc0f7cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:49:23.506 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:23.503+0000 7fa900866700 1 -- 172.21.15.55:0/3535011269 shutdown_connections 2023-11-15T01:49:23.506 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:23.503+0000 7fa900866700 1 -- 172.21.15.55:0/3535011269 wait complete. 2023-11-15T01:49:23.780 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:23 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:49:21.591711+0000 mgr.smithi055.szwngw (mgr.14116) 10 : audit [DBG] from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T01:49:23.781 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:23 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cephadm 2023-11-15T01:49:21.592542+0000 mgr.smithi055.szwngw (mgr.14116) 11 : cephadm [INF] Saving service mon spec with placement count:5 2023-11-15T01:49:23.781 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:23 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:49:22.527226+0000 mgr.smithi055.szwngw (mgr.14116) 12 : audit [DBG] from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T01:49:23.781 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:23 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cephadm 2023-11-15T01:49:22.527799+0000 mgr.smithi055.szwngw (mgr.14116) 13 : cephadm [INF] Saving service mgr spec with placement count:2 2023-11-15T01:49:23.781 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:23 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:49:22.530213+0000 mon.smithi055 (mon.0) 58 : audit [INF] from='mgr.14116 172.21.15.55:0/3759764227' entity='mgr.smithi055.szwngw' 2023-11-15T01:49:23.782 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:23 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:49:23.502090+0000 mon.smithi055 (mon.0) 59 : audit [INF] from='mgr.14116 172.21.15.55:0/3759764227' entity='mgr.smithi055.szwngw' 2023-11-15T01:49:23.836 INFO:teuthology.orchestra.run.smithi055.stderr:Enabling mgr prometheus module... 2023-11-15T01:49:23.836 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi055 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/0cb4fc0c-8359-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpc1yi9jse:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpra3bp1py:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable prometheus 2023-11-15T01:49:24.234 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:24.233+0000 7f40e6098700 1 Processor -- start 2023-11-15T01:49:24.235 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:24.233+0000 7f40e6098700 1 -- start start 2023-11-15T01:49:24.235 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:24.234+0000 7f40e6098700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f40e0072500 0x7f40e0070aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T01:49:24.237 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:24.234+0000 7f40e6098700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40e0072920 con 0x7f40e0072500 2023-11-15T01:49:24.237 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:24.234+0000 7f40df7fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f40e0072500 0x7f40e0070aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:49:24.237 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:24.234+0000 7f40df7fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f40e0072500 0x7f40e0070aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:53182/0 (socket says 172.21.15.55:53182) 2023-11-15T01:49:24.237 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:24.234+0000 7f40df7fe700 1 -- 172.21.15.55:0/3673812049 learned_addr learned my addr 172.21.15.55:0/3673812049 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-11-15T01:49:24.238 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:24.234+0000 7f40df7fe700 1 -- 172.21.15.55:0/3673812049 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40e0070f90 con 0x7f40e0072500 2023-11-15T01:49:24.238 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:24.234+0000 7f40df7fe700 1 --2- 172.21.15.55:0/3673812049 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f40e0072500 0x7f40e0070aa0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f40c800b5f0 tx=0x7f40c8006ee0).ready entity=mon.0 client_cookie=ea8889d4aa0357a2 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:49:24.238 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:24.235+0000 7f40de7fc700 1 -- 172.21.15.55:0/3673812049 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f40c800d600 con 0x7f40e0072500 2023-11-15T01:49:24.238 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:24.235+0000 7f40de7fc700 1 -- 172.21.15.55:0/3673812049 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f40c800dbd0 con 0x7f40e0072500 2023-11-15T01:49:24.238 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:24.235+0000 7f40de7fc700 1 -- 172.21.15.55:0/3673812049 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f40c8011c00 con 0x7f40e0072500 2023-11-15T01:49:24.239 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:24.235+0000 7f40e6098700 1 -- 172.21.15.55:0/3673812049 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f40e0072500 msgr2=0x7f40e0070aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:49:24.239 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:24.235+0000 7f40e6098700 1 --2- 172.21.15.55:0/3673812049 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f40e0072500 0x7f40e0070aa0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f40c800b5f0 tx=0x7f40c8006ee0).stop 2023-11-15T01:49:24.239 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:24.235+0000 7f40e6098700 1 -- 172.21.15.55:0/3673812049 shutdown_connections 2023-11-15T01:49:24.239 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:24.235+0000 7f40e6098700 1 --2- 172.21.15.55:0/3673812049 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f40e0072500 0x7f40e0070aa0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:49:24.239 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:24.235+0000 7f40e6098700 1 -- 172.21.15.55:0/3673812049 >> 172.21.15.55:0/3673812049 conn(0x7f40e00f74b0 msgr2=0x7f40e00f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:49:24.240 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:24.236+0000 7f40e6098700 1 -- 172.21.15.55:0/3673812049 shutdown_connections 2023-11-15T01:49:24.240 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:24.236+0000 7f40e6098700 1 -- 172.21.15.55:0/3673812049 wait complete. 2023-11-15T01:49:24.241 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:24.236+0000 7f40e6098700 1 Processor -- start 2023-11-15T01:49:24.241 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:24.236+0000 7f40e6098700 1 -- start start 2023-11-15T01:49:24.241 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:24.237+0000 7f40e6098700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f40e0072500 0x7f40e010cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:49:24.241 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:24.237+0000 7f40e6098700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40e010d240 con 0x7f40e0072500 2023-11-15T01:49:24.241 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:24.237+0000 7f40df7fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f40e0072500 0x7f40e010cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:49:24.242 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:24.237+0000 7f40df7fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f40e0072500 0x7f40e010cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:53192/0 (socket says 172.21.15.55:53192) 2023-11-15T01:49:24.242 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:24.237+0000 7f40df7fe700 1 -- 172.21.15.55:0/837874435 learned_addr learned my addr 172.21.15.55:0/837874435 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-11-15T01:49:24.242 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:24.237+0000 7f40df7fe700 1 -- 172.21.15.55:0/837874435 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40c800b200 con 0x7f40e0072500 2023-11-15T01:49:24.242 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:24.237+0000 7f40df7fe700 1 --2- 172.21.15.55:0/837874435 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f40e0072500 0x7f40e010cd50 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f40c800b190 tx=0x7f40c8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:49:24.242 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:24.237+0000 7f40d7fff700 1 -- 172.21.15.55:0/837874435 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f40c8011a20 con 0x7f40e0072500 2023-11-15T01:49:24.243 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:24.237+0000 7f40d7fff700 1 -- 172.21.15.55:0/837874435 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f40c801b880 con 0x7f40e0072500 2023-11-15T01:49:24.243 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:24.237+0000 7f40e6098700 1 -- 172.21.15.55:0/837874435 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40e010d440 con 0x7f40e0072500 2023-11-15T01:49:24.243 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:24.237+0000 7f40d7fff700 1 -- 172.21.15.55:0/837874435 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f40c801a710 con 0x7f40e0072500 2023-11-15T01:49:24.243 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:24.237+0000 7f40e6098700 1 -- 172.21.15.55:0/837874435 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40e010d850 con 0x7f40e0072500 2023-11-15T01:49:24.244 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:24.238+0000 7f40d7fff700 1 -- 172.21.15.55:0/837874435 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f40c800f070 con 0x7f40e0072500 2023-11-15T01:49:24.244 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:24.238+0000 7f40d7fff700 1 --2- 172.21.15.55:0/837874435 >> [v2:172.21.15.55:6800/1950005071,v1:172.21.15.55:6801/1950005071] conn(0x7f40c002fd50 0x7f40c0032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:49:24.244 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:24.238+0000 7f40d7fff700 1 -- 172.21.15.55:0/837874435 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f40c8041140 con 0x7f40e0072500 2023-11-15T01:49:24.244 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:24.239+0000 7f40deffd700 1 --2- 172.21.15.55:0/837874435 >> [v2:172.21.15.55:6800/1950005071,v1:172.21.15.55:6801/1950005071] conn(0x7f40c002fd50 0x7f40c0032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:49:24.244 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:24.239+0000 7f40deffd700 1 --2- 172.21.15.55:0/837874435 >> [v2:172.21.15.55:6800/1950005071,v1:172.21.15.55:6801/1950005071] conn(0x7f40c002fd50 0x7f40c0032210 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f40d00096a0 tx=0x7f40d0006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:49:24.245 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:24.239+0000 7f40e6098700 1 -- 172.21.15.55:0/837874435 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40e0045ad0 con 0x7f40e0072500 2023-11-15T01:49:24.246 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:24.243+0000 7f40d7fff700 1 -- 172.21.15.55:0/837874435 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f40c8025460 con 0x7f40e0072500 2023-11-15T01:49:24.448 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:24.447+0000 7f40e6098700 1 -- 172.21.15.55:0/837874435 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "prometheus"} v 0) v1 -- 0x7f40e0106a60 con 0x7f40e0072500 2023-11-15T01:49:24.536 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:24.535+0000 7f40d7fff700 1 -- 172.21.15.55:0/837874435 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mgrmap(e 9) v1 ==== 36134+0+0 (secure 0 0 0) 0x7f40c80257e0 con 0x7f40e0072500 2023-11-15T01:49:24.537 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:24.537+0000 7f40d7fff700 1 -- 172.21.15.55:0/837874435 <== mon.0 v2:172.21.15.55:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "prometheus"}]=0 v9) v1 ==== 89+0+0 (secure 0 0 0) 0x7f40c80235f0 con 0x7f40e0072500 2023-11-15T01:49:24.540 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:24.539+0000 7f40e6098700 1 -- 172.21.15.55:0/837874435 >> [v2:172.21.15.55:6800/1950005071,v1:172.21.15.55:6801/1950005071] conn(0x7f40c002fd50 msgr2=0x7f40c0032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:49:24.540 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:24.539+0000 7f40e6098700 1 --2- 172.21.15.55:0/837874435 >> [v2:172.21.15.55:6800/1950005071,v1:172.21.15.55:6801/1950005071] conn(0x7f40c002fd50 0x7f40c0032210 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f40d00096a0 tx=0x7f40d0006b40).stop 2023-11-15T01:49:24.540 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:24.539+0000 7f40e6098700 1 -- 172.21.15.55:0/837874435 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f40e0072500 msgr2=0x7f40e010cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:49:24.541 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:24.539+0000 7f40e6098700 1 --2- 172.21.15.55:0/837874435 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f40e0072500 0x7f40e010cd50 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f40c800b190 tx=0x7f40c8006ee0).stop 2023-11-15T01:49:24.541 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:24.539+0000 7f40e6098700 1 -- 172.21.15.55:0/837874435 shutdown_connections 2023-11-15T01:49:24.542 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:24.539+0000 7f40e6098700 1 --2- 172.21.15.55:0/837874435 >> [v2:172.21.15.55:6800/1950005071,v1:172.21.15.55:6801/1950005071] conn(0x7f40c002fd50 0x7f40c0032210 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:49:24.542 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:24.539+0000 7f40e6098700 1 --2- 172.21.15.55:0/837874435 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f40e0072500 0x7f40e010cd50 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:49:24.542 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:24.539+0000 7f40e6098700 1 -- 172.21.15.55:0/837874435 >> 172.21.15.55:0/837874435 conn(0x7f40e00f74b0 msgr2=0x7f40e00f9030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:49:24.542 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:24.541+0000 7f40e6098700 1 -- 172.21.15.55:0/837874435 shutdown_connections 2023-11-15T01:49:24.543 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:24.541+0000 7f40e6098700 1 -- 172.21.15.55:0/837874435 wait complete. 2023-11-15T01:49:24.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:24 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:49:23.494095+0000 mgr.smithi055.szwngw (mgr.14116) 14 : audit [DBG] from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T01:49:24.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:24 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cephadm 2023-11-15T01:49:23.494842+0000 mgr.smithi055.szwngw (mgr.14116) 15 : cephadm [INF] Saving service crash spec with placement * 2023-11-15T01:49:24.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:24 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:49:24.448873+0000 mon.smithi055 (mon.0) 60 : audit [INF] from='client.? 172.21.15.55:0/837874435' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2023-11-15T01:49:24.902 INFO:teuthology.orchestra.run.smithi055.stderr:Deploying prometheus service with default placement... 2023-11-15T01:49:24.902 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi055 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/0cb4fc0c-8359-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpc1yi9jse:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpra3bp1py:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply prometheus 2023-11-15T01:49:25.249 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:25.247+0000 7fda77d9d700 1 Processor -- start 2023-11-15T01:49:25.249 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:25.248+0000 7fda77d9d700 1 -- start start 2023-11-15T01:49:25.249 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:25.248+0000 7fda77d9d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fda700724b0 0x7fda70070a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T01:49:25.251 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:25.248+0000 7fda77d9d700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda700728d0 con 0x7fda700724b0 2023-11-15T01:49:25.251 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:25.248+0000 7fda75b39700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fda700724b0 0x7fda70070a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:49:25.251 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:25.248+0000 7fda75b39700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fda700724b0 0x7fda70070a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:53230/0 (socket says 172.21.15.55:53230) 2023-11-15T01:49:25.252 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:25.248+0000 7fda75b39700 1 -- 172.21.15.55:0/770803180 learned_addr learned my addr 172.21.15.55:0/770803180 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-11-15T01:49:25.252 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:25.249+0000 7fda75b39700 1 -- 172.21.15.55:0/770803180 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda70070f40 con 0x7fda700724b0 2023-11-15T01:49:25.252 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:25.249+0000 7fda75b39700 1 --2- 172.21.15.55:0/770803180 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fda700724b0 0x7fda70070a50 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7fda6c00b5f0 tx=0x7fda6c006ee0).ready entity=mon.0 client_cookie=adec70eac42362e0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:49:25.252 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:25.249+0000 7fda74b37700 1 -- 172.21.15.55:0/770803180 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda6c00d600 con 0x7fda700724b0 2023-11-15T01:49:25.252 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:25.249+0000 7fda74b37700 1 -- 172.21.15.55:0/770803180 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fda6c00dbd0 con 0x7fda700724b0 2023-11-15T01:49:25.253 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:25.249+0000 7fda74b37700 1 -- 172.21.15.55:0/770803180 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda6c011c00 con 0x7fda700724b0 2023-11-15T01:49:25.253 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:25.250+0000 7fda77d9d700 1 -- 172.21.15.55:0/770803180 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fda700724b0 msgr2=0x7fda70070a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:49:25.253 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:25.250+0000 7fda77d9d700 1 --2- 172.21.15.55:0/770803180 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fda700724b0 0x7fda70070a50 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7fda6c00b5f0 tx=0x7fda6c006ee0).stop 2023-11-15T01:49:25.253 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:25.250+0000 7fda77d9d700 1 -- 172.21.15.55:0/770803180 shutdown_connections 2023-11-15T01:49:25.255 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:25.250+0000 7fda77d9d700 1 --2- 172.21.15.55:0/770803180 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fda700724b0 0x7fda70070a50 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:49:25.255 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:25.250+0000 7fda77d9d700 1 -- 172.21.15.55:0/770803180 >> 172.21.15.55:0/770803180 conn(0x7fda700f74c0 msgr2=0x7fda700f98e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:49:25.255 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:25.250+0000 7fda77d9d700 1 -- 172.21.15.55:0/770803180 shutdown_connections 2023-11-15T01:49:25.255 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:25.250+0000 7fda77d9d700 1 -- 172.21.15.55:0/770803180 wait complete. 2023-11-15T01:49:25.255 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:25.251+0000 7fda77d9d700 1 Processor -- start 2023-11-15T01:49:25.256 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:25.251+0000 7fda77d9d700 1 -- start start 2023-11-15T01:49:25.256 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:25.251+0000 7fda77d9d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fda700724b0 0x7fda7010ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:49:25.256 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:25.251+0000 7fda77d9d700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda7010d1e0 con 0x7fda700724b0 2023-11-15T01:49:25.256 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:25.251+0000 7fda75b39700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fda700724b0 0x7fda7010ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:49:25.256 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:25.251+0000 7fda75b39700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fda700724b0 0x7fda7010ccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:53242/0 (socket says 172.21.15.55:53242) 2023-11-15T01:49:25.257 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:25.251+0000 7fda75b39700 1 -- 172.21.15.55:0/799740731 learned_addr learned my addr 172.21.15.55:0/799740731 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-11-15T01:49:25.257 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:25.252+0000 7fda75b39700 1 -- 172.21.15.55:0/799740731 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda6c00b200 con 0x7fda700724b0 2023-11-15T01:49:25.257 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:25.252+0000 7fda75b39700 1 --2- 172.21.15.55:0/799740731 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fda700724b0 0x7fda7010ccf0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7fda6c00e040 tx=0x7fda6c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:49:25.257 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:25.252+0000 7fda5e7fc700 1 -- 172.21.15.55:0/799740731 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda6c00d9f0 con 0x7fda700724b0 2023-11-15T01:49:25.257 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:25.252+0000 7fda77d9d700 1 -- 172.21.15.55:0/799740731 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda7010d3e0 con 0x7fda700724b0 2023-11-15T01:49:25.258 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:25.252+0000 7fda5e7fc700 1 -- 172.21.15.55:0/799740731 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fda6c01a9f0 con 0x7fda700724b0 2023-11-15T01:49:25.258 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:25.252+0000 7fda5e7fc700 1 -- 172.21.15.55:0/799740731 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda6c01b8c0 con 0x7fda700724b0 2023-11-15T01:49:25.258 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:25.252+0000 7fda77d9d700 1 -- 172.21.15.55:0/799740731 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda7010d7f0 con 0x7fda700724b0 2023-11-15T01:49:25.258 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:25.253+0000 7fda5e7fc700 1 -- 172.21.15.55:0/799740731 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 9) v1 ==== 36134+0+0 (secure 0 0 0) 0x7fda6c00f070 con 0x7fda700724b0 2023-11-15T01:49:25.258 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:25.253+0000 7fda77d9d700 1 -- 172.21.15.55:0/799740731 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda70045ad0 con 0x7fda700724b0 2023-11-15T01:49:25.259 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:25.254+0000 7fda5e7fc700 1 --2- 172.21.15.55:0/799740731 >> [v2:172.21.15.55:6800/1950005071,v1:172.21.15.55:6801/1950005071] conn(0x7fda5802fe00 0x7fda580322c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:49:25.259 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:25.254+0000 7fda5e7fc700 1 -- 172.21.15.55:0/799740731 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fda6c041470 con 0x7fda700724b0 2023-11-15T01:49:25.259 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:25.254+0000 7fda75338700 1 -- 172.21.15.55:0/799740731 >> [v2:172.21.15.55:6800/1950005071,v1:172.21.15.55:6801/1950005071] conn(0x7fda5802fe00 msgr2=0x7fda580322c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.55:6800/1950005071 2023-11-15T01:49:25.259 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:25.254+0000 7fda75338700 1 --2- 172.21.15.55:0/799740731 >> [v2:172.21.15.55:6800/1950005071,v1:172.21.15.55:6801/1950005071] conn(0x7fda5802fe00 0x7fda580322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-11-15T01:49:25.260 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:25.258+0000 7fda5e7fc700 1 -- 172.21.15.55:0/799740731 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fda6c0224f0 con 0x7fda700724b0 2023-11-15T01:49:25.456 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:25.455+0000 7fda75338700 1 -- 172.21.15.55:0/799740731 >> [v2:172.21.15.55:6800/1950005071,v1:172.21.15.55:6801/1950005071] conn(0x7fda5802fe00 msgr2=0x7fda580322c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.55:6800/1950005071 2023-11-15T01:49:25.456 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:25.455+0000 7fda75338700 1 --2- 172.21.15.55:0/799740731 >> [v2:172.21.15.55:6800/1950005071,v1:172.21.15.55:6801/1950005071] conn(0x7fda5802fe00 0x7fda580322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-11-15T01:49:25.456 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:25.455+0000 7fda77d9d700 1 -- 172.21.15.55:0/799740731 --> [v2:172.21.15.55:6800/1950005071,v1:172.21.15.55:6801/1950005071] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7fda7010e050 con 0x7fda5802fe00 2023-11-15T01:49:25.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:25 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:49:24.537968+0000 mon.smithi055 (mon.0) 61 : audit [INF] from='client.? 172.21.15.55:0/837874435' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2023-11-15T01:49:25.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:25 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:49:24.538046+0000 mon.smithi055 (mon.0) 62 : cluster [DBG] mgrmap e9: smithi055.szwngw(active, since 16s) 2023-11-15T01:49:25.858 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:25.856+0000 7fda75338700 1 -- 172.21.15.55:0/799740731 >> [v2:172.21.15.55:6800/1950005071,v1:172.21.15.55:6801/1950005071] conn(0x7fda5802fe00 msgr2=0x7fda580322c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.55:6800/1950005071 2023-11-15T01:49:25.859 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:25.856+0000 7fda75338700 1 --2- 172.21.15.55:0/799740731 >> [v2:172.21.15.55:6800/1950005071,v1:172.21.15.55:6801/1950005071] conn(0x7fda5802fe00 0x7fda580322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-11-15T01:49:26.658 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:26.657+0000 7fda75338700 1 -- 172.21.15.55:0/799740731 >> [v2:172.21.15.55:6800/1950005071,v1:172.21.15.55:6801/1950005071] conn(0x7fda5802fe00 msgr2=0x7fda580322c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.55:6800/1950005071 2023-11-15T01:49:26.658 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:26.657+0000 7fda75338700 1 --2- 172.21.15.55:0/799740731 >> [v2:172.21.15.55:6800/1950005071,v1:172.21.15.55:6801/1950005071] conn(0x7fda5802fe00 0x7fda580322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-11-15T01:49:28.259 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:28.258+0000 7fda75338700 1 -- 172.21.15.55:0/799740731 >> [v2:172.21.15.55:6800/1950005071,v1:172.21.15.55:6801/1950005071] conn(0x7fda5802fe00 msgr2=0x7fda580322c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.55:6800/1950005071 2023-11-15T01:49:28.260 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:28.258+0000 7fda75338700 1 --2- 172.21.15.55:0/799740731 >> [v2:172.21.15.55:6800/1950005071,v1:172.21.15.55:6801/1950005071] conn(0x7fda5802fe00 0x7fda580322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2023-11-15T01:49:30.208 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:30.207+0000 7fda5e7fc700 1 -- 172.21.15.55:0/799740731 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mgrmap(e 10) v1 ==== 35984+0+0 (secure 0 0 0) 0x7fda6c022d80 con 0x7fda700724b0 2023-11-15T01:49:30.209 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:30.207+0000 7fda5e7fc700 1 -- 172.21.15.55:0/799740731 >> [v2:172.21.15.55:6800/1950005071,v1:172.21.15.55:6801/1950005071] conn(0x7fda5802fe00 msgr2=0x7fda580322c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-15T01:49:30.209 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:30.207+0000 7fda5e7fc700 1 --2- 172.21.15.55:0/799740731 >> [v2:172.21.15.55:6800/1950005071,v1:172.21.15.55:6801/1950005071] conn(0x7fda5802fe00 0x7fda580322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-15T01:49:30.460 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:30 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:49:30.155550+0000 mon.smithi055 (mon.0) 63 : cluster [INF] Active manager daemon smithi055.szwngw restarted 2023-11-15T01:49:30.460 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:30 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:49:30.156071+0000 mon.smithi055 (mon.0) 64 : cluster [INF] Activating manager daemon smithi055.szwngw 2023-11-15T01:49:30.461 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:30 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:49:30.162727+0000 mon.smithi055 (mon.0) 65 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2023-11-15T01:49:31.222 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:31.220+0000 7fda5e7fc700 1 -- 172.21.15.55:0/799740731 <== mon.0 v2:172.21.15.55:3300/0 8 ==== mgrmap(e 11) v1 ==== 36054+0+0 (secure 0 0 0) 0x7fda6c01ac20 con 0x7fda700724b0 2023-11-15T01:49:31.223 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:31.221+0000 7fda5e7fc700 1 --2- 172.21.15.55:0/799740731 >> [v2:172.21.15.55:6800/2574607930,v1:172.21.15.55:6801/2574607930] conn(0x7fda58033c20 0x7fda58036010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:49:31.223 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:31.221+0000 7fda5e7fc700 1 -- 172.21.15.55:0/799740731 --> [v2:172.21.15.55:6800/2574607930,v1:172.21.15.55:6801/2574607930] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7fda7010e050 con 0x7fda58033c20 2023-11-15T01:49:31.224 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:31.221+0000 7fda75338700 1 --2- 172.21.15.55:0/799740731 >> [v2:172.21.15.55:6800/2574607930,v1:172.21.15.55:6801/2574607930] conn(0x7fda58033c20 0x7fda58036010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:49:31.224 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:31.222+0000 7fda75338700 1 --2- 172.21.15.55:0/799740731 >> [v2:172.21.15.55:6800/2574607930,v1:172.21.15.55:6801/2574607930] conn(0x7fda58033c20 0x7fda58036010 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fda60006fd0 tx=0x7fda60006d30).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:49:31.228 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:31.227+0000 7fda5e7fc700 1 -- 172.21.15.55:0/799740731 <== mgr.14144 v2:172.21.15.55:6800/2574607930 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7fda7010e050 con 0x7fda58033c20 2023-11-15T01:49:31.228 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: Scheduled prometheus update... 2023-11-15T01:49:31.234 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:31.230+0000 7fda77d9d700 1 -- 172.21.15.55:0/799740731 >> [v2:172.21.15.55:6800/2574607930,v1:172.21.15.55:6801/2574607930] conn(0x7fda58033c20 msgr2=0x7fda58036010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:49:31.234 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:31.230+0000 7fda77d9d700 1 --2- 172.21.15.55:0/799740731 >> [v2:172.21.15.55:6800/2574607930,v1:172.21.15.55:6801/2574607930] conn(0x7fda58033c20 0x7fda58036010 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fda60006fd0 tx=0x7fda60006d30).stop 2023-11-15T01:49:31.235 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:31.230+0000 7fda77d9d700 1 -- 172.21.15.55:0/799740731 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fda700724b0 msgr2=0x7fda7010ccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:49:31.235 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:31.230+0000 7fda77d9d700 1 --2- 172.21.15.55:0/799740731 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fda700724b0 0x7fda7010ccf0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7fda6c00e040 tx=0x7fda6c006ee0).stop 2023-11-15T01:49:31.235 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:31.230+0000 7fda77d9d700 1 -- 172.21.15.55:0/799740731 shutdown_connections 2023-11-15T01:49:31.235 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:31.230+0000 7fda77d9d700 1 --2- 172.21.15.55:0/799740731 >> [v2:172.21.15.55:6800/2574607930,v1:172.21.15.55:6801/2574607930] conn(0x7fda58033c20 0x7fda58036010 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:49:31.236 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:31.230+0000 7fda77d9d700 1 --2- 172.21.15.55:0/799740731 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fda700724b0 0x7fda7010ccf0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:49:31.236 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:31.230+0000 7fda77d9d700 1 --2- 172.21.15.55:0/799740731 >> [v2:172.21.15.55:6800/1950005071,v1:172.21.15.55:6801/1950005071] conn(0x7fda5802fe00 0x7fda580322c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-15T01:49:31.236 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:31.230+0000 7fda77d9d700 1 -- 172.21.15.55:0/799740731 >> 172.21.15.55:0/799740731 conn(0x7fda700f74c0 msgr2=0x7fda700f9030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:49:31.236 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:31.230+0000 7fda77d9d700 1 -- 172.21.15.55:0/799740731 shutdown_connections 2023-11-15T01:49:31.236 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:31.230+0000 7fda77d9d700 1 -- 172.21.15.55:0/799740731 wait complete. 2023-11-15T01:49:31.552 INFO:teuthology.orchestra.run.smithi055.stderr:Deploying grafana service with default placement... 2023-11-15T01:49:31.552 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi055 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/0cb4fc0c-8359-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpc1yi9jse:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpra3bp1py:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply grafana 2023-11-15T01:49:31.555 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:31 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:49:30.208939+0000 mon.smithi055 (mon.0) 66 : cluster [DBG] mgrmap e10: smithi055.szwngw(active, starting, since 0.0529871s) 2023-11-15T01:49:31.555 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:31 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:49:30.211959+0000 mon.smithi055 (mon.0) 67 : audit [DBG] from='mgr.14144 172.21.15.55:0/227792247' entity='mgr.smithi055.szwngw' cmd=[{"prefix": "mon metadata", "id": "smithi055"}]: dispatch 2023-11-15T01:49:31.555 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:31 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:49:30.212844+0000 mon.smithi055 (mon.0) 68 : audit [DBG] from='mgr.14144 172.21.15.55:0/227792247' entity='mgr.smithi055.szwngw' cmd=[{"prefix": "mgr metadata", "who": "smithi055.szwngw", "id": "smithi055.szwngw"}]: dispatch 2023-11-15T01:49:31.555 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:31 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:49:30.213654+0000 mon.smithi055 (mon.0) 69 : audit [DBG] from='mgr.14144 172.21.15.55:0/227792247' entity='mgr.smithi055.szwngw' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-11-15T01:49:31.555 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:31 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:49:30.213811+0000 mon.smithi055 (mon.0) 70 : audit [DBG] from='mgr.14144 172.21.15.55:0/227792247' entity='mgr.smithi055.szwngw' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-11-15T01:49:31.556 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:31 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:49:30.213959+0000 mon.smithi055 (mon.0) 71 : audit [DBG] from='mgr.14144 172.21.15.55:0/227792247' entity='mgr.smithi055.szwngw' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-11-15T01:49:31.556 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:31 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:49:30.228411+0000 mon.smithi055 (mon.0) 72 : cluster [INF] Manager daemon smithi055.szwngw is now available 2023-11-15T01:49:31.556 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:31 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:49:30.340386+0000 mon.smithi055 (mon.0) 73 : audit [DBG] from='mgr.14144 172.21.15.55:0/227792247' entity='mgr.smithi055.szwngw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-15T01:49:31.556 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:31 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:49:30.342270+0000 mon.smithi055 (mon.0) 74 : audit [DBG] from='mgr.14144 172.21.15.55:0/227792247' entity='mgr.smithi055.szwngw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-15T01:49:31.556 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:31 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:49:30.359599+0000 mon.smithi055 (mon.0) 75 : audit [INF] from='mgr.14144 172.21.15.55:0/227792247' entity='mgr.smithi055.szwngw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi055.szwngw/mirror_snapshot_schedule"}]: dispatch 2023-11-15T01:49:31.557 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:31 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:49:30.369897+0000 mon.smithi055 (mon.0) 76 : audit [INF] from='mgr.14144 172.21.15.55:0/227792247' entity='mgr.smithi055.szwngw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi055.szwngw/trash_purge_schedule"}]: dispatch 2023-11-15T01:49:31.927 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:31.926+0000 7f7e09ead700 1 Processor -- start 2023-11-15T01:49:31.928 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:31.926+0000 7f7e09ead700 1 -- start start 2023-11-15T01:49:31.928 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:31.926+0000 7f7e09ead700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7e040703b0 0x7f7e04100d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T01:49:31.928 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:31.926+0000 7f7e09ead700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e040fc750 con 0x7f7e040703b0 2023-11-15T01:49:31.928 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:31.927+0000 7f7e037fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7e040703b0 0x7f7e04100d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:49:31.928 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:31.927+0000 7f7e037fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7e040703b0 0x7f7e04100d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:47348/0 (socket says 172.21.15.55:47348) 2023-11-15T01:49:31.931 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:31.927+0000 7f7e037fe700 1 -- 172.21.15.55:0/4242127830 learned_addr learned my addr 172.21.15.55:0/4242127830 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-11-15T01:49:31.931 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:31.927+0000 7f7e037fe700 1 -- 172.21.15.55:0/4242127830 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e04101250 con 0x7f7e040703b0 2023-11-15T01:49:31.932 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:31.927+0000 7f7e037fe700 1 --2- 172.21.15.55:0/4242127830 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7e040703b0 0x7f7e04100d60 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f7df400b5f0 tx=0x7f7df4006ee0).ready entity=mon.0 client_cookie=8b7cbdc6a72dc980 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:49:31.932 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:31.927+0000 7f7e027fc700 1 -- 172.21.15.55:0/4242127830 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7df400d600 con 0x7f7e040703b0 2023-11-15T01:49:31.932 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:31.927+0000 7f7e027fc700 1 -- 172.21.15.55:0/4242127830 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f7df400dbd0 con 0x7f7e040703b0 2023-11-15T01:49:31.933 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:31.927+0000 7f7e027fc700 1 -- 172.21.15.55:0/4242127830 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7df4011c00 con 0x7f7e040703b0 2023-11-15T01:49:31.933 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:31.928+0000 7f7e09ead700 1 -- 172.21.15.55:0/4242127830 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7e040703b0 msgr2=0x7f7e04100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:49:31.933 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:31.928+0000 7f7e09ead700 1 --2- 172.21.15.55:0/4242127830 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7e040703b0 0x7f7e04100d60 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f7df400b5f0 tx=0x7f7df4006ee0).stop 2023-11-15T01:49:31.933 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:31.928+0000 7f7e09ead700 1 -- 172.21.15.55:0/4242127830 shutdown_connections 2023-11-15T01:49:31.933 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:31.928+0000 7f7e09ead700 1 --2- 172.21.15.55:0/4242127830 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7e040703b0 0x7f7e04100d60 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:49:31.934 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:31.928+0000 7f7e09ead700 1 -- 172.21.15.55:0/4242127830 >> 172.21.15.55:0/4242127830 conn(0x7f7e040f63c0 msgr2=0x7f7e040f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:49:31.934 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:31.928+0000 7f7e09ead700 1 -- 172.21.15.55:0/4242127830 shutdown_connections 2023-11-15T01:49:31.934 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:31.928+0000 7f7e09ead700 1 -- 172.21.15.55:0/4242127830 wait complete. 2023-11-15T01:49:31.934 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:31.929+0000 7f7e09ead700 1 Processor -- start 2023-11-15T01:49:31.934 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:31.929+0000 7f7e09ead700 1 -- start start 2023-11-15T01:49:31.935 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:31.929+0000 7f7e09ead700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7e040703b0 0x7f7e0410cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:49:31.935 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:31.929+0000 7f7e09ead700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e0410d100 con 0x7f7e040703b0 2023-11-15T01:49:31.935 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:31.930+0000 7f7e037fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7e040703b0 0x7f7e0410cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:49:31.935 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:31.930+0000 7f7e037fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7e040703b0 0x7f7e0410cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:47352/0 (socket says 172.21.15.55:47352) 2023-11-15T01:49:31.935 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:31.930+0000 7f7e037fe700 1 -- 172.21.15.55:0/3248823168 learned_addr learned my addr 172.21.15.55:0/3248823168 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-11-15T01:49:31.936 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:31.930+0000 7f7e037fe700 1 -- 172.21.15.55:0/3248823168 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7df400b200 con 0x7f7e040703b0 2023-11-15T01:49:31.936 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:31.930+0000 7f7e037fe700 1 --2- 172.21.15.55:0/3248823168 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7e040703b0 0x7f7e0410cc10 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f7df400b190 tx=0x7f7df4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:49:31.936 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:31.930+0000 7f7debfff700 1 -- 172.21.15.55:0/3248823168 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7df401b410 con 0x7f7e040703b0 2023-11-15T01:49:31.936 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:31.930+0000 7f7debfff700 1 -- 172.21.15.55:0/3248823168 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f7df401b9e0 con 0x7f7e040703b0 2023-11-15T01:49:31.936 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:31.930+0000 7f7e09ead700 1 -- 172.21.15.55:0/3248823168 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e0410d300 con 0x7f7e040703b0 2023-11-15T01:49:31.936 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:31.930+0000 7f7debfff700 1 -- 172.21.15.55:0/3248823168 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7df401a880 con 0x7f7e040703b0 2023-11-15T01:49:31.937 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:31.930+0000 7f7e09ead700 1 -- 172.21.15.55:0/3248823168 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e0410d710 con 0x7f7e040703b0 2023-11-15T01:49:31.938 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:31.932+0000 7f7e09ead700 1 -- 172.21.15.55:0/3248823168 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e04045ad0 con 0x7f7e040703b0 2023-11-15T01:49:31.938 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:31.932+0000 7f7debfff700 1 -- 172.21.15.55:0/3248823168 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 11) v1 ==== 36054+0+0 (secure 0 0 0) 0x7f7df400f070 con 0x7f7e040703b0 2023-11-15T01:49:31.938 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:31.932+0000 7f7debfff700 1 --2- 172.21.15.55:0/3248823168 >> [v2:172.21.15.55:6800/2574607930,v1:172.21.15.55:6801/2574607930] conn(0x7f7df002fd50 0x7f7df0032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:49:31.938 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:31.932+0000 7f7debfff700 1 -- 172.21.15.55:0/3248823168 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f7df4041cf0 con 0x7f7e040703b0 2023-11-15T01:49:31.939 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:31.932+0000 7f7e02ffd700 1 --2- 172.21.15.55:0/3248823168 >> [v2:172.21.15.55:6800/2574607930,v1:172.21.15.55:6801/2574607930] conn(0x7f7df002fd50 0x7f7df0032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:49:31.939 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:31.933+0000 7f7e02ffd700 1 --2- 172.21.15.55:0/3248823168 >> [v2:172.21.15.55:6800/2574607930,v1:172.21.15.55:6801/2574607930] conn(0x7f7df002fd50 0x7f7df0032210 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f7df80096a0 tx=0x7f7df8006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:49:31.939 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:31.936+0000 7f7debfff700 1 -- 172.21.15.55:0/3248823168 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7df4023360 con 0x7f7e040703b0 2023-11-15T01:49:32.131 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:32.130+0000 7f7e09ead700 1 -- 172.21.15.55:0/3248823168 --> [v2:172.21.15.55:6800/2574607930,v1:172.21.15.55:6801/2574607930] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7f7e0410dff0 con 0x7f7df002fd50 2023-11-15T01:49:32.137 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:32.137+0000 7f7debfff700 1 -- 172.21.15.55:0/3248823168 <== mgr.14144 v2:172.21.15.55:6800/2574607930 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f7e0410dff0 con 0x7f7df002fd50 2023-11-15T01:49:32.138 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: Scheduled grafana update... 2023-11-15T01:49:32.140 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:32.139+0000 7f7e09ead700 1 -- 172.21.15.55:0/3248823168 >> [v2:172.21.15.55:6800/2574607930,v1:172.21.15.55:6801/2574607930] conn(0x7f7df002fd50 msgr2=0x7f7df0032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:49:32.140 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:32.139+0000 7f7e09ead700 1 --2- 172.21.15.55:0/3248823168 >> [v2:172.21.15.55:6800/2574607930,v1:172.21.15.55:6801/2574607930] conn(0x7f7df002fd50 0x7f7df0032210 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f7df80096a0 tx=0x7f7df8006b40).stop 2023-11-15T01:49:32.140 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:32.139+0000 7f7e09ead700 1 -- 172.21.15.55:0/3248823168 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7e040703b0 msgr2=0x7f7e0410cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:49:32.140 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:32.139+0000 7f7e09ead700 1 --2- 172.21.15.55:0/3248823168 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7e040703b0 0x7f7e0410cc10 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f7df400b190 tx=0x7f7df4006ee0).stop 2023-11-15T01:49:32.141 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:32.139+0000 7f7e09ead700 1 -- 172.21.15.55:0/3248823168 shutdown_connections 2023-11-15T01:49:32.141 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:32.139+0000 7f7e09ead700 1 --2- 172.21.15.55:0/3248823168 >> [v2:172.21.15.55:6800/2574607930,v1:172.21.15.55:6801/2574607930] conn(0x7f7df002fd50 0x7f7df0032210 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:49:32.142 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:32.139+0000 7f7e09ead700 1 --2- 172.21.15.55:0/3248823168 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7e040703b0 0x7f7e0410cc10 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:49:32.142 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:32.139+0000 7f7e09ead700 1 -- 172.21.15.55:0/3248823168 >> 172.21.15.55:0/3248823168 conn(0x7f7e040f63c0 msgr2=0x7f7e040f7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:49:32.142 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:32.139+0000 7f7e09ead700 1 -- 172.21.15.55:0/3248823168 shutdown_connections 2023-11-15T01:49:32.142 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:32.139+0000 7f7e09ead700 1 -- 172.21.15.55:0/3248823168 wait complete. 2023-11-15T01:49:32.437 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:32 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:49:31.224152+0000 mon.smithi055 (mon.0) 77 : cluster [DBG] mgrmap e11: smithi055.szwngw(active, since 1.0682s) 2023-11-15T01:49:32.438 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:32 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cephadm 2023-11-15T01:49:31.225416+0000 mgr.smithi055.szwngw (mgr.14144) 2 : cephadm [INF] Saving service prometheus spec with placement count:1 2023-11-15T01:49:32.438 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:32 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:49:31.227867+0000 mon.smithi055 (mon.0) 78 : audit [INF] from='mgr.14144 172.21.15.55:0/227792247' entity='mgr.smithi055.szwngw' 2023-11-15T01:49:32.438 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:32 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:49:32.137665+0000 mon.smithi055 (mon.0) 79 : audit [INF] from='mgr.14144 172.21.15.55:0/227792247' entity='mgr.smithi055.szwngw' 2023-11-15T01:49:32.492 INFO:teuthology.orchestra.run.smithi055.stderr:Deploying node-exporter service with default placement... 2023-11-15T01:49:32.493 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi055 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/0cb4fc0c-8359-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpc1yi9jse:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpra3bp1py:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply node-exporter 2023-11-15T01:49:32.908 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:32.907+0000 7f82c2098700 1 Processor -- start 2023-11-15T01:49:32.911 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:32.907+0000 7f82c2098700 1 -- start start 2023-11-15T01:49:32.911 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:32.907+0000 7f82c2098700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f82bc072540 0x7f82bc070ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T01:49:32.911 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:32.907+0000 7f82c2098700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82bc072960 con 0x7f82bc072540 2023-11-15T01:49:32.912 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:32.907+0000 7f82bb7fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f82bc072540 0x7f82bc070ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:49:32.912 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:32.907+0000 7f82bb7fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f82bc072540 0x7f82bc070ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:47358/0 (socket says 172.21.15.55:47358) 2023-11-15T01:49:32.912 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:32.907+0000 7f82bb7fe700 1 -- 172.21.15.55:0/4204961200 learned_addr learned my addr 172.21.15.55:0/4204961200 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-11-15T01:49:32.912 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:32.908+0000 7f82bb7fe700 1 -- 172.21.15.55:0/4204961200 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82bc070fd0 con 0x7f82bc072540 2023-11-15T01:49:32.912 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:32.908+0000 7f82bb7fe700 1 --2- 172.21.15.55:0/4204961200 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f82bc072540 0x7f82bc070ae0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f82ac00b5f0 tx=0x7f82ac006ee0).ready entity=mon.0 client_cookie=29b7c5471913099d server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:49:32.913 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:32.908+0000 7f82ba7fc700 1 -- 172.21.15.55:0/4204961200 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f82ac00d600 con 0x7f82bc072540 2023-11-15T01:49:32.913 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:32.908+0000 7f82ba7fc700 1 -- 172.21.15.55:0/4204961200 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f82ac00dbd0 con 0x7f82bc072540 2023-11-15T01:49:32.913 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:32.908+0000 7f82ba7fc700 1 -- 172.21.15.55:0/4204961200 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f82ac011c00 con 0x7f82bc072540 2023-11-15T01:49:32.913 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:32.908+0000 7f82c2098700 1 -- 172.21.15.55:0/4204961200 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f82bc072540 msgr2=0x7f82bc070ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:49:32.913 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:32.908+0000 7f82c2098700 1 --2- 172.21.15.55:0/4204961200 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f82bc072540 0x7f82bc070ae0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f82ac00b5f0 tx=0x7f82ac006ee0).stop 2023-11-15T01:49:32.914 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:32.908+0000 7f82c2098700 1 -- 172.21.15.55:0/4204961200 shutdown_connections 2023-11-15T01:49:32.914 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:32.908+0000 7f82c2098700 1 --2- 172.21.15.55:0/4204961200 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f82bc072540 0x7f82bc070ae0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:49:32.914 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:32.908+0000 7f82c2098700 1 -- 172.21.15.55:0/4204961200 >> 172.21.15.55:0/4204961200 conn(0x7f82bc0f74b0 msgr2=0x7f82bc0f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:49:32.914 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:32.909+0000 7f82c2098700 1 -- 172.21.15.55:0/4204961200 shutdown_connections 2023-11-15T01:49:32.914 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:32.909+0000 7f82c2098700 1 -- 172.21.15.55:0/4204961200 wait complete. 2023-11-15T01:49:32.915 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:32.909+0000 7f82c2098700 1 Processor -- start 2023-11-15T01:49:32.915 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:32.910+0000 7f82c2098700 1 -- start start 2023-11-15T01:49:32.915 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:32.910+0000 7f82c2098700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f82bc072540 0x7f82bc10cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:49:32.915 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:32.910+0000 7f82c2098700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82bc10d220 con 0x7f82bc072540 2023-11-15T01:49:32.916 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:32.910+0000 7f82bb7fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f82bc072540 0x7f82bc10cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:49:32.916 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:32.910+0000 7f82bb7fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f82bc072540 0x7f82bc10cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:47370/0 (socket says 172.21.15.55:47370) 2023-11-15T01:49:32.916 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:32.910+0000 7f82bb7fe700 1 -- 172.21.15.55:0/614759379 learned_addr learned my addr 172.21.15.55:0/614759379 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-11-15T01:49:32.916 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:32.910+0000 7f82bb7fe700 1 -- 172.21.15.55:0/614759379 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82ac00b200 con 0x7f82bc072540 2023-11-15T01:49:32.916 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:32.910+0000 7f82bb7fe700 1 --2- 172.21.15.55:0/614759379 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f82bc072540 0x7f82bc10cd30 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f82ac006c70 tx=0x7f82ac006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:49:32.918 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:32.911+0000 7f82a3fff700 1 -- 172.21.15.55:0/614759379 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f82ac00da30 con 0x7f82bc072540 2023-11-15T01:49:32.918 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:32.911+0000 7f82a3fff700 1 -- 172.21.15.55:0/614759379 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f82ac01a9f0 con 0x7f82bc072540 2023-11-15T01:49:32.918 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:32.911+0000 7f82a3fff700 1 -- 172.21.15.55:0/614759379 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f82ac01b890 con 0x7f82bc072540 2023-11-15T01:49:32.919 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:32.911+0000 7f82c2098700 1 -- 172.21.15.55:0/614759379 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82bc10d420 con 0x7f82bc072540 2023-11-15T01:49:32.919 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:32.911+0000 7f82c2098700 1 -- 172.21.15.55:0/614759379 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82bc10d830 con 0x7f82bc072540 2023-11-15T01:49:32.919 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:32.911+0000 7f82a3fff700 1 -- 172.21.15.55:0/614759379 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 11) v1 ==== 36054+0+0 (secure 0 0 0) 0x7f82ac00f070 con 0x7f82bc072540 2023-11-15T01:49:32.919 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:32.912+0000 7f82a3fff700 1 --2- 172.21.15.55:0/614759379 >> [v2:172.21.15.55:6800/2574607930,v1:172.21.15.55:6801/2574607930] conn(0x7f82a802fd70 0x7f82a8032230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:49:32.919 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:32.912+0000 7f82a3fff700 1 -- 172.21.15.55:0/614759379 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f82ac041220 con 0x7f82bc072540 2023-11-15T01:49:32.920 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:32.912+0000 7f82baffd700 1 --2- 172.21.15.55:0/614759379 >> [v2:172.21.15.55:6800/2574607930,v1:172.21.15.55:6801/2574607930] conn(0x7f82a802fd70 0x7f82a8032230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:49:32.920 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:32.912+0000 7f82baffd700 1 --2- 172.21.15.55:0/614759379 >> [v2:172.21.15.55:6800/2574607930,v1:172.21.15.55:6801/2574607930] conn(0x7f82a802fd70 0x7f82a8032230 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f82b00096a0 tx=0x7f82b0006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:49:32.920 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:32.913+0000 7f82c2098700 1 -- 172.21.15.55:0/614759379 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82bc101e10 con 0x7f82bc072540 2023-11-15T01:49:32.920 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:32.917+0000 7f82a3fff700 1 -- 172.21.15.55:0/614759379 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f82ac025460 con 0x7f82bc072540 2023-11-15T01:49:33.118 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:33.117+0000 7f82c2098700 1 -- 172.21.15.55:0/614759379 --> [v2:172.21.15.55:6800/2574607930,v1:172.21.15.55:6801/2574607930] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f82bc101fd0 con 0x7f82a802fd70 2023-11-15T01:49:33.125 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:33.124+0000 7f82a3fff700 1 -- 172.21.15.55:0/614759379 <== mgr.14144 v2:172.21.15.55:6800/2574607930 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f82bc101fd0 con 0x7f82a802fd70 2023-11-15T01:49:33.125 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: Scheduled node-exporter update... 2023-11-15T01:49:33.127 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:33.126+0000 7f82c2098700 1 -- 172.21.15.55:0/614759379 >> [v2:172.21.15.55:6800/2574607930,v1:172.21.15.55:6801/2574607930] conn(0x7f82a802fd70 msgr2=0x7f82a8032230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:49:33.128 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:33.126+0000 7f82c2098700 1 --2- 172.21.15.55:0/614759379 >> [v2:172.21.15.55:6800/2574607930,v1:172.21.15.55:6801/2574607930] conn(0x7f82a802fd70 0x7f82a8032230 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f82b00096a0 tx=0x7f82b0006b40).stop 2023-11-15T01:49:33.128 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:33.126+0000 7f82c2098700 1 -- 172.21.15.55:0/614759379 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f82bc072540 msgr2=0x7f82bc10cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:49:33.128 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:33.126+0000 7f82c2098700 1 --2- 172.21.15.55:0/614759379 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f82bc072540 0x7f82bc10cd30 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f82ac006c70 tx=0x7f82ac006ee0).stop 2023-11-15T01:49:33.128 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:33.126+0000 7f82c2098700 1 -- 172.21.15.55:0/614759379 shutdown_connections 2023-11-15T01:49:33.128 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:33.126+0000 7f82c2098700 1 --2- 172.21.15.55:0/614759379 >> [v2:172.21.15.55:6800/2574607930,v1:172.21.15.55:6801/2574607930] conn(0x7f82a802fd70 0x7f82a8032230 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:49:33.129 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:33.126+0000 7f82c2098700 1 --2- 172.21.15.55:0/614759379 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f82bc072540 0x7f82bc10cd30 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:49:33.129 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:33.126+0000 7f82c2098700 1 -- 172.21.15.55:0/614759379 >> 172.21.15.55:0/614759379 conn(0x7f82bc0f74b0 msgr2=0x7f82bc0f8fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:49:33.129 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:33.126+0000 7f82c2098700 1 -- 172.21.15.55:0/614759379 shutdown_connections 2023-11-15T01:49:33.129 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:33.126+0000 7f82c2098700 1 -- 172.21.15.55:0/614759379 wait complete. 2023-11-15T01:49:33.437 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:33 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:49:32.132180+0000 mgr.smithi055.szwngw (mgr.14144) 3 : audit [DBG] from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T01:49:33.438 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:33 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cephadm 2023-11-15T01:49:32.132727+0000 mgr.smithi055.szwngw (mgr.14144) 4 : cephadm [INF] Saving service grafana spec with placement count:1 2023-11-15T01:49:33.438 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:33 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:49:33.124871+0000 mon.smithi055 (mon.0) 80 : audit [INF] from='mgr.14144 172.21.15.55:0/227792247' entity='mgr.smithi055.szwngw' 2023-11-15T01:49:33.438 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:33 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:49:33.139321+0000 mon.smithi055 (mon.0) 81 : cluster [DBG] mgrmap e12: smithi055.szwngw(active, since 2s) 2023-11-15T01:49:33.501 INFO:teuthology.orchestra.run.smithi055.stderr:Deploying alertmanager service with default placement... 2023-11-15T01:49:33.501 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi055 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/0cb4fc0c-8359-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpc1yi9jse:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpra3bp1py:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply alertmanager 2023-11-15T01:49:33.884 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:33.882+0000 7f30f24f8700 1 Processor -- start 2023-11-15T01:49:33.884 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:33.882+0000 7f30f24f8700 1 -- start start 2023-11-15T01:49:33.884 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:33.882+0000 7f30f24f8700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f30ec0703b0 0x7f30ec100d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T01:49:33.885 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:33.882+0000 7f30f24f8700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30ec0fc750 con 0x7f30ec0703b0 2023-11-15T01:49:33.885 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:33.882+0000 7f30ebfff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f30ec0703b0 0x7f30ec100d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:49:33.885 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:33.882+0000 7f30ebfff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f30ec0703b0 0x7f30ec100d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:47378/0 (socket says 172.21.15.55:47378) 2023-11-15T01:49:33.885 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:33.882+0000 7f30ebfff700 1 -- 172.21.15.55:0/871958199 learned_addr learned my addr 172.21.15.55:0/871958199 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-11-15T01:49:33.885 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:33.883+0000 7f30ebfff700 1 -- 172.21.15.55:0/871958199 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30ec101250 con 0x7f30ec0703b0 2023-11-15T01:49:33.886 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:33.883+0000 7f30ebfff700 1 --2- 172.21.15.55:0/871958199 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f30ec0703b0 0x7f30ec100d60 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f30e000b5f0 tx=0x7f30e0006ee0).ready entity=mon.0 client_cookie=b710a36d6e730f1f server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:49:33.886 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:33.883+0000 7f30eaffd700 1 -- 172.21.15.55:0/871958199 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f30e000d600 con 0x7f30ec0703b0 2023-11-15T01:49:33.886 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:33.883+0000 7f30eaffd700 1 -- 172.21.15.55:0/871958199 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f30e000dbd0 con 0x7f30ec0703b0 2023-11-15T01:49:33.889 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:33.883+0000 7f30eaffd700 1 -- 172.21.15.55:0/871958199 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f30e0011c00 con 0x7f30ec0703b0 2023-11-15T01:49:33.889 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:33.884+0000 7f30f24f8700 1 -- 172.21.15.55:0/871958199 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f30ec0703b0 msgr2=0x7f30ec100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:49:33.889 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:33.884+0000 7f30f24f8700 1 --2- 172.21.15.55:0/871958199 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f30ec0703b0 0x7f30ec100d60 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f30e000b5f0 tx=0x7f30e0006ee0).stop 2023-11-15T01:49:33.889 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:33.884+0000 7f30f24f8700 1 -- 172.21.15.55:0/871958199 shutdown_connections 2023-11-15T01:49:33.890 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:33.884+0000 7f30f24f8700 1 --2- 172.21.15.55:0/871958199 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f30ec0703b0 0x7f30ec100d60 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:49:33.890 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:33.884+0000 7f30f24f8700 1 -- 172.21.15.55:0/871958199 >> 172.21.15.55:0/871958199 conn(0x7f30ec0f63c0 msgr2=0x7f30ec0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:49:33.890 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:33.884+0000 7f30f24f8700 1 -- 172.21.15.55:0/871958199 shutdown_connections 2023-11-15T01:49:33.890 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:33.884+0000 7f30f24f8700 1 -- 172.21.15.55:0/871958199 wait complete. 2023-11-15T01:49:33.891 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:33.884+0000 7f30f24f8700 1 Processor -- start 2023-11-15T01:49:33.891 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:33.885+0000 7f30f24f8700 1 -- start start 2023-11-15T01:49:33.891 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:33.885+0000 7f30f24f8700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f30ec0703b0 0x7f30ec10cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:49:33.891 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:33.885+0000 7f30f24f8700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30ec10d100 con 0x7f30ec0703b0 2023-11-15T01:49:33.891 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:33.885+0000 7f30ebfff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f30ec0703b0 0x7f30ec10cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:49:33.891 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:33.885+0000 7f30ebfff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f30ec0703b0 0x7f30ec10cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:47386/0 (socket says 172.21.15.55:47386) 2023-11-15T01:49:33.892 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:33.885+0000 7f30ebfff700 1 -- 172.21.15.55:0/4208382664 learned_addr learned my addr 172.21.15.55:0/4208382664 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-11-15T01:49:33.892 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:33.885+0000 7f30ebfff700 1 -- 172.21.15.55:0/4208382664 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30e000b200 con 0x7f30ec0703b0 2023-11-15T01:49:33.892 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:33.885+0000 7f30ebfff700 1 --2- 172.21.15.55:0/4208382664 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f30ec0703b0 0x7f30ec10cc10 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f30e000b190 tx=0x7f30e0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:49:33.892 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:33.886+0000 7f30e8ff9700 1 -- 172.21.15.55:0/4208382664 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f30e001b410 con 0x7f30ec0703b0 2023-11-15T01:49:33.892 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:33.886+0000 7f30f24f8700 1 -- 172.21.15.55:0/4208382664 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30ec10d300 con 0x7f30ec0703b0 2023-11-15T01:49:33.893 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:33.886+0000 7f30e8ff9700 1 -- 172.21.15.55:0/4208382664 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f30e001b9e0 con 0x7f30ec0703b0 2023-11-15T01:49:33.893 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:33.886+0000 7f30e8ff9700 1 -- 172.21.15.55:0/4208382664 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f30e001a880 con 0x7f30ec0703b0 2023-11-15T01:49:33.893 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:33.886+0000 7f30f24f8700 1 -- 172.21.15.55:0/4208382664 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30ec10d710 con 0x7f30ec0703b0 2023-11-15T01:49:33.893 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:33.887+0000 7f30e8ff9700 1 -- 172.21.15.55:0/4208382664 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f30e000f070 con 0x7f30ec0703b0 2023-11-15T01:49:33.893 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:33.887+0000 7f30e8ff9700 1 --2- 172.21.15.55:0/4208382664 >> [v2:172.21.15.55:6800/2574607930,v1:172.21.15.55:6801/2574607930] conn(0x7f30d802fe60 0x7f30d8032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:49:33.894 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:33.887+0000 7f30f24f8700 1 -- 172.21.15.55:0/4208382664 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30ec045ad0 con 0x7f30ec0703b0 2023-11-15T01:49:33.894 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:33.887+0000 7f30e8ff9700 1 -- 172.21.15.55:0/4208382664 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f30e0041450 con 0x7f30ec0703b0 2023-11-15T01:49:33.894 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:33.887+0000 7f30eb7fe700 1 --2- 172.21.15.55:0/4208382664 >> [v2:172.21.15.55:6800/2574607930,v1:172.21.15.55:6801/2574607930] conn(0x7f30d802fe60 0x7f30d8032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:49:33.894 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:33.887+0000 7f30eb7fe700 1 --2- 172.21.15.55:0/4208382664 >> [v2:172.21.15.55:6800/2574607930,v1:172.21.15.55:6801/2574607930] conn(0x7f30d802fe60 0x7f30d8032320 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f30dc0096a0 tx=0x7f30dc006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:49:33.895 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:33.892+0000 7f30e8ff9700 1 -- 172.21.15.55:0/4208382664 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f30e0016070 con 0x7f30ec0703b0 2023-11-15T01:49:34.086 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:34.085+0000 7f30f24f8700 1 -- 172.21.15.55:0/4208382664 --> [v2:172.21.15.55:6800/2574607930,v1:172.21.15.55:6801/2574607930] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7f30ec10e050 con 0x7f30d802fe60 2023-11-15T01:49:34.090 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:34.089+0000 7f30e8ff9700 1 -- 172.21.15.55:0/4208382664 <== mgr.14144 v2:172.21.15.55:6800/2574607930 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7f30ec10e050 con 0x7f30d802fe60 2023-11-15T01:49:34.090 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: Scheduled alertmanager update... 2023-11-15T01:49:34.093 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:34.091+0000 7f30f24f8700 1 -- 172.21.15.55:0/4208382664 >> [v2:172.21.15.55:6800/2574607930,v1:172.21.15.55:6801/2574607930] conn(0x7f30d802fe60 msgr2=0x7f30d8032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:49:34.093 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:34.091+0000 7f30f24f8700 1 --2- 172.21.15.55:0/4208382664 >> [v2:172.21.15.55:6800/2574607930,v1:172.21.15.55:6801/2574607930] conn(0x7f30d802fe60 0x7f30d8032320 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f30dc0096a0 tx=0x7f30dc006b40).stop 2023-11-15T01:49:34.093 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:34.091+0000 7f30f24f8700 1 -- 172.21.15.55:0/4208382664 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f30ec0703b0 msgr2=0x7f30ec10cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:49:34.093 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:34.091+0000 7f30f24f8700 1 --2- 172.21.15.55:0/4208382664 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f30ec0703b0 0x7f30ec10cc10 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f30e000b190 tx=0x7f30e0006ee0).stop 2023-11-15T01:49:34.094 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:34.091+0000 7f30f24f8700 1 -- 172.21.15.55:0/4208382664 shutdown_connections 2023-11-15T01:49:34.094 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:34.091+0000 7f30f24f8700 1 --2- 172.21.15.55:0/4208382664 >> [v2:172.21.15.55:6800/2574607930,v1:172.21.15.55:6801/2574607930] conn(0x7f30d802fe60 0x7f30d8032320 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:49:34.094 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:34.091+0000 7f30f24f8700 1 --2- 172.21.15.55:0/4208382664 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f30ec0703b0 0x7f30ec10cc10 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:49:34.094 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:34.091+0000 7f30f24f8700 1 -- 172.21.15.55:0/4208382664 >> 172.21.15.55:0/4208382664 conn(0x7f30ec0f63c0 msgr2=0x7f30ec0f7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:49:34.095 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:34.092+0000 7f30f24f8700 1 -- 172.21.15.55:0/4208382664 shutdown_connections 2023-11-15T01:49:34.095 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:34.092+0000 7f30f24f8700 1 -- 172.21.15.55:0/4208382664 wait complete. 2023-11-15T01:49:34.370 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:34 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:49:33.119224+0000 mgr.smithi055.szwngw (mgr.14144) 5 : audit [DBG] from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T01:49:34.371 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:34 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cephadm 2023-11-15T01:49:33.119932+0000 mgr.smithi055.szwngw (mgr.14144) 6 : cephadm [INF] Saving service node-exporter spec with placement * 2023-11-15T01:49:34.371 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:34 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:49:34.090264+0000 mon.smithi055 (mon.0) 82 : audit [INF] from='mgr.14144 172.21.15.55:0/227792247' entity='mgr.smithi055.szwngw' 2023-11-15T01:49:34.435 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi055 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/0cb4fc0c-8359-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpc1yi9jse:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpra3bp1py:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/cephadm/container_init True --force 2023-11-15T01:49:34.836 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:34.835+0000 7f9311bd3700 1 Processor -- start 2023-11-15T01:49:34.837 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:34.835+0000 7f9311bd3700 1 -- start start 2023-11-15T01:49:34.837 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:34.835+0000 7f9311bd3700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f930c0fa9f0 0x7f930c0fae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T01:49:34.837 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:34.835+0000 7f9311bd3700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f930c0fb300 con 0x7f930c0fa9f0 2023-11-15T01:49:34.840 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:34.835+0000 7f9310bd1700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f930c0fa9f0 0x7f930c0fae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:49:34.840 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:34.835+0000 7f9310bd1700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f930c0fa9f0 0x7f930c0fae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:47394/0 (socket says 172.21.15.55:47394) 2023-11-15T01:49:34.840 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:34.835+0000 7f9310bd1700 1 -- 172.21.15.55:0/909445613 learned_addr learned my addr 172.21.15.55:0/909445613 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-11-15T01:49:34.841 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:34.836+0000 7f9310bd1700 1 -- 172.21.15.55:0/909445613 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f930c0fb440 con 0x7f930c0fa9f0 2023-11-15T01:49:34.841 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:34.836+0000 7f9310bd1700 1 --2- 172.21.15.55:0/909445613 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f930c0fa9f0 0x7f930c0fae10 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f930000b5f0 tx=0x7f9300006ee0).ready entity=mon.0 client_cookie=43306e2a3460a1a8 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:49:34.841 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:34.836+0000 7f930b7fe700 1 -- 172.21.15.55:0/909445613 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f930000d600 con 0x7f930c0fa9f0 2023-11-15T01:49:34.841 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:34.836+0000 7f930b7fe700 1 -- 172.21.15.55:0/909445613 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f930000dbd0 con 0x7f930c0fa9f0 2023-11-15T01:49:34.841 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:34.836+0000 7f930b7fe700 1 -- 172.21.15.55:0/909445613 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9300011c00 con 0x7f930c0fa9f0 2023-11-15T01:49:34.841 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:34.837+0000 7f9311bd3700 1 -- 172.21.15.55:0/909445613 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f930c0fa9f0 msgr2=0x7f930c0fae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:49:34.842 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:34.837+0000 7f9311bd3700 1 --2- 172.21.15.55:0/909445613 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f930c0fa9f0 0x7f930c0fae10 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f930000b5f0 tx=0x7f9300006ee0).stop 2023-11-15T01:49:34.842 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:34.837+0000 7f9311bd3700 1 -- 172.21.15.55:0/909445613 shutdown_connections 2023-11-15T01:49:34.842 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:34.837+0000 7f9311bd3700 1 --2- 172.21.15.55:0/909445613 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f930c0fa9f0 0x7f930c0fae10 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:49:34.842 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:34.837+0000 7f9311bd3700 1 -- 172.21.15.55:0/909445613 >> 172.21.15.55:0/909445613 conn(0x7f930c0f6210 msgr2=0x7f930c0f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:49:34.842 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:34.837+0000 7f9311bd3700 1 -- 172.21.15.55:0/909445613 shutdown_connections 2023-11-15T01:49:34.843 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:34.837+0000 7f9311bd3700 1 -- 172.21.15.55:0/909445613 wait complete. 2023-11-15T01:49:34.843 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:34.837+0000 7f9311bd3700 1 Processor -- start 2023-11-15T01:49:34.843 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:34.837+0000 7f9311bd3700 1 -- start start 2023-11-15T01:49:34.843 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:34.838+0000 7f9311bd3700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f930c0fa9f0 0x7f930c100c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:49:34.843 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:34.838+0000 7f9311bd3700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f930c0ff100 con 0x7f930c0fa9f0 2023-11-15T01:49:34.844 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:34.838+0000 7f9310bd1700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f930c0fa9f0 0x7f930c100c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:49:34.844 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:34.838+0000 7f9310bd1700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f930c0fa9f0 0x7f930c100c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:47404/0 (socket says 172.21.15.55:47404) 2023-11-15T01:49:34.844 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:34.838+0000 7f9310bd1700 1 -- 172.21.15.55:0/4119977132 learned_addr learned my addr 172.21.15.55:0/4119977132 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-11-15T01:49:34.844 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:34.839+0000 7f9310bd1700 1 -- 172.21.15.55:0/4119977132 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f930000b200 con 0x7f930c0fa9f0 2023-11-15T01:49:34.844 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:34.839+0000 7f9310bd1700 1 --2- 172.21.15.55:0/4119977132 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f930c0fa9f0 0x7f930c100c80 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f930000b1d0 tx=0x7f9300006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:49:34.844 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:34.839+0000 7f93097fa700 1 -- 172.21.15.55:0/4119977132 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f930001b410 con 0x7f930c0fa9f0 2023-11-15T01:49:34.845 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:34.839+0000 7f93097fa700 1 -- 172.21.15.55:0/4119977132 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f930001b9e0 con 0x7f930c0fa9f0 2023-11-15T01:49:34.845 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:34.839+0000 7f9311bd3700 1 -- 172.21.15.55:0/4119977132 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f930c0ff2a0 con 0x7f930c0fa9f0 2023-11-15T01:49:34.845 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:34.839+0000 7f93097fa700 1 -- 172.21.15.55:0/4119977132 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f930001a840 con 0x7f930c0fa9f0 2023-11-15T01:49:34.845 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:34.839+0000 7f9311bd3700 1 -- 172.21.15.55:0/4119977132 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f930c0ff6b0 con 0x7f930c0fa9f0 2023-11-15T01:49:34.846 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:34.840+0000 7f93097fa700 1 -- 172.21.15.55:0/4119977132 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f930000f070 con 0x7f930c0fa9f0 2023-11-15T01:49:34.846 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:34.840+0000 7f93097fa700 1 --2- 172.21.15.55:0/4119977132 >> [v2:172.21.15.55:6800/2574607930,v1:172.21.15.55:6801/2574607930] conn(0x7f92f002fe60 0x7f92f0032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:49:34.847 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:34.840+0000 7f93097fa700 1 -- 172.21.15.55:0/4119977132 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f9300041410 con 0x7f930c0fa9f0 2023-11-15T01:49:34.847 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:34.840+0000 7f930bfff700 1 --2- 172.21.15.55:0/4119977132 >> [v2:172.21.15.55:6800/2574607930,v1:172.21.15.55:6801/2574607930] conn(0x7f92f002fe60 0x7f92f0032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:49:34.847 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:34.841+0000 7f930bfff700 1 --2- 172.21.15.55:0/4119977132 >> [v2:172.21.15.55:6800/2574607930,v1:172.21.15.55:6801/2574607930] conn(0x7f92f002fe60 0x7f92f0032320 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f92f80096a0 tx=0x7f92f8006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:49:34.847 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:34.841+0000 7f9311bd3700 1 -- 172.21.15.55:0/4119977132 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f930c059100 con 0x7f930c0fa9f0 2023-11-15T01:49:34.847 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:34.845+0000 7f93097fa700 1 -- 172.21.15.55:0/4119977132 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9300016070 con 0x7f930c0fa9f0 2023-11-15T01:49:35.011 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:35.010+0000 7f9311bd3700 1 -- 172.21.15.55:0/4119977132 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f930c1047a0 con 0x7f930c0fa9f0 2023-11-15T01:49:35.014 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:35.013+0000 7f93097fa700 1 -- 172.21.15.55:0/4119977132 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v8)=0 v8) v1 ==== 142+0+0 (secure 0 0 0) 0x7f9300025460 con 0x7f930c0fa9f0 2023-11-15T01:49:35.018 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:35.015+0000 7f9311bd3700 1 -- 172.21.15.55:0/4119977132 >> [v2:172.21.15.55:6800/2574607930,v1:172.21.15.55:6801/2574607930] conn(0x7f92f002fe60 msgr2=0x7f92f0032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:49:35.018 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:35.015+0000 7f9311bd3700 1 --2- 172.21.15.55:0/4119977132 >> [v2:172.21.15.55:6800/2574607930,v1:172.21.15.55:6801/2574607930] conn(0x7f92f002fe60 0x7f92f0032320 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f92f80096a0 tx=0x7f92f8006b40).stop 2023-11-15T01:49:35.018 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:35.015+0000 7f9311bd3700 1 -- 172.21.15.55:0/4119977132 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f930c0fa9f0 msgr2=0x7f930c100c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:49:35.018 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:35.015+0000 7f9311bd3700 1 --2- 172.21.15.55:0/4119977132 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f930c0fa9f0 0x7f930c100c80 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f930000b1d0 tx=0x7f9300006ee0).stop 2023-11-15T01:49:35.019 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:35.015+0000 7f9311bd3700 1 -- 172.21.15.55:0/4119977132 shutdown_connections 2023-11-15T01:49:35.019 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:35.016+0000 7f9311bd3700 1 --2- 172.21.15.55:0/4119977132 >> [v2:172.21.15.55:6800/2574607930,v1:172.21.15.55:6801/2574607930] conn(0x7f92f002fe60 0x7f92f0032320 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:49:35.019 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:35.016+0000 7f9311bd3700 1 --2- 172.21.15.55:0/4119977132 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f930c0fa9f0 0x7f930c100c80 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:49:35.019 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:35.016+0000 7f9311bd3700 1 -- 172.21.15.55:0/4119977132 >> 172.21.15.55:0/4119977132 conn(0x7f930c0f6210 msgr2=0x7f930c070010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:49:35.020 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:35.016+0000 7f9311bd3700 1 -- 172.21.15.55:0/4119977132 shutdown_connections 2023-11-15T01:49:35.020 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:35.016+0000 7f9311bd3700 1 -- 172.21.15.55:0/4119977132 wait complete. 2023-11-15T01:49:35.320 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:35 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:49:34.087048+0000 mgr.smithi055.szwngw (mgr.14144) 7 : audit [DBG] from='client.14156 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T01:49:35.320 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:35 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cephadm 2023-11-15T01:49:34.087646+0000 mgr.smithi055.szwngw (mgr.14144) 8 : cephadm [INF] Saving service alertmanager spec with placement count:1 2023-11-15T01:49:35.321 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:35 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:49:35.014385+0000 mon.smithi055 (mon.0) 83 : audit [INF] from='client.? 172.21.15.55:0/4119977132' entity='client.admin' 2023-11-15T01:49:35.376 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi055 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/0cb4fc0c-8359-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpc1yi9jse:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpra3bp1py:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/dashboard/ssl_server_port 8443 2023-11-15T01:49:35.755 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:35.754+0000 7fa9deb58700 1 Processor -- start 2023-11-15T01:49:35.756 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:35.754+0000 7fa9deb58700 1 -- start start 2023-11-15T01:49:35.756 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:35.754+0000 7fa9deb58700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa9d80703b0 0x7fa9d8100d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T01:49:35.756 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:35.754+0000 7fa9deb58700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9d80fc750 con 0x7fa9d80703b0 2023-11-15T01:49:35.756 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:35.754+0000 7fa9dc8f4700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa9d80703b0 0x7fa9d8100d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:49:35.757 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:35.754+0000 7fa9dc8f4700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa9d80703b0 0x7fa9d8100d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:47418/0 (socket says 172.21.15.55:47418) 2023-11-15T01:49:35.757 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:35.754+0000 7fa9dc8f4700 1 -- 172.21.15.55:0/3473476865 learned_addr learned my addr 172.21.15.55:0/3473476865 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-11-15T01:49:35.760 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:35.755+0000 7fa9dc8f4700 1 -- 172.21.15.55:0/3473476865 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9d8101250 con 0x7fa9d80703b0 2023-11-15T01:49:35.760 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:35.755+0000 7fa9dc8f4700 1 --2- 172.21.15.55:0/3473476865 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa9d80703b0 0x7fa9d8100d60 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fa9cc00b5f0 tx=0x7fa9cc006ee0).ready entity=mon.0 client_cookie=e4ab1757266a00d7 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:49:35.761 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:35.755+0000 7fa9d77fe700 1 -- 172.21.15.55:0/3473476865 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa9cc00d600 con 0x7fa9d80703b0 2023-11-15T01:49:35.761 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:35.755+0000 7fa9d77fe700 1 -- 172.21.15.55:0/3473476865 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fa9cc00dbd0 con 0x7fa9d80703b0 2023-11-15T01:49:35.761 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:35.755+0000 7fa9d77fe700 1 -- 172.21.15.55:0/3473476865 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa9cc011c00 con 0x7fa9d80703b0 2023-11-15T01:49:35.761 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:35.756+0000 7fa9deb58700 1 -- 172.21.15.55:0/3473476865 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa9d80703b0 msgr2=0x7fa9d8100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:49:35.761 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:35.756+0000 7fa9deb58700 1 --2- 172.21.15.55:0/3473476865 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa9d80703b0 0x7fa9d8100d60 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fa9cc00b5f0 tx=0x7fa9cc006ee0).stop 2023-11-15T01:49:35.762 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:35.756+0000 7fa9deb58700 1 -- 172.21.15.55:0/3473476865 shutdown_connections 2023-11-15T01:49:35.762 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:35.756+0000 7fa9deb58700 1 --2- 172.21.15.55:0/3473476865 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa9d80703b0 0x7fa9d8100d60 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:49:35.762 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:35.756+0000 7fa9deb58700 1 -- 172.21.15.55:0/3473476865 >> 172.21.15.55:0/3473476865 conn(0x7fa9d80f63c0 msgr2=0x7fa9d80f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:49:35.762 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:35.756+0000 7fa9deb58700 1 -- 172.21.15.55:0/3473476865 shutdown_connections 2023-11-15T01:49:35.762 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:35.756+0000 7fa9deb58700 1 -- 172.21.15.55:0/3473476865 wait complete. 2023-11-15T01:49:35.763 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:35.756+0000 7fa9deb58700 1 Processor -- start 2023-11-15T01:49:35.763 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:35.757+0000 7fa9deb58700 1 -- start start 2023-11-15T01:49:35.763 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:35.757+0000 7fa9deb58700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa9d80703b0 0x7fa9d810cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:49:35.763 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:35.757+0000 7fa9deb58700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9d810d0f0 con 0x7fa9d80703b0 2023-11-15T01:49:35.763 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:35.757+0000 7fa9dc8f4700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa9d80703b0 0x7fa9d810cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:49:35.764 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:35.757+0000 7fa9dc8f4700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa9d80703b0 0x7fa9d810cc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:47432/0 (socket says 172.21.15.55:47432) 2023-11-15T01:49:35.764 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:35.757+0000 7fa9dc8f4700 1 -- 172.21.15.55:0/2441354812 learned_addr learned my addr 172.21.15.55:0/2441354812 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-11-15T01:49:35.764 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:35.757+0000 7fa9dc8f4700 1 -- 172.21.15.55:0/2441354812 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9cc00b200 con 0x7fa9d80703b0 2023-11-15T01:49:35.764 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:35.758+0000 7fa9dc8f4700 1 --2- 172.21.15.55:0/2441354812 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa9d80703b0 0x7fa9d810cc00 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fa9cc00b190 tx=0x7fa9cc006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:49:35.764 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:35.758+0000 7fa9d57fa700 1 -- 172.21.15.55:0/2441354812 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa9cc01b410 con 0x7fa9d80703b0 2023-11-15T01:49:35.765 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:35.758+0000 7fa9d57fa700 1 -- 172.21.15.55:0/2441354812 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fa9cc01b9e0 con 0x7fa9d80703b0 2023-11-15T01:49:35.765 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:35.758+0000 7fa9deb58700 1 -- 172.21.15.55:0/2441354812 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9d810d2f0 con 0x7fa9d80703b0 2023-11-15T01:49:35.765 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:35.758+0000 7fa9d57fa700 1 -- 172.21.15.55:0/2441354812 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa9cc01a880 con 0x7fa9d80703b0 2023-11-15T01:49:35.765 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:35.758+0000 7fa9deb58700 1 -- 172.21.15.55:0/2441354812 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9d810d700 con 0x7fa9d80703b0 2023-11-15T01:49:35.765 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:35.759+0000 7fa9d57fa700 1 -- 172.21.15.55:0/2441354812 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7fa9cc00f070 con 0x7fa9d80703b0 2023-11-15T01:49:35.766 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:35.759+0000 7fa9d57fa700 1 --2- 172.21.15.55:0/2441354812 >> [v2:172.21.15.55:6800/2574607930,v1:172.21.15.55:6801/2574607930] conn(0x7fa9bc02fe60 0x7fa9bc032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:49:35.766 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:35.759+0000 7fa9d57fa700 1 -- 172.21.15.55:0/2441354812 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fa9cc0414a0 con 0x7fa9d80703b0 2023-11-15T01:49:35.766 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:35.759+0000 7fa9deb58700 1 -- 172.21.15.55:0/2441354812 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9d8045ad0 con 0x7fa9d80703b0 2023-11-15T01:49:35.766 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:35.759+0000 7fa9d7fff700 1 --2- 172.21.15.55:0/2441354812 >> [v2:172.21.15.55:6800/2574607930,v1:172.21.15.55:6801/2574607930] conn(0x7fa9bc02fe60 0x7fa9bc032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:49:35.766 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:35.759+0000 7fa9d7fff700 1 --2- 172.21.15.55:0/2441354812 >> [v2:172.21.15.55:6800/2574607930,v1:172.21.15.55:6801/2574607930] conn(0x7fa9bc02fe60 0x7fa9bc032320 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fa9c40096a0 tx=0x7fa9c4006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:49:35.767 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:35.764+0000 7fa9d57fa700 1 -- 172.21.15.55:0/2441354812 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa9cc016080 con 0x7fa9d80703b0 2023-11-15T01:49:35.928 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:35.927+0000 7fa9deb58700 1 -- 172.21.15.55:0/2441354812 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7fa9d8101db0 con 0x7fa9d80703b0 2023-11-15T01:49:35.933 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:35.933+0000 7fa9d57fa700 1 -- 172.21.15.55:0/2441354812 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v9)=0 v9) v1 ==== 130+0+0 (secure 0 0 0) 0x7fa9cc019e60 con 0x7fa9d80703b0 2023-11-15T01:49:35.936 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:35.935+0000 7fa9deb58700 1 -- 172.21.15.55:0/2441354812 >> [v2:172.21.15.55:6800/2574607930,v1:172.21.15.55:6801/2574607930] conn(0x7fa9bc02fe60 msgr2=0x7fa9bc032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:49:35.936 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:35.935+0000 7fa9deb58700 1 --2- 172.21.15.55:0/2441354812 >> [v2:172.21.15.55:6800/2574607930,v1:172.21.15.55:6801/2574607930] conn(0x7fa9bc02fe60 0x7fa9bc032320 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fa9c40096a0 tx=0x7fa9c4006b40).stop 2023-11-15T01:49:35.937 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:35.935+0000 7fa9deb58700 1 -- 172.21.15.55:0/2441354812 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa9d80703b0 msgr2=0x7fa9d810cc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:49:35.937 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:35.935+0000 7fa9deb58700 1 --2- 172.21.15.55:0/2441354812 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa9d80703b0 0x7fa9d810cc00 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fa9cc00b190 tx=0x7fa9cc006ee0).stop 2023-11-15T01:49:35.937 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:35.935+0000 7fa9deb58700 1 -- 172.21.15.55:0/2441354812 shutdown_connections 2023-11-15T01:49:35.937 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:35.935+0000 7fa9deb58700 1 --2- 172.21.15.55:0/2441354812 >> [v2:172.21.15.55:6800/2574607930,v1:172.21.15.55:6801/2574607930] conn(0x7fa9bc02fe60 0x7fa9bc032320 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:49:35.937 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:35.935+0000 7fa9deb58700 1 --2- 172.21.15.55:0/2441354812 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa9d80703b0 0x7fa9d810cc00 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:49:35.937 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:35.935+0000 7fa9deb58700 1 -- 172.21.15.55:0/2441354812 >> 172.21.15.55:0/2441354812 conn(0x7fa9d80f63c0 msgr2=0x7fa9d80f7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:49:35.938 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:35.935+0000 7fa9deb58700 1 -- 172.21.15.55:0/2441354812 shutdown_connections 2023-11-15T01:49:35.938 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:35.935+0000 7fa9deb58700 1 -- 172.21.15.55:0/2441354812 wait complete. 2023-11-15T01:49:36.276 INFO:teuthology.orchestra.run.smithi055.stderr:Enabling the dashboard module... 2023-11-15T01:49:36.276 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi055 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/0cb4fc0c-8359-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpc1yi9jse:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpra3bp1py:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable dashboard 2023-11-15T01:49:36.666 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:36.664+0000 7f32a5c36700 1 Processor -- start 2023-11-15T01:49:36.666 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:36.664+0000 7f32a5c36700 1 -- start start 2023-11-15T01:49:36.666 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:36.664+0000 7f32a5c36700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f32a00703f0 0x7f32a0100b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T01:49:36.667 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:36.664+0000 7f32a5c36700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32a00fc560 con 0x7f32a00703f0 2023-11-15T01:49:36.667 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:36.664+0000 7f32a4c34700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f32a00703f0 0x7f32a0100b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:49:36.667 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:36.665+0000 7f32a4c34700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f32a00703f0 0x7f32a0100b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:47446/0 (socket says 172.21.15.55:47446) 2023-11-15T01:49:36.667 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:36.665+0000 7f32a4c34700 1 -- 172.21.15.55:0/267038346 learned_addr learned my addr 172.21.15.55:0/267038346 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-11-15T01:49:36.667 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:36.665+0000 7f32a4c34700 1 -- 172.21.15.55:0/267038346 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32a0101060 con 0x7f32a00703f0 2023-11-15T01:49:36.668 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:36.665+0000 7f32a4c34700 1 --2- 172.21.15.55:0/267038346 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f32a00703f0 0x7f32a0100b70 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f329400b5f0 tx=0x7f3294006ee0).ready entity=mon.0 client_cookie=a45864f3bd387ea7 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:49:36.668 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:36.665+0000 7f329f7fe700 1 -- 172.21.15.55:0/267038346 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f329400d600 con 0x7f32a00703f0 2023-11-15T01:49:36.670 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:36.665+0000 7f329f7fe700 1 -- 172.21.15.55:0/267038346 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f329400dbd0 con 0x7f32a00703f0 2023-11-15T01:49:36.670 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:36.665+0000 7f329f7fe700 1 -- 172.21.15.55:0/267038346 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3294011c00 con 0x7f32a00703f0 2023-11-15T01:49:36.670 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:36.666+0000 7f32a5c36700 1 -- 172.21.15.55:0/267038346 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f32a00703f0 msgr2=0x7f32a0100b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:49:36.671 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:36.666+0000 7f32a5c36700 1 --2- 172.21.15.55:0/267038346 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f32a00703f0 0x7f32a0100b70 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f329400b5f0 tx=0x7f3294006ee0).stop 2023-11-15T01:49:36.671 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:36.666+0000 7f32a5c36700 1 -- 172.21.15.55:0/267038346 shutdown_connections 2023-11-15T01:49:36.671 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:36.666+0000 7f32a5c36700 1 --2- 172.21.15.55:0/267038346 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f32a00703f0 0x7f32a0100b70 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:49:36.671 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:36.666+0000 7f32a5c36700 1 -- 172.21.15.55:0/267038346 >> 172.21.15.55:0/267038346 conn(0x7f32a00f6210 msgr2=0x7f32a00f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:49:36.671 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:36.666+0000 7f32a5c36700 1 -- 172.21.15.55:0/267038346 shutdown_connections 2023-11-15T01:49:36.672 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:36.666+0000 7f32a5c36700 1 -- 172.21.15.55:0/267038346 wait complete. 2023-11-15T01:49:36.672 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:36.666+0000 7f32a5c36700 1 Processor -- start 2023-11-15T01:49:36.672 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:36.667+0000 7f32a5c36700 1 -- start start 2023-11-15T01:49:36.672 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:36.667+0000 7f32a5c36700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f32a00703f0 0x7f32a010ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:49:36.672 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:36.667+0000 7f32a5c36700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32a010cf00 con 0x7f32a00703f0 2023-11-15T01:49:36.673 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:36.667+0000 7f32a4c34700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f32a00703f0 0x7f32a010ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:49:36.673 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:36.667+0000 7f32a4c34700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f32a00703f0 0x7f32a010ca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:47448/0 (socket says 172.21.15.55:47448) 2023-11-15T01:49:36.673 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:36.667+0000 7f32a4c34700 1 -- 172.21.15.55:0/4105356394 learned_addr learned my addr 172.21.15.55:0/4105356394 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-11-15T01:49:36.673 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:36.667+0000 7f32a4c34700 1 -- 172.21.15.55:0/4105356394 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f329400b200 con 0x7f32a00703f0 2023-11-15T01:49:36.673 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:36.667+0000 7f32a4c34700 1 --2- 172.21.15.55:0/4105356394 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f32a00703f0 0x7f32a010ca10 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f329400b190 tx=0x7f3294006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:49:36.673 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:36.668+0000 7f329d7fa700 1 -- 172.21.15.55:0/4105356394 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f329401b410 con 0x7f32a00703f0 2023-11-15T01:49:36.674 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:36.668+0000 7f329d7fa700 1 -- 172.21.15.55:0/4105356394 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f329401b9e0 con 0x7f32a00703f0 2023-11-15T01:49:36.674 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:36.668+0000 7f32a5c36700 1 -- 172.21.15.55:0/4105356394 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32a010d100 con 0x7f32a00703f0 2023-11-15T01:49:36.674 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:36.668+0000 7f329d7fa700 1 -- 172.21.15.55:0/4105356394 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f329401a880 con 0x7f32a00703f0 2023-11-15T01:49:36.674 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:36.668+0000 7f32a5c36700 1 -- 172.21.15.55:0/4105356394 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32a010d510 con 0x7f32a00703f0 2023-11-15T01:49:36.674 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:36.669+0000 7f329d7fa700 1 -- 172.21.15.55:0/4105356394 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f329400f070 con 0x7f32a00703f0 2023-11-15T01:49:36.675 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:36.669+0000 7f329d7fa700 1 --2- 172.21.15.55:0/4105356394 >> [v2:172.21.15.55:6800/2574607930,v1:172.21.15.55:6801/2574607930] conn(0x7f328402fe60 0x7f3284032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:49:36.675 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:36.669+0000 7f329d7fa700 1 -- 172.21.15.55:0/4105356394 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f3294041470 con 0x7f32a00703f0 2023-11-15T01:49:36.675 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:36.669+0000 7f329ffff700 1 --2- 172.21.15.55:0/4105356394 >> [v2:172.21.15.55:6800/2574607930,v1:172.21.15.55:6801/2574607930] conn(0x7f328402fe60 0x7f3284032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:49:36.675 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:36.669+0000 7f329ffff700 1 --2- 172.21.15.55:0/4105356394 >> [v2:172.21.15.55:6800/2574607930,v1:172.21.15.55:6801/2574607930] conn(0x7f328402fe60 0x7f3284032320 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f328c0096a0 tx=0x7f328c006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:49:36.675 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:36.669+0000 7f32a5c36700 1 -- 172.21.15.55:0/4105356394 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32a0059100 con 0x7f32a00703f0 2023-11-15T01:49:36.676 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:36.674+0000 7f329d7fa700 1 -- 172.21.15.55:0/4105356394 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3294016070 con 0x7f32a00703f0 2023-11-15T01:49:36.879 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:36.878+0000 7f32a5c36700 1 -- 172.21.15.55:0/4105356394 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f32a0101c20 con 0x7f32a00703f0 2023-11-15T01:49:36.934 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:36.933+0000 7f329d7fa700 1 -- 172.21.15.55:0/4105356394 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f3294025980 con 0x7f32a00703f0 2023-11-15T01:49:36.935 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:36.934+0000 7f329d7fa700 1 -- 172.21.15.55:0/4105356394 <== mon.0 v2:172.21.15.55:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v13) v1 ==== 88+0+0 (secure 0 0 0) 0x7f3294010360 con 0x7f32a00703f0 2023-11-15T01:49:36.938 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:36.936+0000 7f32a5c36700 1 -- 172.21.15.55:0/4105356394 >> [v2:172.21.15.55:6800/2574607930,v1:172.21.15.55:6801/2574607930] conn(0x7f328402fe60 msgr2=0x7f3284032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:49:36.938 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:36.936+0000 7f32a5c36700 1 --2- 172.21.15.55:0/4105356394 >> [v2:172.21.15.55:6800/2574607930,v1:172.21.15.55:6801/2574607930] conn(0x7f328402fe60 0x7f3284032320 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f328c0096a0 tx=0x7f328c006b40).stop 2023-11-15T01:49:36.938 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:36.936+0000 7f32a5c36700 1 -- 172.21.15.55:0/4105356394 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f32a00703f0 msgr2=0x7f32a010ca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:49:36.939 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:36.936+0000 7f32a5c36700 1 --2- 172.21.15.55:0/4105356394 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f32a00703f0 0x7f32a010ca10 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f329400b190 tx=0x7f3294006ee0).stop 2023-11-15T01:49:36.939 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:36.936+0000 7f32a5c36700 1 -- 172.21.15.55:0/4105356394 shutdown_connections 2023-11-15T01:49:36.939 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:36.936+0000 7f32a5c36700 1 --2- 172.21.15.55:0/4105356394 >> [v2:172.21.15.55:6800/2574607930,v1:172.21.15.55:6801/2574607930] conn(0x7f328402fe60 0x7f3284032320 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:49:36.939 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:36.936+0000 7f32a5c36700 1 --2- 172.21.15.55:0/4105356394 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f32a00703f0 0x7f32a010ca10 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:49:36.940 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:36.936+0000 7f32a5c36700 1 -- 172.21.15.55:0/4105356394 >> 172.21.15.55:0/4105356394 conn(0x7f32a00f6210 msgr2=0x7f32a00f7ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:49:36.940 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:36.936+0000 7f32a5c36700 1 -- 172.21.15.55:0/4105356394 shutdown_connections 2023-11-15T01:49:36.940 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:36.937+0000 7f32a5c36700 1 -- 172.21.15.55:0/4105356394 wait complete. 2023-11-15T01:49:37.228 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:36 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:49:35.933858+0000 mon.smithi055 (mon.0) 84 : audit [INF] from='client.? 172.21.15.55:0/2441354812' entity='client.admin' 2023-11-15T01:49:37.228 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:36 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:49:36.879377+0000 mon.smithi055 (mon.0) 85 : audit [INF] from='client.? 172.21.15.55:0/4105356394' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2023-11-15T01:49:37.292 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi055 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/0cb4fc0c-8359-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpc1yi9jse:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpra3bp1py:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2023-11-15T01:49:37.700 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:37.698+0000 7f3205613700 1 Processor -- start 2023-11-15T01:49:37.701 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:37.698+0000 7f3205613700 1 -- start start 2023-11-15T01:49:37.701 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:37.698+0000 7f3205613700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f32000703d0 0x7f32000fe940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T01:49:37.701 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:37.698+0000 7f3205613700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32000fee30 con 0x7f32000703d0 2023-11-15T01:49:37.701 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:37.699+0000 7f31feffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f32000703d0 0x7f32000fe940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:49:37.702 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:37.699+0000 7f31feffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f32000703d0 0x7f32000fe940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:47464/0 (socket says 172.21.15.55:47464) 2023-11-15T01:49:37.702 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:37.699+0000 7f31feffd700 1 -- 172.21.15.55:0/477152713 learned_addr learned my addr 172.21.15.55:0/477152713 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-11-15T01:49:37.702 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:37.699+0000 7f31feffd700 1 -- 172.21.15.55:0/477152713 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32000fef70 con 0x7f32000703d0 2023-11-15T01:49:37.702 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:37.699+0000 7f31feffd700 1 --2- 172.21.15.55:0/477152713 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f32000703d0 0x7f32000fe940 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f31f000b5f0 tx=0x7f31f0006ee0).ready entity=mon.0 client_cookie=3577cccbc26e511d server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:49:37.702 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:37.699+0000 7f31fdffb700 1 -- 172.21.15.55:0/477152713 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f31f000d600 con 0x7f32000703d0 2023-11-15T01:49:37.702 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:37.699+0000 7f31fdffb700 1 -- 172.21.15.55:0/477152713 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f31f000dbd0 con 0x7f32000703d0 2023-11-15T01:49:37.703 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:37.699+0000 7f31fdffb700 1 -- 172.21.15.55:0/477152713 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f31f0011c00 con 0x7f32000703d0 2023-11-15T01:49:37.704 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:37.700+0000 7f3205613700 1 -- 172.21.15.55:0/477152713 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f32000703d0 msgr2=0x7f32000fe940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:49:37.704 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:37.700+0000 7f3205613700 1 --2- 172.21.15.55:0/477152713 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f32000703d0 0x7f32000fe940 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f31f000b5f0 tx=0x7f31f0006ee0).stop 2023-11-15T01:49:37.704 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:37.700+0000 7f3205613700 1 -- 172.21.15.55:0/477152713 shutdown_connections 2023-11-15T01:49:37.704 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:37.700+0000 7f3205613700 1 --2- 172.21.15.55:0/477152713 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f32000703d0 0x7f32000fe940 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:49:37.704 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:37.700+0000 7f3205613700 1 -- 172.21.15.55:0/477152713 >> 172.21.15.55:0/477152713 conn(0x7f32000f63c0 msgr2=0x7f32000f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:49:37.705 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:37.703+0000 7f3205613700 1 -- 172.21.15.55:0/477152713 shutdown_connections 2023-11-15T01:49:37.705 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:37.703+0000 7f3205613700 1 -- 172.21.15.55:0/477152713 wait complete. 2023-11-15T01:49:37.705 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:37.703+0000 7f3205613700 1 Processor -- start 2023-11-15T01:49:37.706 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:37.704+0000 7f3205613700 1 -- start start 2023-11-15T01:49:37.707 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:37.704+0000 7f3205613700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f32000703d0 0x7f320010aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:49:37.707 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:37.704+0000 7f3205613700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f320010af20 con 0x7f32000703d0 2023-11-15T01:49:37.707 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:37.704+0000 7f31feffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f32000703d0 0x7f320010aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:49:37.707 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:37.704+0000 7f31feffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f32000703d0 0x7f320010aa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:47478/0 (socket says 172.21.15.55:47478) 2023-11-15T01:49:37.707 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:37.704+0000 7f31feffd700 1 -- 172.21.15.55:0/3622791815 learned_addr learned my addr 172.21.15.55:0/3622791815 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-11-15T01:49:37.707 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:37.705+0000 7f31feffd700 1 -- 172.21.15.55:0/3622791815 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31f000b200 con 0x7f32000703d0 2023-11-15T01:49:37.708 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:37.705+0000 7f31feffd700 1 --2- 172.21.15.55:0/3622791815 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f32000703d0 0x7f320010aa30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f31f000e040 tx=0x7f31f0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:49:37.708 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:37.705+0000 7f31e77fe700 1 -- 172.21.15.55:0/3622791815 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f31f000d9f0 con 0x7f32000703d0 2023-11-15T01:49:37.708 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:37.705+0000 7f31e77fe700 1 -- 172.21.15.55:0/3622791815 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f31f001a9f0 con 0x7f32000703d0 2023-11-15T01:49:37.708 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:37.705+0000 7f31e77fe700 1 -- 172.21.15.55:0/3622791815 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f31f001b8c0 con 0x7f32000703d0 2023-11-15T01:49:37.708 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:37.705+0000 7f3205613700 1 -- 172.21.15.55:0/3622791815 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f320010b120 con 0x7f32000703d0 2023-11-15T01:49:37.709 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:37.705+0000 7f3205613700 1 -- 172.21.15.55:0/3622791815 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f320010b530 con 0x7f32000703d0 2023-11-15T01:49:37.709 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:37.706+0000 7f31e77fe700 1 -- 172.21.15.55:0/3622791815 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f31f000f070 con 0x7f32000703d0 2023-11-15T01:49:37.710 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:37.706+0000 7f31e77fe700 1 --2- 172.21.15.55:0/3622791815 >> [v2:172.21.15.55:6800/2574607930,v1:172.21.15.55:6801/2574607930] conn(0x7f31ec02ff00 0x7f31ec0323c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:49:37.710 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:37.706+0000 7f31e77fe700 1 -- 172.21.15.55:0/3622791815 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f31f0042210 con 0x7f32000703d0 2023-11-15T01:49:37.710 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:37.706+0000 7f31fe7fc700 1 -- 172.21.15.55:0/3622791815 >> [v2:172.21.15.55:6800/2574607930,v1:172.21.15.55:6801/2574607930] conn(0x7f31ec02ff00 msgr2=0x7f31ec0323c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.55:6800/2574607930 2023-11-15T01:49:37.710 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:37.706+0000 7f31fe7fc700 1 --2- 172.21.15.55:0/3622791815 >> [v2:172.21.15.55:6800/2574607930,v1:172.21.15.55:6801/2574607930] conn(0x7f31ec02ff00 0x7f31ec0323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-11-15T01:49:37.710 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:37.707+0000 7f3205613700 1 -- 172.21.15.55:0/3622791815 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31e0004fa0 con 0x7f32000703d0 2023-11-15T01:49:37.713 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:37.712+0000 7f31e77fe700 1 -- 172.21.15.55:0/3622791815 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f31f0016070 con 0x7f32000703d0 2023-11-15T01:49:37.908 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:37.907+0000 7f31fe7fc700 1 -- 172.21.15.55:0/3622791815 >> [v2:172.21.15.55:6800/2574607930,v1:172.21.15.55:6801/2574607930] conn(0x7f31ec02ff00 msgr2=0x7f31ec0323c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.55:6800/2574607930 2023-11-15T01:49:37.908 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:37.907+0000 7f31fe7fc700 1 --2- 172.21.15.55:0/3622791815 >> [v2:172.21.15.55:6800/2574607930,v1:172.21.15.55:6801/2574607930] conn(0x7f31ec02ff00 0x7f31ec0323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-11-15T01:49:37.978 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:37.977+0000 7f3205613700 1 -- 172.21.15.55:0/3622791815 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f31e0005e80 con 0x7f32000703d0 2023-11-15T01:49:37.978 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:37.977+0000 7f31e77fe700 1 -- 172.21.15.55:0/3622791815 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v13) v1 ==== 56+0+104 (secure 0 0 0) 0x7f31f0025460 con 0x7f32000703d0 2023-11-15T01:49:37.978 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: { 2023-11-15T01:49:37.979 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "epoch": 13, 2023-11-15T01:49:37.979 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "available": true, 2023-11-15T01:49:37.980 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "active_name": "smithi055.szwngw", 2023-11-15T01:49:37.980 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "num_standby": 0 2023-11-15T01:49:37.980 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: } 2023-11-15T01:49:37.981 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:37.980+0000 7f3205613700 1 -- 172.21.15.55:0/3622791815 >> [v2:172.21.15.55:6800/2574607930,v1:172.21.15.55:6801/2574607930] conn(0x7f31ec02ff00 msgr2=0x7f31ec0323c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-15T01:49:37.981 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:37.980+0000 7f3205613700 1 --2- 172.21.15.55:0/3622791815 >> [v2:172.21.15.55:6800/2574607930,v1:172.21.15.55:6801/2574607930] conn(0x7f31ec02ff00 0x7f31ec0323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-15T01:49:37.981 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:37.980+0000 7f3205613700 1 -- 172.21.15.55:0/3622791815 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f32000703d0 msgr2=0x7f320010aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:49:37.982 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:37.980+0000 7f3205613700 1 --2- 172.21.15.55:0/3622791815 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f32000703d0 0x7f320010aa30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f31f000e040 tx=0x7f31f0006ee0).stop 2023-11-15T01:49:37.982 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:37.980+0000 7f3205613700 1 -- 172.21.15.55:0/3622791815 shutdown_connections 2023-11-15T01:49:37.982 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:37.980+0000 7f3205613700 1 --2- 172.21.15.55:0/3622791815 >> [v2:172.21.15.55:6800/2574607930,v1:172.21.15.55:6801/2574607930] conn(0x7f31ec02ff00 0x7f31ec0323c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-15T01:49:37.983 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:37.980+0000 7f3205613700 1 --2- 172.21.15.55:0/3622791815 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f32000703d0 0x7f320010aa30 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:49:37.983 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:37.980+0000 7f3205613700 1 -- 172.21.15.55:0/3622791815 >> 172.21.15.55:0/3622791815 conn(0x7f32000f63c0 msgr2=0x7f32000f7f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:49:37.984 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:37.980+0000 7f3205613700 1 -- 172.21.15.55:0/3622791815 shutdown_connections 2023-11-15T01:49:37.984 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:37.982+0000 7f3205613700 1 -- 172.21.15.55:0/3622791815 wait complete. 2023-11-15T01:49:38.013 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:37 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:49:36.935202+0000 mon.smithi055 (mon.0) 86 : audit [INF] from='client.? 172.21.15.55:0/4105356394' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2023-11-15T01:49:38.013 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:37 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:49:36.935260+0000 mon.smithi055 (mon.0) 87 : cluster [DBG] mgrmap e13: smithi055.szwngw(active, since 6s) 2023-11-15T01:49:38.525 INFO:teuthology.orchestra.run.smithi055.stderr:Waiting for the mgr to restart... 2023-11-15T01:49:38.525 INFO:teuthology.orchestra.run.smithi055.stderr:Waiting for mgr epoch 13... 2023-11-15T01:49:38.526 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi055 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/0cb4fc0c-8359-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpc1yi9jse:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpra3bp1py:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2023-11-15T01:49:38.891 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:38.889+0000 7f8c2221c700 1 Processor -- start 2023-11-15T01:49:38.891 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:38.889+0000 7f8c2221c700 1 -- start start 2023-11-15T01:49:38.891 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:38.889+0000 7f8c2221c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8c1c070410 0x7f8c1c0fe750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T01:49:38.891 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:38.889+0000 7f8c2221c700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c1c0fec40 con 0x7f8c1c070410 2023-11-15T01:49:38.892 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:38.890+0000 7f8c2121a700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8c1c070410 0x7f8c1c0fe750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:49:38.892 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:38.890+0000 7f8c2121a700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8c1c070410 0x7f8c1c0fe750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:47494/0 (socket says 172.21.15.55:47494) 2023-11-15T01:49:38.892 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:38.890+0000 7f8c2121a700 1 -- 172.21.15.55:0/405400693 learned_addr learned my addr 172.21.15.55:0/405400693 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-11-15T01:49:38.894 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:38.890+0000 7f8c2121a700 1 -- 172.21.15.55:0/405400693 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c1c0fed80 con 0x7f8c1c070410 2023-11-15T01:49:38.894 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:38.890+0000 7f8c2121a700 1 --2- 172.21.15.55:0/405400693 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8c1c070410 0x7f8c1c0fe750 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f8c1800b5f0 tx=0x7f8c18006ee0).ready entity=mon.0 client_cookie=c06c69239926e4a1 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:49:38.894 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:38.890+0000 7f8c0bfff700 1 -- 172.21.15.55:0/405400693 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8c1800d600 con 0x7f8c1c070410 2023-11-15T01:49:38.894 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:38.891+0000 7f8c0bfff700 1 -- 172.21.15.55:0/405400693 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f8c1800dbd0 con 0x7f8c1c070410 2023-11-15T01:49:38.894 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:38.891+0000 7f8c0bfff700 1 -- 172.21.15.55:0/405400693 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8c18011c00 con 0x7f8c1c070410 2023-11-15T01:49:38.895 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:38.891+0000 7f8c2221c700 1 -- 172.21.15.55:0/405400693 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8c1c070410 msgr2=0x7f8c1c0fe750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:49:38.895 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:38.891+0000 7f8c2221c700 1 --2- 172.21.15.55:0/405400693 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8c1c070410 0x7f8c1c0fe750 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f8c1800b5f0 tx=0x7f8c18006ee0).stop 2023-11-15T01:49:38.895 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:38.891+0000 7f8c2221c700 1 -- 172.21.15.55:0/405400693 shutdown_connections 2023-11-15T01:49:38.895 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:38.891+0000 7f8c2221c700 1 --2- 172.21.15.55:0/405400693 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8c1c070410 0x7f8c1c0fe750 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:49:38.895 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:38.891+0000 7f8c2221c700 1 -- 172.21.15.55:0/405400693 >> 172.21.15.55:0/405400693 conn(0x7f8c1c0f6210 msgr2=0x7f8c1c0f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:49:38.896 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:38.891+0000 7f8c2221c700 1 -- 172.21.15.55:0/405400693 shutdown_connections 2023-11-15T01:49:38.896 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:38.891+0000 7f8c2221c700 1 -- 172.21.15.55:0/405400693 wait complete. 2023-11-15T01:49:38.896 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:38.892+0000 7f8c2221c700 1 Processor -- start 2023-11-15T01:49:38.896 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:38.892+0000 7f8c2221c700 1 -- start start 2023-11-15T01:49:38.897 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:38.892+0000 7f8c2221c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8c1c070410 0x7f8c1c10a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:49:38.897 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:38.893+0000 7f8c2221c700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c1c10ad70 con 0x7f8c1c070410 2023-11-15T01:49:38.898 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:38.893+0000 7f8c2121a700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8c1c070410 0x7f8c1c10a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:49:38.898 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:38.893+0000 7f8c2121a700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8c1c070410 0x7f8c1c10a880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:47502/0 (socket says 172.21.15.55:47502) 2023-11-15T01:49:38.898 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:38.893+0000 7f8c2121a700 1 -- 172.21.15.55:0/1018611935 learned_addr learned my addr 172.21.15.55:0/1018611935 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-11-15T01:49:38.898 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:38.893+0000 7f8c2121a700 1 -- 172.21.15.55:0/1018611935 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c1800b200 con 0x7f8c1c070410 2023-11-15T01:49:38.899 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:38.893+0000 7f8c2121a700 1 --2- 172.21.15.55:0/1018611935 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8c1c070410 0x7f8c1c10a880 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f8c1800e040 tx=0x7f8c18006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:49:38.899 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:38.893+0000 7f8c09ffb700 1 -- 172.21.15.55:0/1018611935 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8c1800d9f0 con 0x7f8c1c070410 2023-11-15T01:49:38.899 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:38.893+0000 7f8c09ffb700 1 -- 172.21.15.55:0/1018611935 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f8c1801a9f0 con 0x7f8c1c070410 2023-11-15T01:49:38.899 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:38.893+0000 7f8c2221c700 1 -- 172.21.15.55:0/1018611935 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c1c10af70 con 0x7f8c1c070410 2023-11-15T01:49:38.899 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:38.893+0000 7f8c09ffb700 1 -- 172.21.15.55:0/1018611935 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8c1801b8c0 con 0x7f8c1c070410 2023-11-15T01:49:38.899 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:38.894+0000 7f8c2221c700 1 -- 172.21.15.55:0/1018611935 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c1c10b380 con 0x7f8c1c070410 2023-11-15T01:49:38.900 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:38.894+0000 7f8c09ffb700 1 -- 172.21.15.55:0/1018611935 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f8c1800f070 con 0x7f8c1c070410 2023-11-15T01:49:38.900 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:38.895+0000 7f8c09ffb700 1 --2- 172.21.15.55:0/1018611935 >> [v2:172.21.15.55:6800/2574607930,v1:172.21.15.55:6801/2574607930] conn(0x7f8c0402ff00 0x7f8c040323c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:49:38.900 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:38.895+0000 7f8c2221c700 1 -- 172.21.15.55:0/1018611935 --> [v2:172.21.15.55:6800/2574607930,v1:172.21.15.55:6801/2574607930] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8c1c059100 con 0x7f8c0402ff00 2023-11-15T01:49:38.900 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:38.895+0000 7f8c09ffb700 1 -- 172.21.15.55:0/1018611935 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f8c18042300 con 0x7f8c1c070410 2023-11-15T01:49:38.900 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:38.895+0000 7f8c20a19700 1 -- 172.21.15.55:0/1018611935 >> [v2:172.21.15.55:6800/2574607930,v1:172.21.15.55:6801/2574607930] conn(0x7f8c0402ff00 msgr2=0x7f8c040323c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.55:6800/2574607930 2023-11-15T01:49:38.901 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:38.895+0000 7f8c20a19700 1 --2- 172.21.15.55:0/1018611935 >> [v2:172.21.15.55:6800/2574607930,v1:172.21.15.55:6801/2574607930] conn(0x7f8c0402ff00 0x7f8c040323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-11-15T01:49:39.097 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:39.096+0000 7f8c20a19700 1 -- 172.21.15.55:0/1018611935 >> [v2:172.21.15.55:6800/2574607930,v1:172.21.15.55:6801/2574607930] conn(0x7f8c0402ff00 msgr2=0x7f8c040323c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.55:6800/2574607930 2023-11-15T01:49:39.097 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:39.096+0000 7f8c20a19700 1 --2- 172.21.15.55:0/1018611935 >> [v2:172.21.15.55:6800/2574607930,v1:172.21.15.55:6801/2574607930] conn(0x7f8c0402ff00 0x7f8c040323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-11-15T01:49:39.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:38 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:49:37.978045+0000 mon.smithi055 (mon.0) 88 : audit [DBG] from='client.? 172.21.15.55:0/3622791815' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-11-15T01:49:39.498 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:39.497+0000 7f8c20a19700 1 -- 172.21.15.55:0/1018611935 >> [v2:172.21.15.55:6800/2574607930,v1:172.21.15.55:6801/2574607930] conn(0x7f8c0402ff00 msgr2=0x7f8c040323c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.55:6800/2574607930 2023-11-15T01:49:39.498 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:39.497+0000 7f8c20a19700 1 --2- 172.21.15.55:0/1018611935 >> [v2:172.21.15.55:6800/2574607930,v1:172.21.15.55:6801/2574607930] conn(0x7f8c0402ff00 0x7f8c040323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-11-15T01:49:40.299 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:40.298+0000 7f8c20a19700 1 -- 172.21.15.55:0/1018611935 >> [v2:172.21.15.55:6800/2574607930,v1:172.21.15.55:6801/2574607930] conn(0x7f8c0402ff00 msgr2=0x7f8c040323c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.55:6800/2574607930 2023-11-15T01:49:40.299 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:40.298+0000 7f8c20a19700 1 --2- 172.21.15.55:0/1018611935 >> [v2:172.21.15.55:6800/2574607930,v1:172.21.15.55:6801/2574607930] conn(0x7f8c0402ff00 0x7f8c040323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-11-15T01:49:41.901 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:41.899+0000 7f8c20a19700 1 -- 172.21.15.55:0/1018611935 >> [v2:172.21.15.55:6800/2574607930,v1:172.21.15.55:6801/2574607930] conn(0x7f8c0402ff00 msgr2=0x7f8c040323c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.55:6800/2574607930 2023-11-15T01:49:41.902 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:41.899+0000 7f8c20a19700 1 --2- 172.21.15.55:0/1018611935 >> [v2:172.21.15.55:6800/2574607930,v1:172.21.15.55:6801/2574607930] conn(0x7f8c0402ff00 0x7f8c040323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2023-11-15T01:49:42.684 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:42.683+0000 7f8c09ffb700 1 -- 172.21.15.55:0/1018611935 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mgrmap(e 14) v1 ==== 35997+0+0 (secure 0 0 0) 0x7f8c180257e0 con 0x7f8c1c070410 2023-11-15T01:49:42.684 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:42.683+0000 7f8c09ffb700 1 -- 172.21.15.55:0/1018611935 >> [v2:172.21.15.55:6800/2574607930,v1:172.21.15.55:6801/2574607930] conn(0x7f8c0402ff00 msgr2=0x7f8c040323c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-15T01:49:42.685 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:42.683+0000 7f8c09ffb700 1 --2- 172.21.15.55:0/1018611935 >> [v2:172.21.15.55:6800/2574607930,v1:172.21.15.55:6801/2574607930] conn(0x7f8c0402ff00 0x7f8c040323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-15T01:49:42.937 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:42 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:49:42.631757+0000 mon.smithi055 (mon.0) 89 : cluster [INF] Active manager daemon smithi055.szwngw restarted 2023-11-15T01:49:42.937 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:42 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:49:42.632419+0000 mon.smithi055 (mon.0) 90 : cluster [INF] Activating manager daemon smithi055.szwngw 2023-11-15T01:49:42.937 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:42 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:49:42.638361+0000 mon.smithi055 (mon.0) 91 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2023-11-15T01:49:43.697 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:43.695+0000 7f8c09ffb700 1 -- 172.21.15.55:0/1018611935 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mgrmap(e 15) v1 ==== 36067+0+0 (secure 0 0 0) 0x7f8c180105f0 con 0x7f8c1c070410 2023-11-15T01:49:43.698 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:43.695+0000 7f8c09ffb700 1 --2- 172.21.15.55:0/1018611935 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f8c040339e0 0x7f8c04035dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:49:43.698 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:43.695+0000 7f8c09ffb700 1 -- 172.21.15.55:0/1018611935 --> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8c1c059100 con 0x7f8c040339e0 2023-11-15T01:49:43.698 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:43.695+0000 7f8c20a19700 1 --2- 172.21.15.55:0/1018611935 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f8c040339e0 0x7f8c04035dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:49:43.698 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:43.696+0000 7f8c20a19700 1 --2- 172.21.15.55:0/1018611935 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f8c040339e0 0x7f8c04035dd0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f8c0c006fd0 tx=0x7f8c0c006d70).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:49:43.699 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:43.697+0000 7f8c09ffb700 1 -- 172.21.15.55:0/1018611935 <== mgr.14164 v2:172.21.15.55:6800/157802929 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7f8c1c059100 con 0x7f8c040339e0 2023-11-15T01:49:43.704 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:43.704+0000 7f8c2221c700 1 -- 172.21.15.55:0/1018611935 --> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f8c1c045ad0 con 0x7f8c040339e0 2023-11-15T01:49:43.705 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:43.704+0000 7f8c09ffb700 1 -- 172.21.15.55:0/1018611935 <== mgr.14164 v2:172.21.15.55:6800/157802929 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f8c1c045ad0 con 0x7f8c040339e0 2023-11-15T01:49:43.705 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: { 2023-11-15T01:49:43.706 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "mgrmap_epoch": 15, 2023-11-15T01:49:43.706 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: "initialized": true 2023-11-15T01:49:43.706 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: } 2023-11-15T01:49:43.707 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:43.705+0000 7f8c2221c700 1 -- 172.21.15.55:0/1018611935 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f8c040339e0 msgr2=0x7f8c04035dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:49:43.708 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:43.705+0000 7f8c2221c700 1 --2- 172.21.15.55:0/1018611935 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f8c040339e0 0x7f8c04035dd0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f8c0c006fd0 tx=0x7f8c0c006d70).stop 2023-11-15T01:49:43.708 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:43.705+0000 7f8c2221c700 1 -- 172.21.15.55:0/1018611935 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8c1c070410 msgr2=0x7f8c1c10a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:49:43.708 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:43.705+0000 7f8c2221c700 1 --2- 172.21.15.55:0/1018611935 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8c1c070410 0x7f8c1c10a880 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f8c1800e040 tx=0x7f8c18006ee0).stop 2023-11-15T01:49:43.708 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:43.705+0000 7f8c2221c700 1 -- 172.21.15.55:0/1018611935 shutdown_connections 2023-11-15T01:49:43.708 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:43.705+0000 7f8c2221c700 1 --2- 172.21.15.55:0/1018611935 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f8c040339e0 0x7f8c04035dd0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:49:43.709 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:43.705+0000 7f8c2221c700 1 --2- 172.21.15.55:0/1018611935 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8c1c070410 0x7f8c1c10a880 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:49:43.709 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:43.705+0000 7f8c2221c700 1 --2- 172.21.15.55:0/1018611935 >> [v2:172.21.15.55:6800/2574607930,v1:172.21.15.55:6801/2574607930] conn(0x7f8c0402ff00 0x7f8c040323c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-15T01:49:43.709 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:43.705+0000 7f8c2221c700 1 -- 172.21.15.55:0/1018611935 >> 172.21.15.55:0/1018611935 conn(0x7f8c1c0f6210 msgr2=0x7f8c1c0f7cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:49:43.709 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:43.705+0000 7f8c2221c700 1 -- 172.21.15.55:0/1018611935 shutdown_connections 2023-11-15T01:49:43.709 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:43.705+0000 7f8c2221c700 1 -- 172.21.15.55:0/1018611935 wait complete. 2023-11-15T01:49:44.008 INFO:teuthology.orchestra.run.smithi055.stderr:mgr epoch 13 is available 2023-11-15T01:49:44.009 INFO:teuthology.orchestra.run.smithi055.stderr:Generating a dashboard self-signed certificate... 2023-11-15T01:49:44.009 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi055 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/0cb4fc0c-8359-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpc1yi9jse:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpra3bp1py:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 dashboard create-self-signed-cert 2023-11-15T01:49:44.012 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:43 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:49:42.685292+0000 mon.smithi055 (mon.0) 92 : cluster [DBG] mgrmap e14: smithi055.szwngw(active, starting, since 0.0530092s) 2023-11-15T01:49:44.012 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:43 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:49:42.687974+0000 mon.smithi055 (mon.0) 93 : audit [DBG] from='mgr.14164 172.21.15.55:0/59463371' entity='mgr.smithi055.szwngw' cmd=[{"prefix": "mon metadata", "id": "smithi055"}]: dispatch 2023-11-15T01:49:44.012 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:43 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:49:42.688692+0000 mon.smithi055 (mon.0) 94 : audit [DBG] from='mgr.14164 172.21.15.55:0/59463371' entity='mgr.smithi055.szwngw' cmd=[{"prefix": "mgr metadata", "who": "smithi055.szwngw", "id": "smithi055.szwngw"}]: dispatch 2023-11-15T01:49:44.013 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:43 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:49:42.689669+0000 mon.smithi055 (mon.0) 95 : audit [DBG] from='mgr.14164 172.21.15.55:0/59463371' entity='mgr.smithi055.szwngw' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-11-15T01:49:44.013 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:43 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:49:42.689951+0000 mon.smithi055 (mon.0) 96 : audit [DBG] from='mgr.14164 172.21.15.55:0/59463371' entity='mgr.smithi055.szwngw' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-11-15T01:49:44.013 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:43 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:49:42.690214+0000 mon.smithi055 (mon.0) 97 : audit [DBG] from='mgr.14164 172.21.15.55:0/59463371' entity='mgr.smithi055.szwngw' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-11-15T01:49:44.013 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:43 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:49:42.704359+0000 mon.smithi055 (mon.0) 98 : cluster [INF] Manager daemon smithi055.szwngw is now available 2023-11-15T01:49:44.013 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:43 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:49:42.817694+0000 mon.smithi055 (mon.0) 99 : audit [DBG] from='mgr.14164 172.21.15.55:0/59463371' entity='mgr.smithi055.szwngw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-15T01:49:44.014 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:43 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:49:42.819331+0000 mon.smithi055 (mon.0) 100 : audit [DBG] from='mgr.14164 172.21.15.55:0/59463371' entity='mgr.smithi055.szwngw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-15T01:49:44.014 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:43 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:49:42.844814+0000 mon.smithi055 (mon.0) 101 : audit [INF] from='mgr.14164 172.21.15.55:0/59463371' entity='mgr.smithi055.szwngw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi055.szwngw/mirror_snapshot_schedule"}]: dispatch 2023-11-15T01:49:44.014 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:43 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:49:42.849193+0000 mon.smithi055 (mon.0) 102 : audit [INF] from='mgr.14164 172.21.15.55:0/59463371' entity='mgr.smithi055.szwngw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi055.szwngw/trash_purge_schedule"}]: dispatch 2023-11-15T01:49:44.373 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:44.372+0000 7f1bec38f700 1 Processor -- start 2023-11-15T01:49:44.373 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:44.372+0000 7f1bec38f700 1 -- start start 2023-11-15T01:49:44.374 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:44.372+0000 7f1bec38f700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1be4070390 0x7f1be4100d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T01:49:44.374 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:44.372+0000 7f1bec38f700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1be40fc770 con 0x7f1be4070390 2023-11-15T01:49:44.374 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:44.372+0000 7f1bea12b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1be4070390 0x7f1be4100d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:49:44.378 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:44.372+0000 7f1bea12b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1be4070390 0x7f1be4100d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:40144/0 (socket says 172.21.15.55:40144) 2023-11-15T01:49:44.378 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:44.372+0000 7f1bea12b700 1 -- 172.21.15.55:0/4274693536 learned_addr learned my addr 172.21.15.55:0/4274693536 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-11-15T01:49:44.381 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:44.373+0000 7f1bea12b700 1 -- 172.21.15.55:0/4274693536 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1be4101270 con 0x7f1be4070390 2023-11-15T01:49:44.382 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:44.373+0000 7f1bea12b700 1 --2- 172.21.15.55:0/4274693536 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1be4070390 0x7f1be4100d80 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f1be000b5f0 tx=0x7f1be0006ee0).ready entity=mon.0 client_cookie=726d9f0f2c19ba7 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:49:44.382 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:44.373+0000 7f1be9129700 1 -- 172.21.15.55:0/4274693536 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1be000d600 con 0x7f1be4070390 2023-11-15T01:49:44.382 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:44.373+0000 7f1be9129700 1 -- 172.21.15.55:0/4274693536 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f1be000dbd0 con 0x7f1be4070390 2023-11-15T01:49:44.382 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:44.373+0000 7f1be9129700 1 -- 172.21.15.55:0/4274693536 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1be0011c00 con 0x7f1be4070390 2023-11-15T01:49:44.382 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:44.374+0000 7f1bec38f700 1 -- 172.21.15.55:0/4274693536 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1be4070390 msgr2=0x7f1be4100d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:49:44.383 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:44.374+0000 7f1bec38f700 1 --2- 172.21.15.55:0/4274693536 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1be4070390 0x7f1be4100d80 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f1be000b5f0 tx=0x7f1be0006ee0).stop 2023-11-15T01:49:44.383 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:44.374+0000 7f1bec38f700 1 -- 172.21.15.55:0/4274693536 shutdown_connections 2023-11-15T01:49:44.383 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:44.374+0000 7f1bec38f700 1 --2- 172.21.15.55:0/4274693536 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1be4070390 0x7f1be4100d80 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:49:44.383 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:44.374+0000 7f1bec38f700 1 -- 172.21.15.55:0/4274693536 >> 172.21.15.55:0/4274693536 conn(0x7f1be40f63c0 msgr2=0x7f1be40f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:49:44.383 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:44.374+0000 7f1bec38f700 1 -- 172.21.15.55:0/4274693536 shutdown_connections 2023-11-15T01:49:44.383 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:44.374+0000 7f1bec38f700 1 -- 172.21.15.55:0/4274693536 wait complete. 2023-11-15T01:49:44.384 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:44.375+0000 7f1bec38f700 1 Processor -- start 2023-11-15T01:49:44.384 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:44.375+0000 7f1bec38f700 1 -- start start 2023-11-15T01:49:44.384 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:44.375+0000 7f1bec38f700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1be4070390 0x7f1be410cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:49:44.384 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:44.375+0000 7f1bec38f700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1be410d200 con 0x7f1be4070390 2023-11-15T01:49:44.384 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:44.375+0000 7f1bea12b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1be4070390 0x7f1be410cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:49:44.385 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:44.375+0000 7f1bea12b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1be4070390 0x7f1be410cd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:40154/0 (socket says 172.21.15.55:40154) 2023-11-15T01:49:44.385 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:44.375+0000 7f1bea12b700 1 -- 172.21.15.55:0/521081161 learned_addr learned my addr 172.21.15.55:0/521081161 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-11-15T01:49:44.386 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:44.375+0000 7f1bea12b700 1 -- 172.21.15.55:0/521081161 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1be000b200 con 0x7f1be4070390 2023-11-15T01:49:44.386 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:44.376+0000 7f1bea12b700 1 --2- 172.21.15.55:0/521081161 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1be4070390 0x7f1be410cd10 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f1be000e040 tx=0x7f1be0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:49:44.386 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:44.376+0000 7f1bd2ffd700 1 -- 172.21.15.55:0/521081161 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1be000d9f0 con 0x7f1be4070390 2023-11-15T01:49:44.386 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:44.376+0000 7f1bd2ffd700 1 -- 172.21.15.55:0/521081161 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f1be001a9f0 con 0x7f1be4070390 2023-11-15T01:49:44.387 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:44.376+0000 7f1bec38f700 1 -- 172.21.15.55:0/521081161 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1be410d400 con 0x7f1be4070390 2023-11-15T01:49:44.387 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:44.376+0000 7f1bd2ffd700 1 -- 172.21.15.55:0/521081161 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1be001b8c0 con 0x7f1be4070390 2023-11-15T01:49:44.387 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:44.376+0000 7f1bec38f700 1 -- 172.21.15.55:0/521081161 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1be410d810 con 0x7f1be4070390 2023-11-15T01:49:44.387 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:44.377+0000 7f1bd2ffd700 1 -- 172.21.15.55:0/521081161 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 15) v1 ==== 36067+0+0 (secure 0 0 0) 0x7f1be000f070 con 0x7f1be4070390 2023-11-15T01:49:44.387 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:44.377+0000 7f1bd2ffd700 1 --2- 172.21.15.55:0/521081161 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f1bcc02fdc0 0x7f1bcc032280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:49:44.388 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:44.377+0000 7f1bd2ffd700 1 -- 172.21.15.55:0/521081161 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1be00413e0 con 0x7f1be4070390 2023-11-15T01:49:44.388 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:44.377+0000 7f1be992a700 1 --2- 172.21.15.55:0/521081161 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f1bcc02fdc0 0x7f1bcc032280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:49:44.388 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:44.377+0000 7f1bec38f700 1 -- 172.21.15.55:0/521081161 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1be4101d70 con 0x7f1be4070390 2023-11-15T01:49:44.388 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:44.377+0000 7f1be992a700 1 --2- 172.21.15.55:0/521081161 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f1bcc02fdc0 0x7f1bcc032280 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f1bd40096a0 tx=0x7f1bd4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:49:44.389 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:44.382+0000 7f1bd2ffd700 1 -- 172.21.15.55:0/521081161 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1be0016070 con 0x7f1be4070390 2023-11-15T01:49:44.553 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:44.552+0000 7f1bec38f700 1 -- 172.21.15.55:0/521081161 --> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f1be4101f30 con 0x7f1bcc02fdc0 2023-11-15T01:49:44.621 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:44.620+0000 7f1bd2ffd700 1 -- 172.21.15.55:0/521081161 <== mgr.14164 v2:172.21.15.55:6800/157802929 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f1be4101f30 con 0x7f1bcc02fdc0 2023-11-15T01:49:44.622 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: Self-signed certificate created 2023-11-15T01:49:44.624 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:44.622+0000 7f1bec38f700 1 -- 172.21.15.55:0/521081161 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f1bcc02fdc0 msgr2=0x7f1bcc032280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:49:44.624 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:44.623+0000 7f1bec38f700 1 --2- 172.21.15.55:0/521081161 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f1bcc02fdc0 0x7f1bcc032280 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f1bd40096a0 tx=0x7f1bd4006b40).stop 2023-11-15T01:49:44.624 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:44.623+0000 7f1bec38f700 1 -- 172.21.15.55:0/521081161 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1be4070390 msgr2=0x7f1be410cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:49:44.625 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:44.623+0000 7f1bec38f700 1 --2- 172.21.15.55:0/521081161 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1be4070390 0x7f1be410cd10 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f1be000e040 tx=0x7f1be0006ee0).stop 2023-11-15T01:49:44.625 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:44.623+0000 7f1bec38f700 1 -- 172.21.15.55:0/521081161 shutdown_connections 2023-11-15T01:49:44.625 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:44.623+0000 7f1bec38f700 1 --2- 172.21.15.55:0/521081161 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f1bcc02fdc0 0x7f1bcc032280 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:49:44.625 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:44.623+0000 7f1bec38f700 1 --2- 172.21.15.55:0/521081161 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1be4070390 0x7f1be410cd10 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:49:44.625 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:44.623+0000 7f1bec38f700 1 -- 172.21.15.55:0/521081161 >> 172.21.15.55:0/521081161 conn(0x7f1be40f63c0 msgr2=0x7f1be40f7f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:49:44.626 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:44.623+0000 7f1bec38f700 1 -- 172.21.15.55:0/521081161 shutdown_connections 2023-11-15T01:49:44.626 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:44.623+0000 7f1bec38f700 1 -- 172.21.15.55:0/521081161 wait complete. 2023-11-15T01:49:44.911 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:44 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:49:43.697398+0000 mon.smithi055 (mon.0) 103 : cluster [DBG] mgrmap e15: smithi055.szwngw(active, since 1.06511s) 2023-11-15T01:49:44.911 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:44 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:49:43.697497+0000 mgr.smithi055.szwngw (mgr.14164) 1 : audit [DBG] from='client.14168 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-11-15T01:49:44.912 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:44 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:49:43.705155+0000 mgr.smithi055.szwngw (mgr.14164) 2 : audit [DBG] from='client.14168 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-11-15T01:49:44.912 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:44 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:49:44.616651+0000 mon.smithi055 (mon.0) 104 : audit [INF] from='mgr.14164 172.21.15.55:0/59463371' entity='mgr.smithi055.szwngw' 2023-11-15T01:49:44.912 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:44 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:49:44.621283+0000 mon.smithi055 (mon.0) 105 : audit [INF] from='mgr.14164 172.21.15.55:0/59463371' entity='mgr.smithi055.szwngw' 2023-11-15T01:49:44.974 INFO:teuthology.orchestra.run.smithi055.stderr:Creating initial admin user... 2023-11-15T01:49:44.975 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi055 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/0cb4fc0c-8359-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpc1yi9jse:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpra3bp1py:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp5r3mrwd_:/tmp/dashboard.pw:z docker.io/ceph/ceph:v16.2.4 dashboard ac-user-create admin -i /tmp/dashboard.pw administrator --force-password --pwd-update-required 2023-11-15T01:49:45.358 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:45.356+0000 7efc2ab24700 1 Processor -- start 2023-11-15T01:49:45.359 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:45.356+0000 7efc2ab24700 1 -- start start 2023-11-15T01:49:45.359 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:45.356+0000 7efc2ab24700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efc24070430 0x7efc240fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T01:49:45.359 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:45.356+0000 7efc2ab24700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc240fedd0 con 0x7efc24070430 2023-11-15T01:49:45.359 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:45.356+0000 7efc288c0700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efc24070430 0x7efc240fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:49:45.359 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:45.356+0000 7efc288c0700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efc24070430 0x7efc240fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:40168/0 (socket says 172.21.15.55:40168) 2023-11-15T01:49:45.360 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:45.356+0000 7efc288c0700 1 -- 172.21.15.55:0/3104730918 learned_addr learned my addr 172.21.15.55:0/3104730918 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-11-15T01:49:45.360 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:45.357+0000 7efc288c0700 1 -- 172.21.15.55:0/3104730918 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc240fef10 con 0x7efc24070430 2023-11-15T01:49:45.360 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:45.357+0000 7efc288c0700 1 --2- 172.21.15.55:0/3104730918 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efc24070430 0x7efc240fe8e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7efc1800b5f0 tx=0x7efc18006ee0).ready entity=mon.0 client_cookie=15222ced18de9ad6 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:49:45.360 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:45.357+0000 7efc237fe700 1 -- 172.21.15.55:0/3104730918 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efc1800d600 con 0x7efc24070430 2023-11-15T01:49:45.360 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:45.357+0000 7efc237fe700 1 -- 172.21.15.55:0/3104730918 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7efc1800dbd0 con 0x7efc24070430 2023-11-15T01:49:45.361 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:45.357+0000 7efc237fe700 1 -- 172.21.15.55:0/3104730918 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efc18011c00 con 0x7efc24070430 2023-11-15T01:49:45.361 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:45.357+0000 7efc2ab24700 1 -- 172.21.15.55:0/3104730918 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efc24070430 msgr2=0x7efc240fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:49:45.361 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:45.358+0000 7efc2ab24700 1 --2- 172.21.15.55:0/3104730918 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efc24070430 0x7efc240fe8e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7efc1800b5f0 tx=0x7efc18006ee0).stop 2023-11-15T01:49:45.364 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:45.358+0000 7efc2ab24700 1 -- 172.21.15.55:0/3104730918 shutdown_connections 2023-11-15T01:49:45.364 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:45.358+0000 7efc2ab24700 1 --2- 172.21.15.55:0/3104730918 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efc24070430 0x7efc240fe8e0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:49:45.364 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:45.358+0000 7efc2ab24700 1 -- 172.21.15.55:0/3104730918 >> 172.21.15.55:0/3104730918 conn(0x7efc240f63c0 msgr2=0x7efc240f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:49:45.364 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:45.358+0000 7efc2ab24700 1 -- 172.21.15.55:0/3104730918 shutdown_connections 2023-11-15T01:49:45.364 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:45.358+0000 7efc2ab24700 1 -- 172.21.15.55:0/3104730918 wait complete. 2023-11-15T01:49:45.365 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:45.359+0000 7efc2ab24700 1 Processor -- start 2023-11-15T01:49:45.365 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:45.359+0000 7efc2ab24700 1 -- start start 2023-11-15T01:49:45.365 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:45.359+0000 7efc2ab24700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efc24070430 0x7efc2410aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:49:45.365 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:45.359+0000 7efc2ab24700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc2410af20 con 0x7efc24070430 2023-11-15T01:49:45.365 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:45.359+0000 7efc288c0700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efc24070430 0x7efc2410aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:49:45.366 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:45.359+0000 7efc288c0700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efc24070430 0x7efc2410aa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:40172/0 (socket says 172.21.15.55:40172) 2023-11-15T01:49:45.366 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:45.359+0000 7efc288c0700 1 -- 172.21.15.55:0/3205562545 learned_addr learned my addr 172.21.15.55:0/3205562545 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-11-15T01:49:45.366 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:45.360+0000 7efc288c0700 1 -- 172.21.15.55:0/3205562545 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc1800b200 con 0x7efc24070430 2023-11-15T01:49:45.366 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:45.360+0000 7efc288c0700 1 --2- 172.21.15.55:0/3205562545 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efc24070430 0x7efc2410aa30 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7efc1800b190 tx=0x7efc18006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:49:45.366 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:45.360+0000 7efc217fa700 1 -- 172.21.15.55:0/3205562545 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efc1801b410 con 0x7efc24070430 2023-11-15T01:49:45.367 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:45.360+0000 7efc217fa700 1 -- 172.21.15.55:0/3205562545 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7efc1801b9e0 con 0x7efc24070430 2023-11-15T01:49:45.367 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:45.360+0000 7efc2ab24700 1 -- 172.21.15.55:0/3205562545 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc2410b120 con 0x7efc24070430 2023-11-15T01:49:45.367 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:45.360+0000 7efc217fa700 1 -- 172.21.15.55:0/3205562545 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efc1801a880 con 0x7efc24070430 2023-11-15T01:49:45.367 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:45.360+0000 7efc2ab24700 1 -- 172.21.15.55:0/3205562545 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc2410b530 con 0x7efc24070430 2023-11-15T01:49:45.367 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:45.361+0000 7efc217fa700 1 -- 172.21.15.55:0/3205562545 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 15) v1 ==== 36067+0+0 (secure 0 0 0) 0x7efc1800f070 con 0x7efc24070430 2023-11-15T01:49:45.368 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:45.361+0000 7efc2ab24700 1 -- 172.21.15.55:0/3205562545 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc24045ad0 con 0x7efc24070430 2023-11-15T01:49:45.368 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:45.361+0000 7efc217fa700 1 --2- 172.21.15.55:0/3205562545 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7efc0802fda0 0x7efc08032260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:49:45.368 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:45.361+0000 7efc217fa700 1 -- 172.21.15.55:0/3205562545 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efc180412f0 con 0x7efc24070430 2023-11-15T01:49:45.368 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:45.361+0000 7efc23fff700 1 --2- 172.21.15.55:0/3205562545 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7efc0802fda0 0x7efc08032260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:49:45.368 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:45.362+0000 7efc23fff700 1 --2- 172.21.15.55:0/3205562545 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7efc0802fda0 0x7efc08032260 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7efc100096a0 tx=0x7efc10006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:49:45.369 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:45.366+0000 7efc217fa700 1 -- 172.21.15.55:0/3205562545 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efc18019950 con 0x7efc24070430 2023-11-15T01:49:45.551 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:45.550+0000 7efc2ab24700 1 -- 172.21.15.55:0/3205562545 --> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7efc24058f20 con 0x7efc0802fda0 2023-11-15T01:49:45.624 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:45.623+0000 7efc217fa700 1 -- 172.21.15.55:0/3205562545 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7efc1801bb50 con 0x7efc24070430 2023-11-15T01:49:45.806 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:45.805+0000 7efc217fa700 1 -- 172.21.15.55:0/3205562545 <== mgr.14164 v2:172.21.15.55:6800/157802929 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7efc24058f20 con 0x7efc0802fda0 2023-11-15T01:49:45.806 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$hNNPdv3S.lkWEScw3IglWu9egr8zQWCl6BrrFGYtJp1rFyWKtc4Fa", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1700012985, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2023-11-15T01:49:45.811 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:45.807+0000 7efc2ab24700 1 -- 172.21.15.55:0/3205562545 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7efc0802fda0 msgr2=0x7efc08032260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:49:45.812 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:45.807+0000 7efc2ab24700 1 --2- 172.21.15.55:0/3205562545 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7efc0802fda0 0x7efc08032260 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7efc100096a0 tx=0x7efc10006b40).stop 2023-11-15T01:49:45.812 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:45.807+0000 7efc2ab24700 1 -- 172.21.15.55:0/3205562545 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efc24070430 msgr2=0x7efc2410aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:49:45.812 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:45.807+0000 7efc2ab24700 1 --2- 172.21.15.55:0/3205562545 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efc24070430 0x7efc2410aa30 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7efc1800b190 tx=0x7efc18006ee0).stop 2023-11-15T01:49:45.812 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:45.807+0000 7efc2ab24700 1 -- 172.21.15.55:0/3205562545 shutdown_connections 2023-11-15T01:49:45.812 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:45.807+0000 7efc2ab24700 1 --2- 172.21.15.55:0/3205562545 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7efc0802fda0 0x7efc08032260 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:49:45.813 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:45.807+0000 7efc2ab24700 1 --2- 172.21.15.55:0/3205562545 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efc24070430 0x7efc2410aa30 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:49:45.813 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:45.807+0000 7efc2ab24700 1 -- 172.21.15.55:0/3205562545 >> 172.21.15.55:0/3205562545 conn(0x7efc240f63c0 msgr2=0x7efc240f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:49:45.813 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:45.807+0000 7efc2ab24700 1 -- 172.21.15.55:0/3205562545 shutdown_connections 2023-11-15T01:49:45.813 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:45.807+0000 7efc2ab24700 1 -- 172.21.15.55:0/3205562545 wait complete. 2023-11-15T01:49:45.981 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:45 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:49:44.553464+0000 mgr.smithi055.szwngw (mgr.14164) 3 : audit [DBG] from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T01:49:45.982 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:45 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:49:45.626078+0000 mon.smithi055 (mon.0) 106 : cluster [DBG] mgrmap e16: smithi055.szwngw(active, since 2s) 2023-11-15T01:49:46.182 INFO:teuthology.orchestra.run.smithi055.stderr:Fetching dashboard port number... 2023-11-15T01:49:46.183 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi055 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/0cb4fc0c-8359-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpc1yi9jse:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpra3bp1py:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config get mgr mgr/dashboard/ssl_server_port 2023-11-15T01:49:46.572 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:46.571+0000 7fce488e9700 1 Processor -- start 2023-11-15T01:49:46.572 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:46.571+0000 7fce488e9700 1 -- start start 2023-11-15T01:49:46.572 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:46.571+0000 7fce488e9700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fce44070430 0x7fce440fe730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T01:49:46.573 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:46.571+0000 7fce488e9700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce440fec20 con 0x7fce44070430 2023-11-15T01:49:46.575 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:46.571+0000 7fce437fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fce44070430 0x7fce440fe730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:49:46.575 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:46.571+0000 7fce437fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fce44070430 0x7fce440fe730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:40176/0 (socket says 172.21.15.55:40176) 2023-11-15T01:49:46.575 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:46.571+0000 7fce437fe700 1 -- 172.21.15.55:0/2065535354 learned_addr learned my addr 172.21.15.55:0/2065535354 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-11-15T01:49:46.575 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:46.572+0000 7fce437fe700 1 -- 172.21.15.55:0/2065535354 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce440fed60 con 0x7fce44070430 2023-11-15T01:49:46.576 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:46.572+0000 7fce437fe700 1 --2- 172.21.15.55:0/2065535354 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fce44070430 0x7fce440fe730 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7fce3400b5f0 tx=0x7fce34006ee0).ready entity=mon.0 client_cookie=587dd45b780b9651 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:49:46.576 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:46.572+0000 7fce427fc700 1 -- 172.21.15.55:0/2065535354 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce3400d600 con 0x7fce44070430 2023-11-15T01:49:46.576 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:46.572+0000 7fce427fc700 1 -- 172.21.15.55:0/2065535354 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fce3400dbd0 con 0x7fce44070430 2023-11-15T01:49:46.576 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:46.572+0000 7fce427fc700 1 -- 172.21.15.55:0/2065535354 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce34011c00 con 0x7fce44070430 2023-11-15T01:49:46.576 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:46.573+0000 7fce488e9700 1 -- 172.21.15.55:0/2065535354 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fce44070430 msgr2=0x7fce440fe730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:49:46.577 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:46.573+0000 7fce488e9700 1 --2- 172.21.15.55:0/2065535354 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fce44070430 0x7fce440fe730 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7fce3400b5f0 tx=0x7fce34006ee0).stop 2023-11-15T01:49:46.577 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:46.573+0000 7fce488e9700 1 -- 172.21.15.55:0/2065535354 shutdown_connections 2023-11-15T01:49:46.577 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:46.573+0000 7fce488e9700 1 --2- 172.21.15.55:0/2065535354 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fce44070430 0x7fce440fe730 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:49:46.577 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:46.573+0000 7fce488e9700 1 -- 172.21.15.55:0/2065535354 >> 172.21.15.55:0/2065535354 conn(0x7fce440f6210 msgr2=0x7fce440f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:49:46.577 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:46.573+0000 7fce488e9700 1 -- 172.21.15.55:0/2065535354 shutdown_connections 2023-11-15T01:49:46.578 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:46.573+0000 7fce488e9700 1 -- 172.21.15.55:0/2065535354 wait complete. 2023-11-15T01:49:46.578 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:46.573+0000 7fce488e9700 1 Processor -- start 2023-11-15T01:49:46.578 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:46.574+0000 7fce488e9700 1 -- start start 2023-11-15T01:49:46.578 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:46.574+0000 7fce488e9700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fce44070430 0x7fce4410a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:49:46.578 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:46.574+0000 7fce488e9700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce4410ad20 con 0x7fce44070430 2023-11-15T01:49:46.579 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:46.574+0000 7fce437fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fce44070430 0x7fce4410a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:49:46.580 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:46.574+0000 7fce437fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fce44070430 0x7fce4410a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:40188/0 (socket says 172.21.15.55:40188) 2023-11-15T01:49:46.581 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:46.574+0000 7fce437fe700 1 -- 172.21.15.55:0/2380160917 learned_addr learned my addr 172.21.15.55:0/2380160917 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-11-15T01:49:46.581 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:46.574+0000 7fce437fe700 1 -- 172.21.15.55:0/2380160917 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce3400b200 con 0x7fce44070430 2023-11-15T01:49:46.581 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:46.574+0000 7fce437fe700 1 --2- 172.21.15.55:0/2380160917 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fce44070430 0x7fce4410a830 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fce3400b190 tx=0x7fce34006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:49:46.581 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:46.574+0000 7fce2bfff700 1 -- 172.21.15.55:0/2380160917 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce3401b410 con 0x7fce44070430 2023-11-15T01:49:46.582 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:46.574+0000 7fce2bfff700 1 -- 172.21.15.55:0/2380160917 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fce3401b9e0 con 0x7fce44070430 2023-11-15T01:49:46.582 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:46.575+0000 7fce488e9700 1 -- 172.21.15.55:0/2380160917 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce4410af20 con 0x7fce44070430 2023-11-15T01:49:46.582 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:46.575+0000 7fce2bfff700 1 -- 172.21.15.55:0/2380160917 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce3401a880 con 0x7fce44070430 2023-11-15T01:49:46.582 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:46.575+0000 7fce488e9700 1 -- 172.21.15.55:0/2380160917 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce4410b330 con 0x7fce44070430 2023-11-15T01:49:46.582 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:46.575+0000 7fce2bfff700 1 -- 172.21.15.55:0/2380160917 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7fce3400f070 con 0x7fce44070430 2023-11-15T01:49:46.583 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:46.576+0000 7fce488e9700 1 -- 172.21.15.55:0/2380160917 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce44045ad0 con 0x7fce44070430 2023-11-15T01:49:46.583 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:46.576+0000 7fce2bfff700 1 --2- 172.21.15.55:0/2380160917 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fce3002ff00 0x7fce300323c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:49:46.583 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:46.576+0000 7fce2bfff700 1 -- 172.21.15.55:0/2380160917 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fce3401bb50 con 0x7fce44070430 2023-11-15T01:49:46.583 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:46.576+0000 7fce42ffd700 1 --2- 172.21.15.55:0/2380160917 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fce3002ff00 0x7fce300323c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:49:46.583 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:46.576+0000 7fce42ffd700 1 --2- 172.21.15.55:0/2380160917 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fce3002ff00 0x7fce300323c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fce380096a0 tx=0x7fce38006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:49:46.584 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:46.580+0000 7fce2bfff700 1 -- 172.21.15.55:0/2380160917 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fce34025460 con 0x7fce44070430 2023-11-15T01:49:46.747 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:46.746+0000 7fce488e9700 1 -- 172.21.15.55:0/2380160917 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7fce440ff9e0 con 0x7fce44070430 2023-11-15T01:49:46.747 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:46.746+0000 7fce2bfff700 1 -- 172.21.15.55:0/2380160917 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v9) v1 ==== 112+0+5 (secure 0 0 0) 0x7fce34016020 con 0x7fce44070430 2023-11-15T01:49:46.747 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 8443 2023-11-15T01:49:46.749 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:46.748+0000 7fce488e9700 1 -- 172.21.15.55:0/2380160917 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fce3002ff00 msgr2=0x7fce300323c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:49:46.750 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:46.748+0000 7fce488e9700 1 --2- 172.21.15.55:0/2380160917 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fce3002ff00 0x7fce300323c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fce380096a0 tx=0x7fce38006b40).stop 2023-11-15T01:49:46.750 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:46.748+0000 7fce488e9700 1 -- 172.21.15.55:0/2380160917 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fce44070430 msgr2=0x7fce4410a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:49:46.750 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:46.748+0000 7fce488e9700 1 --2- 172.21.15.55:0/2380160917 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fce44070430 0x7fce4410a830 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fce3400b190 tx=0x7fce34006ee0).stop 2023-11-15T01:49:46.750 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:46.748+0000 7fce488e9700 1 -- 172.21.15.55:0/2380160917 shutdown_connections 2023-11-15T01:49:46.750 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:46.748+0000 7fce488e9700 1 --2- 172.21.15.55:0/2380160917 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fce3002ff00 0x7fce300323c0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:49:46.751 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:46.748+0000 7fce488e9700 1 --2- 172.21.15.55:0/2380160917 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fce44070430 0x7fce4410a830 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:49:46.751 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:46.748+0000 7fce488e9700 1 -- 172.21.15.55:0/2380160917 >> 172.21.15.55:0/2380160917 conn(0x7fce440f6210 msgr2=0x7fce440f7cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:49:46.751 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:46.748+0000 7fce488e9700 1 -- 172.21.15.55:0/2380160917 shutdown_connections 2023-11-15T01:49:46.751 INFO:teuthology.orchestra.run.smithi055.stderr:/usr/bin/ceph: 2023-11-15T01:49:46.748+0000 7fce488e9700 1 -- 172.21.15.55:0/2380160917 wait complete. 2023-11-15T01:49:47.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:46 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:49:45.552234+0000 mgr.smithi055.szwngw (mgr.14164) 4 : audit [DBG] from='client.14176 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T01:49:47.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:46 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:49:45.806065+0000 mon.smithi055 (mon.0) 107 : audit [INF] from='mgr.14164 172.21.15.55:0/59463371' entity='mgr.smithi055.szwngw' 2023-11-15T01:49:47.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:46 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:49:46.746965+0000 mon.smithi055 (mon.0) 108 : audit [DBG] from='client.? 172.21.15.55:0/2380160917' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2023-11-15T01:49:47.108 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: systemctl is-enabled firewalld.service 2023-11-15T01:49:47.119 INFO:teuthology.orchestra.run.smithi055.stderr:systemctl: disabled 2023-11-15T01:49:47.120 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: systemctl is-active firewalld.service 2023-11-15T01:49:47.128 INFO:teuthology.orchestra.run.smithi055.stderr:systemctl: inactive 2023-11-15T01:49:47.129 INFO:teuthology.orchestra.run.smithi055.stderr:firewalld.service is not enabled 2023-11-15T01:49:47.129 INFO:teuthology.orchestra.run.smithi055.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2023-11-15T01:49:47.132 INFO:teuthology.orchestra.run.smithi055.stderr:Ceph Dashboard is now available at: 2023-11-15T01:49:47.132 INFO:teuthology.orchestra.run.smithi055.stderr: 2023-11-15T01:49:47.133 INFO:teuthology.orchestra.run.smithi055.stderr: URL: https://smithi055.front.sepia.ceph.com:8443/ 2023-11-15T01:49:47.133 INFO:teuthology.orchestra.run.smithi055.stderr: User: admin 2023-11-15T01:49:47.133 INFO:teuthology.orchestra.run.smithi055.stderr: Password: v85xlktk1u 2023-11-15T01:49:47.133 INFO:teuthology.orchestra.run.smithi055.stderr: 2023-11-15T01:49:47.134 INFO:teuthology.orchestra.run.smithi055.stderr:You can access the Ceph CLI with: 2023-11-15T01:49:47.134 INFO:teuthology.orchestra.run.smithi055.stderr: 2023-11-15T01:49:47.134 INFO:teuthology.orchestra.run.smithi055.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2023-11-15T01:49:47.135 INFO:teuthology.orchestra.run.smithi055.stderr: 2023-11-15T01:49:47.135 INFO:teuthology.orchestra.run.smithi055.stderr:Please consider enabling telemetry to help improve Ceph: 2023-11-15T01:49:47.135 INFO:teuthology.orchestra.run.smithi055.stderr: 2023-11-15T01:49:47.135 INFO:teuthology.orchestra.run.smithi055.stderr: ceph telemetry on 2023-11-15T01:49:47.135 INFO:teuthology.orchestra.run.smithi055.stderr: 2023-11-15T01:49:47.136 INFO:teuthology.orchestra.run.smithi055.stderr:For more information see: 2023-11-15T01:49:47.136 INFO:teuthology.orchestra.run.smithi055.stderr: 2023-11-15T01:49:47.136 INFO:teuthology.orchestra.run.smithi055.stderr: https://docs.ceph.com/docs/pacific/mgr/telemetry/ 2023-11-15T01:49:47.136 INFO:teuthology.orchestra.run.smithi055.stderr: 2023-11-15T01:49:47.136 INFO:teuthology.orchestra.run.smithi055.stderr:Bootstrap complete. 2023-11-15T01:49:47.136 INFO:teuthology.orchestra.run.smithi055.stderr:Releasing lock 139856514181832 on /run/cephadm/0cb4fc0c-8359-11ee-8dbe-212e2dc638e7.lock 2023-11-15T01:49:47.137 INFO:teuthology.orchestra.run.smithi055.stderr:Lock 139856514181832 released on /run/cephadm/0cb4fc0c-8359-11ee-8dbe-212e2dc638e7.lock 2023-11-15T01:49:47.174 INFO:tasks.cephadm:Fetching config... 2023-11-15T01:49:47.175 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2023-11-15T01:49:47.175 DEBUG:teuthology.orchestra.run.smithi055:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2023-11-15T01:49:47.197 INFO:tasks.cephadm:Fetching client.admin keyring... 2023-11-15T01:49:47.197 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2023-11-15T01:49:47.197 DEBUG:teuthology.orchestra.run.smithi055:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2023-11-15T01:49:47.260 INFO:tasks.cephadm:Fetching mon keyring... 2023-11-15T01:49:47.260 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2023-11-15T01:49:47.260 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/var/lib/ceph/0cb4fc0c-8359-11ee-8dbe-212e2dc638e7/mon.smithi055/keyring of=/dev/stdout 2023-11-15T01:49:47.335 INFO:tasks.cephadm:Fetching pub ssh key... 2023-11-15T01:49:47.335 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2023-11-15T01:49:47.336 DEBUG:teuthology.orchestra.run.smithi055:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2023-11-15T01:49:47.399 INFO:tasks.cephadm:Installing pub ssh key for root users... 2023-11-15T01:49:47.400 DEBUG:teuthology.orchestra.run.smithi055:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCxLJt0GaYl6FwRC+2jujyNfcjnFqDoIxZQt7xyvegv6X8cybULkGSYLvK2zO64L7UGRiP49oFJW7nNqZAhOpR5bTmfY1tfgvHj1I9aS541zHVvC1lr206T7ujPgW3N8/P0usLiIVeb3VwxdxL8LBQvx1YoULbTGxIzaxDCg9pRgf4pDURazpQxmxZ1XcbjkhKesOBEBAqPfibyGsLpAZTgkw4JGXS86QoKGCgEQ8/A/eN0h7UlAZoly3kFEqX5Owfb55y+YVEvZJ0TSRfxwzhHbwOmZy8cCVWFnsvj7rnzqIPNmcVwLz1jr9mXN/+wiaPm1r9XGg2OPRnEyd4YX7JreBaUSwX5vFW5zzvYLpuPHh3+yGTvzYZ/7VZ44LO1EFApLv229iASMFSG6kThktQM6Uk3/gZMKhDTOX6FHnG8RK5KtO0mF2B9wcj36FafX+DUkyVahKm6IsamAsiC29trMQrU8VQKuCduq8SRObbLbz1L1+2vXAOJoYgXXLAl8zE= ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-11-15T01:49:47.486 INFO:teuthology.orchestra.run.smithi055.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCxLJt0GaYl6FwRC+2jujyNfcjnFqDoIxZQt7xyvegv6X8cybULkGSYLvK2zO64L7UGRiP49oFJW7nNqZAhOpR5bTmfY1tfgvHj1I9aS541zHVvC1lr206T7ujPgW3N8/P0usLiIVeb3VwxdxL8LBQvx1YoULbTGxIzaxDCg9pRgf4pDURazpQxmxZ1XcbjkhKesOBEBAqPfibyGsLpAZTgkw4JGXS86QoKGCgEQ8/A/eN0h7UlAZoly3kFEqX5Owfb55y+YVEvZJ0TSRfxwzhHbwOmZy8cCVWFnsvj7rnzqIPNmcVwLz1jr9mXN/+wiaPm1r9XGg2OPRnEyd4YX7JreBaUSwX5vFW5zzvYLpuPHh3+yGTvzYZ/7VZ44LO1EFApLv229iASMFSG6kThktQM6Uk3/gZMKhDTOX6FHnG8RK5KtO0mF2B9wcj36FafX+DUkyVahKm6IsamAsiC29trMQrU8VQKuCduq8SRObbLbz1L1+2vXAOJoYgXXLAl8zE= ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 2023-11-15T01:49:47.499 DEBUG:teuthology.orchestra.run.smithi137:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCxLJt0GaYl6FwRC+2jujyNfcjnFqDoIxZQt7xyvegv6X8cybULkGSYLvK2zO64L7UGRiP49oFJW7nNqZAhOpR5bTmfY1tfgvHj1I9aS541zHVvC1lr206T7ujPgW3N8/P0usLiIVeb3VwxdxL8LBQvx1YoULbTGxIzaxDCg9pRgf4pDURazpQxmxZ1XcbjkhKesOBEBAqPfibyGsLpAZTgkw4JGXS86QoKGCgEQ8/A/eN0h7UlAZoly3kFEqX5Owfb55y+YVEvZJ0TSRfxwzhHbwOmZy8cCVWFnsvj7rnzqIPNmcVwLz1jr9mXN/+wiaPm1r9XGg2OPRnEyd4YX7JreBaUSwX5vFW5zzvYLpuPHh3+yGTvzYZ/7VZ44LO1EFApLv229iASMFSG6kThktQM6Uk3/gZMKhDTOX6FHnG8RK5KtO0mF2B9wcj36FafX+DUkyVahKm6IsamAsiC29trMQrU8VQKuCduq8SRObbLbz1L1+2vXAOJoYgXXLAl8zE= ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-11-15T01:49:47.542 INFO:teuthology.orchestra.run.smithi137.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCxLJt0GaYl6FwRC+2jujyNfcjnFqDoIxZQt7xyvegv6X8cybULkGSYLvK2zO64L7UGRiP49oFJW7nNqZAhOpR5bTmfY1tfgvHj1I9aS541zHVvC1lr206T7ujPgW3N8/P0usLiIVeb3VwxdxL8LBQvx1YoULbTGxIzaxDCg9pRgf4pDURazpQxmxZ1XcbjkhKesOBEBAqPfibyGsLpAZTgkw4JGXS86QoKGCgEQ8/A/eN0h7UlAZoly3kFEqX5Owfb55y+YVEvZJ0TSRfxwzhHbwOmZy8cCVWFnsvj7rnzqIPNmcVwLz1jr9mXN/+wiaPm1r9XGg2OPRnEyd4YX7JreBaUSwX5vFW5zzvYLpuPHh3+yGTvzYZ/7VZ44LO1EFApLv229iASMFSG6kThktQM6Uk3/gZMKhDTOX6FHnG8RK5KtO0mF2B9wcj36FafX+DUkyVahKm6IsamAsiC29trMQrU8VQKuCduq8SRObbLbz1L1+2vXAOJoYgXXLAl8zE= ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 2023-11-15T01:49:47.554 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2023-11-15T01:49:48.995 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-15T01:49:48.993+0000 7f748fceb700 1 -- 172.21.15.55:0/607599594 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7488060200 msgr2=0x7f74880fb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:49:48.996 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-15T01:49:48.993+0000 7f748fceb700 1 --2- 172.21.15.55:0/607599594 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7488060200 0x7f74880fb570 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f7484009fb0 tx=0x7f7484009b30).stop 2023-11-15T01:49:48.996 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-15T01:49:48.993+0000 7f748fceb700 1 -- 172.21.15.55:0/607599594 shutdown_connections 2023-11-15T01:49:48.996 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-15T01:49:48.993+0000 7f748fceb700 1 --2- 172.21.15.55:0/607599594 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7488060200 0x7f74880fb570 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:49:48.996 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-15T01:49:48.993+0000 7f748fceb700 1 -- 172.21.15.55:0/607599594 >> 172.21.15.55:0/607599594 conn(0x7f74880f0b70 msgr2=0x7f74880f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:49:48.997 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-15T01:49:48.994+0000 7f748fceb700 1 -- 172.21.15.55:0/607599594 shutdown_connections 2023-11-15T01:49:48.997 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-15T01:49:48.994+0000 7f748fceb700 1 -- 172.21.15.55:0/607599594 wait complete. 2023-11-15T01:49:48.997 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-15T01:49:48.994+0000 7f748fceb700 1 Processor -- start 2023-11-15T01:49:48.997 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-15T01:49:48.994+0000 7f748fceb700 1 -- start start 2023-11-15T01:49:48.997 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-15T01:49:48.994+0000 7f748fceb700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7488060200 0x7f74881088b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:49:48.998 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-15T01:49:48.994+0000 7f748fceb700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7488108da0 con 0x7f7488060200 2023-11-15T01:49:48.998 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-15T01:49:48.994+0000 7f748da87700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7488060200 0x7f74881088b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:49:48.998 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-15T01:49:48.995+0000 7f748da87700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7488060200 0x7f74881088b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:40204/0 (socket says 172.21.15.55:40204) 2023-11-15T01:49:48.998 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-15T01:49:48.995+0000 7f748da87700 1 -- 172.21.15.55:0/2602567150 learned_addr learned my addr 172.21.15.55:0/2602567150 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-11-15T01:49:48.998 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-15T01:49:48.995+0000 7f748da87700 1 -- 172.21.15.55:0/2602567150 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7484005040 con 0x7f7488060200 2023-11-15T01:49:48.999 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-15T01:49:48.995+0000 7f748da87700 1 --2- 172.21.15.55:0/2602567150 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7488060200 0x7f74881088b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f7484000c00 tx=0x7f748400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:49:48.999 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-15T01:49:48.995+0000 7f747e7fc700 1 -- 172.21.15.55:0/2602567150 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f74840095f0 con 0x7f7488060200 2023-11-15T01:49:48.999 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-15T01:49:48.995+0000 7f747e7fc700 1 -- 172.21.15.55:0/2602567150 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f7484007e30 con 0x7f7488060200 2023-11-15T01:49:49.001 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-15T01:49:48.995+0000 7f748fceb700 1 -- 172.21.15.55:0/2602567150 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7488108fa0 con 0x7f7488060200 2023-11-15T01:49:49.001 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-15T01:49:48.995+0000 7f747e7fc700 1 -- 172.21.15.55:0/2602567150 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7484014d00 con 0x7f7488060200 2023-11-15T01:49:49.003 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-15T01:49:48.995+0000 7f748fceb700 1 -- 172.21.15.55:0/2602567150 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7488109430 con 0x7f7488060200 2023-11-15T01:49:49.003 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-15T01:49:48.996+0000 7f747e7fc700 1 -- 172.21.15.55:0/2602567150 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7484025070 con 0x7f7488060200 2023-11-15T01:49:49.003 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-15T01:49:48.996+0000 7f747e7fc700 1 --2- 172.21.15.55:0/2602567150 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f747402ff60 0x7f7474032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:49:49.003 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-15T01:49:48.996+0000 7f747e7fc700 1 -- 172.21.15.55:0/2602567150 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f74840408b0 con 0x7f7488060200 2023-11-15T01:49:49.004 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-15T01:49:48.996+0000 7f748fceb700 1 -- 172.21.15.55:0/2602567150 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7488045ad0 con 0x7f7488060200 2023-11-15T01:49:49.004 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-15T01:49:48.996+0000 7f748d286700 1 --2- 172.21.15.55:0/2602567150 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f747402ff60 0x7f7474032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:49:49.004 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-15T01:49:48.997+0000 7f748d286700 1 --2- 172.21.15.55:0/2602567150 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f747402ff60 0x7f7474032420 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f74780096a0 tx=0x7f7478006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:49:49.004 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-15T01:49:49.001+0000 7f747e7fc700 1 -- 172.21.15.55:0/2602567150 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f748400f300 con 0x7f7488060200 2023-11-15T01:49:49.166 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-15T01:49:49.166+0000 7f748fceb700 1 -- 172.21.15.55:0/2602567150 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f748810a1c0 con 0x7f7488060200 2023-11-15T01:49:49.170 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-15T01:49:49.170+0000 7f747e7fc700 1 -- 172.21.15.55:0/2602567150 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v10)=0 v10) v1 ==== 125+0+0 (secure 0 0 0) 0x7f748400f300 con 0x7f7488060200 2023-11-15T01:49:49.172 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-15T01:49:49.171+0000 7f748fceb700 1 -- 172.21.15.55:0/2602567150 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f747402ff60 msgr2=0x7f7474032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:49:49.172 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-15T01:49:49.171+0000 7f748fceb700 1 --2- 172.21.15.55:0/2602567150 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f747402ff60 0x7f7474032420 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f74780096a0 tx=0x7f7478006b40).stop 2023-11-15T01:49:49.172 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-15T01:49:49.171+0000 7f748fceb700 1 -- 172.21.15.55:0/2602567150 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7488060200 msgr2=0x7f74881088b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:49:49.173 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-15T01:49:49.171+0000 7f748fceb700 1 --2- 172.21.15.55:0/2602567150 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7488060200 0x7f74881088b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f7484000c00 tx=0x7f748400b040).stop 2023-11-15T01:49:49.173 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-15T01:49:49.172+0000 7f748fceb700 1 -- 172.21.15.55:0/2602567150 shutdown_connections 2023-11-15T01:49:49.173 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-15T01:49:49.172+0000 7f748fceb700 1 --2- 172.21.15.55:0/2602567150 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f747402ff60 0x7f7474032420 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:49:49.173 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-15T01:49:49.172+0000 7f748fceb700 1 --2- 172.21.15.55:0/2602567150 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7488060200 0x7f74881088b0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:49:49.174 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-15T01:49:49.172+0000 7f748fceb700 1 -- 172.21.15.55:0/2602567150 >> 172.21.15.55:0/2602567150 conn(0x7f74880f0b70 msgr2=0x7f74880f1640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:49:49.174 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-15T01:49:49.172+0000 7f748fceb700 1 -- 172.21.15.55:0/2602567150 shutdown_connections 2023-11-15T01:49:49.174 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-15T01:49:49.172+0000 7f748fceb700 1 -- 172.21.15.55:0/2602567150 wait complete. 2023-11-15T01:49:49.552 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi137 2023-11-15T01:49:49.552 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2023-11-15T01:49:49.553 DEBUG:teuthology.orchestra.run.smithi137:> dd of=/etc/ceph/ceph.conf 2023-11-15T01:49:49.574 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2023-11-15T01:49:49.574 DEBUG:teuthology.orchestra.run.smithi137:> dd of=/etc/ceph/ceph.client.admin.keyring 2023-11-15T01:49:49.635 INFO:tasks.cephadm:Adding host smithi137 to orchestrator... 2023-11-15T01:49:49.635 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph orch host add smithi137 2023-11-15T01:49:50.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:49 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:49:48.712990+0000 mon.smithi055 (mon.0) 109 : cluster [DBG] mgrmap e17: smithi055.szwngw(active, since 6s) 2023-11-15T01:49:50.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:49 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:49:49.170539+0000 mon.smithi055 (mon.0) 110 : audit [INF] from='client.? 172.21.15.55:0/2602567150' entity='client.admin' 2023-11-15T01:49:51.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:51 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:49:50.757290+0000 mon.smithi055 (mon.0) 111 : audit [INF] from='mgr.14164 172.21.15.55:0/59463371' entity='mgr.smithi055.szwngw' 2023-11-15T01:49:51.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:51 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:49:51.003382+0000 mon.smithi055 (mon.0) 112 : audit [INF] from='mgr.14164 172.21.15.55:0/59463371' entity='mgr.smithi055.szwngw' 2023-11-15T01:49:53.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:52 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:49:51.787266+0000 mon.smithi055 (mon.0) 113 : audit [INF] from='mgr.14164 172.21.15.55:0/59463371' entity='mgr.smithi055.szwngw' 2023-11-15T01:49:55.578 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:55 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:49:54.241548+0000 mon.smithi055 (mon.0) 114 : audit [INF] from='mgr.14164 172.21.15.55:0/59463371' entity='mgr.smithi055.szwngw' 2023-11-15T01:49:55.578 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:55 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:49:54.545430+0000 mon.smithi055 (mon.0) 115 : audit [INF] from='mgr.14164 172.21.15.55:0/59463371' entity='mgr.smithi055.szwngw' 2023-11-15T01:49:55.578 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:55 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:49:54.550011+0000 mon.smithi055 (mon.0) 116 : audit [INF] from='mgr.14164 172.21.15.55:0/59463371' entity='mgr.smithi055.szwngw' 2023-11-15T01:49:55.579 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:49:55 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cephadm 2023-11-15T01:49:54.557006+0000 mgr.smithi055.szwngw (mgr.14164) 5 : cephadm [INF] Deploying daemon alertmanager.smithi055 on smithi055 2023-11-15T01:50:04.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:50:03 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:50:02.691265+0000 mgr.smithi055.szwngw (mgr.14164) 6 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:50:06.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:50:05 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:50:04.691634+0000 mgr.smithi055.szwngw (mgr.14164) 7 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:50:08.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:50:07 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:50:06.692045+0000 mgr.smithi055.szwngw (mgr.14164) 8 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:50:10.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:50:09 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:50:08.692459+0000 mgr.smithi055.szwngw (mgr.14164) 9 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:50:11.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:50:11 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:50:10.692773+0000 mgr.smithi055.szwngw (mgr.14164) 10 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:50:14.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:50:13 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:50:12.693005+0000 mgr.smithi055.szwngw (mgr.14164) 11 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:50:15.194 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:15.191+0000 7f0cf38cd700 1 -- 172.21.15.137:0/2692848719 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0cec0f5410 msgr2=0x7f0cec0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:50:15.194 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:15.191+0000 7f0cf38cd700 1 --2- 172.21.15.137:0/2692848719 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0cec0f5410 0x7f0cec0f5830 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f0cdc009fb0 tx=0x7f0cdc009b30).stop 2023-11-15T01:50:15.195 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:15.191+0000 7f0cf38cd700 1 -- 172.21.15.137:0/2692848719 shutdown_connections 2023-11-15T01:50:15.195 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:15.191+0000 7f0cf38cd700 1 --2- 172.21.15.137:0/2692848719 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0cec0f5410 0x7f0cec0f5830 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:50:15.195 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:15.191+0000 7f0cf38cd700 1 -- 172.21.15.137:0/2692848719 >> 172.21.15.137:0/2692848719 conn(0x7f0cec0f0b70 msgr2=0x7f0cec0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:50:15.195 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:15.191+0000 7f0cf38cd700 1 -- 172.21.15.137:0/2692848719 shutdown_connections 2023-11-15T01:50:15.195 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:15.192+0000 7f0cf38cd700 1 -- 172.21.15.137:0/2692848719 wait complete. 2023-11-15T01:50:15.196 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:15.192+0000 7f0cf38cd700 1 Processor -- start 2023-11-15T01:50:15.196 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:15.192+0000 7f0cf38cd700 1 -- start start 2023-11-15T01:50:15.196 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:15.192+0000 7f0cf38cd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0cec0f5410 0x7f0cec105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:50:15.196 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:15.192+0000 7f0cf38cd700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cec105b20 con 0x7f0cec0f5410 2023-11-15T01:50:15.197 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:15.192+0000 7f0cf1669700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0cec0f5410 0x7f0cec105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:50:15.197 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:15.192+0000 7f0cf1669700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0cec0f5410 0x7f0cec105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:46092/0 (socket says 172.21.15.137:46092) 2023-11-15T01:50:15.197 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:15.192+0000 7f0cf1669700 1 -- 172.21.15.137:0/1402822740 learned_addr learned my addr 172.21.15.137:0/1402822740 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:50:15.197 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:15.193+0000 7f0cf1669700 1 -- 172.21.15.137:0/1402822740 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cdc005040 con 0x7f0cec0f5410 2023-11-15T01:50:15.197 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:15.193+0000 7f0cf1669700 1 --2- 172.21.15.137:0/1402822740 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0cec0f5410 0x7f0cec105630 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f0cdc013510 tx=0x7f0cdc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:50:15.197 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:15.193+0000 7f0ce1ffb700 1 -- 172.21.15.137:0/1402822740 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0cdc009320 con 0x7f0cec0f5410 2023-11-15T01:50:15.198 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:15.193+0000 7f0ce1ffb700 1 -- 172.21.15.137:0/1402822740 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0cdc013f00 con 0x7f0cec0f5410 2023-11-15T01:50:15.198 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:15.193+0000 7f0cf38cd700 1 -- 172.21.15.137:0/1402822740 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cec105d20 con 0x7f0cec0f5410 2023-11-15T01:50:15.198 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:15.193+0000 7f0ce1ffb700 1 -- 172.21.15.137:0/1402822740 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0cdc014d90 con 0x7f0cec0f5410 2023-11-15T01:50:15.198 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:15.193+0000 7f0cf38cd700 1 -- 172.21.15.137:0/1402822740 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cec1061b0 con 0x7f0cec0f5410 2023-11-15T01:50:15.200 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:15.194+0000 7f0ce1ffb700 1 -- 172.21.15.137:0/1402822740 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0cdc014420 con 0x7f0cec0f5410 2023-11-15T01:50:15.200 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:15.194+0000 7f0cf38cd700 1 -- 172.21.15.137:0/1402822740 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cec045ad0 con 0x7f0cec0f5410 2023-11-15T01:50:15.200 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:15.194+0000 7f0ce1ffb700 1 --2- 172.21.15.137:0/1402822740 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f0cd802ff60 0x7f0cd8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:50:15.200 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:15.194+0000 7f0ce1ffb700 1 -- 172.21.15.137:0/1402822740 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0cdc041000 con 0x7f0cec0f5410 2023-11-15T01:50:15.201 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:15.194+0000 7f0cf0e68700 1 --2- 172.21.15.137:0/1402822740 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f0cd802ff60 0x7f0cd8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:50:15.201 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:15.195+0000 7f0cf0e68700 1 --2- 172.21.15.137:0/1402822740 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f0cd802ff60 0x7f0cd8032420 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f0ce80096a0 tx=0x7f0ce8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:50:15.201 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:15.199+0000 7f0ce1ffb700 1 -- 172.21.15.137:0/1402822740 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0cdc0095f0 con 0x7f0cec0f5410 2023-11-15T01:50:15.369 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:15.368+0000 7f0cf38cd700 1 -- 172.21.15.137:0/1402822740 --> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi137", "target": ["mon-mgr", ""]}) v1 -- 0x7f0cec0f1d70 con 0x7f0cd802ff60 2023-11-15T01:50:15.976 INFO:teuthology.orchestra.run.smithi137.stdout:Added host 'smithi137' 2023-11-15T01:50:15.976 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:15.973+0000 7f0ce1ffb700 1 -- 172.21.15.137:0/1402822740 <== mgr.14164 v2:172.21.15.55:6800/157802929 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7f0cec0f1d70 con 0x7f0cd802ff60 2023-11-15T01:50:15.977 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:15.975+0000 7f0cf38cd700 1 -- 172.21.15.137:0/1402822740 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f0cd802ff60 msgr2=0x7f0cd8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:50:15.977 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:15.975+0000 7f0cf38cd700 1 --2- 172.21.15.137:0/1402822740 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f0cd802ff60 0x7f0cd8032420 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f0ce80096a0 tx=0x7f0ce8006b40).stop 2023-11-15T01:50:15.978 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:15.975+0000 7f0cf38cd700 1 -- 172.21.15.137:0/1402822740 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0cec0f5410 msgr2=0x7f0cec105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:50:15.978 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:15.975+0000 7f0cf38cd700 1 --2- 172.21.15.137:0/1402822740 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0cec0f5410 0x7f0cec105630 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f0cdc013510 tx=0x7f0cdc00b040).stop 2023-11-15T01:50:15.978 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:15.976+0000 7f0cf38cd700 1 -- 172.21.15.137:0/1402822740 shutdown_connections 2023-11-15T01:50:15.978 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:15.976+0000 7f0cf38cd700 1 --2- 172.21.15.137:0/1402822740 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f0cd802ff60 0x7f0cd8032420 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:50:15.978 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:15.976+0000 7f0cf38cd700 1 --2- 172.21.15.137:0/1402822740 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0cec0f5410 0x7f0cec105630 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:50:15.979 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:15.976+0000 7f0cf38cd700 1 -- 172.21.15.137:0/1402822740 >> 172.21.15.137:0/1402822740 conn(0x7f0cec0f0b70 msgr2=0x7f0cec0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:50:15.979 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:15.976+0000 7f0cf38cd700 1 -- 172.21.15.137:0/1402822740 shutdown_connections 2023-11-15T01:50:15.979 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:15.976+0000 7f0cf38cd700 1 -- 172.21.15.137:0/1402822740 wait complete. 2023-11-15T01:50:16.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:50:15 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:50:14.693388+0000 mgr.smithi055.szwngw (mgr.14164) 12 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:50:16.309 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph orch host ls --format=json 2023-11-15T01:50:17.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:50:16 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:50:15.370325+0000 mgr.smithi055.szwngw (mgr.14164) 13 : audit [DBG] from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi137", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T01:50:17.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:50:16 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:50:15.974829+0000 mon.smithi055 (mon.0) 117 : audit [INF] from='mgr.14164 172.21.15.55:0/59463371' entity='mgr.smithi055.szwngw' 2023-11-15T01:50:17.329 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:50:16 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cephadm 2023-11-15T01:50:15.975261+0000 mgr.smithi055.szwngw (mgr.14164) 14 : cephadm [INF] Added host smithi137 2023-11-15T01:50:17.429 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:17.427+0000 7f8d0eee3700 1 -- 172.21.15.137:0/333203493 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8d080f6510 msgr2=0x7f8d080f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:50:17.429 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:17.427+0000 7f8d0eee3700 1 --2- 172.21.15.137:0/333203493 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8d080f6510 0x7f8d080f6930 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f8cfc009fb0 tx=0x7f8cfc009b30).stop 2023-11-15T01:50:17.430 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:17.427+0000 7f8d0eee3700 1 -- 172.21.15.137:0/333203493 shutdown_connections 2023-11-15T01:50:17.430 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:17.427+0000 7f8d0eee3700 1 --2- 172.21.15.137:0/333203493 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8d080f6510 0x7f8d080f6930 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:50:17.430 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:17.427+0000 7f8d0eee3700 1 -- 172.21.15.137:0/333203493 >> 172.21.15.137:0/333203493 conn(0x7f8d080f1c70 msgr2=0x7f8d080f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:50:17.430 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:17.427+0000 7f8d0eee3700 1 -- 172.21.15.137:0/333203493 shutdown_connections 2023-11-15T01:50:17.431 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:17.427+0000 7f8d0eee3700 1 -- 172.21.15.137:0/333203493 wait complete. 2023-11-15T01:50:17.432 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:17.428+0000 7f8d0eee3700 1 Processor -- start 2023-11-15T01:50:17.433 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:17.428+0000 7f8d0eee3700 1 -- start start 2023-11-15T01:50:17.433 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:17.428+0000 7f8d0eee3700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8d080f6510 0x7f8d08106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:50:17.433 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:17.428+0000 7f8d0eee3700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d08106c20 con 0x7f8d080f6510 2023-11-15T01:50:17.433 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:17.428+0000 7f8d0cc7f700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8d080f6510 0x7f8d08106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:50:17.433 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:17.428+0000 7f8d0cc7f700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8d080f6510 0x7f8d08106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:46116/0 (socket says 172.21.15.137:46116) 2023-11-15T01:50:17.434 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:17.428+0000 7f8d0cc7f700 1 -- 172.21.15.137:0/4142861414 learned_addr learned my addr 172.21.15.137:0/4142861414 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:50:17.434 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:17.429+0000 7f8d0cc7f700 1 -- 172.21.15.137:0/4142861414 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cfc005040 con 0x7f8d080f6510 2023-11-15T01:50:17.434 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:17.429+0000 7f8d0cc7f700 1 --2- 172.21.15.137:0/4142861414 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8d080f6510 0x7f8d08106730 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f8cfc013510 tx=0x7f8cfc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:50:17.434 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:17.429+0000 7f8d057fa700 1 -- 172.21.15.137:0/4142861414 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8cfc009320 con 0x7f8d080f6510 2023-11-15T01:50:17.434 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:17.429+0000 7f8d0eee3700 1 -- 172.21.15.137:0/4142861414 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d08106e20 con 0x7f8d080f6510 2023-11-15T01:50:17.435 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:17.429+0000 7f8d057fa700 1 -- 172.21.15.137:0/4142861414 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8cfc013f00 con 0x7f8d080f6510 2023-11-15T01:50:17.435 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:17.429+0000 7f8d057fa700 1 -- 172.21.15.137:0/4142861414 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8cfc014d90 con 0x7f8d080f6510 2023-11-15T01:50:17.435 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:17.429+0000 7f8d0eee3700 1 -- 172.21.15.137:0/4142861414 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d081072b0 con 0x7f8d080f6510 2023-11-15T01:50:17.435 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:17.430+0000 7f8d057fa700 1 -- 172.21.15.137:0/4142861414 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8cfc0095f0 con 0x7f8d080f6510 2023-11-15T01:50:17.435 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:17.430+0000 7f8d0eee3700 1 -- 172.21.15.137:0/4142861414 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d08045ad0 con 0x7f8d080f6510 2023-11-15T01:50:17.436 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:17.430+0000 7f8d057fa700 1 --2- 172.21.15.137:0/4142861414 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f8cf802ff60 0x7f8cf8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:50:17.436 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:17.430+0000 7f8d057fa700 1 -- 172.21.15.137:0/4142861414 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8cfc041000 con 0x7f8d080f6510 2023-11-15T01:50:17.436 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:17.430+0000 7f8d07fff700 1 --2- 172.21.15.137:0/4142861414 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f8cf802ff60 0x7f8cf8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:50:17.436 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:17.431+0000 7f8d07fff700 1 --2- 172.21.15.137:0/4142861414 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f8cf802ff60 0x7f8cf8032420 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f8cf40096a0 tx=0x7f8cf4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:50:17.437 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:17.435+0000 7f8d057fa700 1 -- 172.21.15.137:0/4142861414 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8cfc014420 con 0x7f8d080f6510 2023-11-15T01:50:17.606 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:17.604+0000 7f8d0eee3700 1 -- 172.21.15.137:0/4142861414 --> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8d080f2e70 con 0x7f8cf802ff60 2023-11-15T01:50:17.607 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:17.606+0000 7f8d057fa700 1 -- 172.21.15.137:0/4142861414 <== mgr.14164 v2:172.21.15.55:6800/157802929 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+156 (secure 0 0 0) 0x7f8d080f2e70 con 0x7f8cf802ff60 2023-11-15T01:50:17.608 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:50:17.608 INFO:teuthology.orchestra.run.smithi137.stdout:[{"addr": "172.21.15.55", "hostname": "smithi055", "labels": [], "status": ""}, {"addr": "smithi137", "hostname": "smithi137", "labels": [], "status": ""}] 2023-11-15T01:50:17.609 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:17.607+0000 7f8d0eee3700 1 -- 172.21.15.137:0/4142861414 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f8cf802ff60 msgr2=0x7f8cf8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:50:17.609 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:17.607+0000 7f8d0eee3700 1 --2- 172.21.15.137:0/4142861414 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f8cf802ff60 0x7f8cf8032420 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f8cf40096a0 tx=0x7f8cf4006b40).stop 2023-11-15T01:50:17.610 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:17.607+0000 7f8d0eee3700 1 -- 172.21.15.137:0/4142861414 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8d080f6510 msgr2=0x7f8d08106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:50:17.610 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:17.607+0000 7f8d0eee3700 1 --2- 172.21.15.137:0/4142861414 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8d080f6510 0x7f8d08106730 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f8cfc013510 tx=0x7f8cfc00b040).stop 2023-11-15T01:50:17.610 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:17.607+0000 7f8d0eee3700 1 -- 172.21.15.137:0/4142861414 shutdown_connections 2023-11-15T01:50:17.610 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:17.607+0000 7f8d0eee3700 1 --2- 172.21.15.137:0/4142861414 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f8cf802ff60 0x7f8cf8032420 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:50:17.611 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:17.607+0000 7f8d0eee3700 1 --2- 172.21.15.137:0/4142861414 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8d080f6510 0x7f8d08106730 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:50:17.611 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:17.607+0000 7f8d0eee3700 1 -- 172.21.15.137:0/4142861414 >> 172.21.15.137:0/4142861414 conn(0x7f8d080f1c70 msgr2=0x7f8d080f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:50:17.611 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:17.607+0000 7f8d0eee3700 1 -- 172.21.15.137:0/4142861414 shutdown_connections 2023-11-15T01:50:17.611 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:17.608+0000 7f8d0eee3700 1 -- 172.21.15.137:0/4142861414 wait complete. 2023-11-15T01:50:17.941 INFO:tasks.cephadm:Setting crush tunables to default 2023-11-15T01:50:17.942 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph osd crush tunables default 2023-11-15T01:50:18.286 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:50:17 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:50:16.693754+0000 mgr.smithi055.szwngw (mgr.14164) 15 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:50:18.287 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:50:17 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:50:17.607442+0000 mgr.smithi055.szwngw (mgr.14164) 16 : audit [DBG] from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-15T01:50:19.116 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-15T01:50:19.114+0000 7fe0cf59e700 1 -- 172.21.15.55:0/913451699 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe0d00601e0 msgr2=0x7fe0d00f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:50:19.117 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-15T01:50:19.114+0000 7fe0cf59e700 1 --2- 172.21.15.55:0/913451699 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe0d00601e0 0x7fe0d00f8f20 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7fe0c0009fb0 tx=0x7fe0c0009b30).stop 2023-11-15T01:50:19.117 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-15T01:50:19.114+0000 7fe0cf59e700 1 -- 172.21.15.55:0/913451699 shutdown_connections 2023-11-15T01:50:19.117 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-15T01:50:19.114+0000 7fe0cf59e700 1 --2- 172.21.15.55:0/913451699 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe0d00601e0 0x7fe0d00f8f20 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:50:19.118 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-15T01:50:19.114+0000 7fe0cf59e700 1 -- 172.21.15.55:0/913451699 >> 172.21.15.55:0/913451699 conn(0x7fe0d00f09a0 msgr2=0x7fe0d00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:50:19.118 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-15T01:50:19.114+0000 7fe0cf59e700 1 -- 172.21.15.55:0/913451699 shutdown_connections 2023-11-15T01:50:19.118 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-15T01:50:19.114+0000 7fe0cf59e700 1 -- 172.21.15.55:0/913451699 wait complete. 2023-11-15T01:50:19.118 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-15T01:50:19.114+0000 7fe0cf59e700 1 Processor -- start 2023-11-15T01:50:19.118 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-15T01:50:19.115+0000 7fe0cf59e700 1 -- start start 2023-11-15T01:50:19.119 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-15T01:50:19.115+0000 7fe0cf59e700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe0d00601e0 0x7fe0d0106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:50:19.119 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-15T01:50:19.115+0000 7fe0cf59e700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0d01069f0 con 0x7fe0d00601e0 2023-11-15T01:50:19.119 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-15T01:50:19.115+0000 7fe0ce59c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe0d00601e0 0x7fe0d0106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:50:19.119 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-15T01:50:19.115+0000 7fe0ce59c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe0d00601e0 0x7fe0d0106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:48224/0 (socket says 172.21.15.55:48224) 2023-11-15T01:50:19.119 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-15T01:50:19.115+0000 7fe0ce59c700 1 -- 172.21.15.55:0/3386345908 learned_addr learned my addr 172.21.15.55:0/3386345908 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-11-15T01:50:19.120 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-15T01:50:19.115+0000 7fe0ce59c700 1 -- 172.21.15.55:0/3386345908 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0c0005040 con 0x7fe0d00601e0 2023-11-15T01:50:19.120 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-15T01:50:19.115+0000 7fe0ce59c700 1 --2- 172.21.15.55:0/3386345908 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe0d00601e0 0x7fe0d0106500 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7fe0c0013510 tx=0x7fe0c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:50:19.120 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-15T01:50:19.116+0000 7fe0beffd700 1 -- 172.21.15.55:0/3386345908 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe0c0009320 con 0x7fe0d00601e0 2023-11-15T01:50:19.120 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-15T01:50:19.116+0000 7fe0beffd700 1 -- 172.21.15.55:0/3386345908 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe0c0013f00 con 0x7fe0d00601e0 2023-11-15T01:50:19.121 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-15T01:50:19.116+0000 7fe0cf59e700 1 -- 172.21.15.55:0/3386345908 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0d0106bf0 con 0x7fe0d00601e0 2023-11-15T01:50:19.121 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-15T01:50:19.116+0000 7fe0beffd700 1 -- 172.21.15.55:0/3386345908 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe0c0014d90 con 0x7fe0d00601e0 2023-11-15T01:50:19.121 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-15T01:50:19.116+0000 7fe0cf59e700 1 -- 172.21.15.55:0/3386345908 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0d0107080 con 0x7fe0d00601e0 2023-11-15T01:50:19.122 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-15T01:50:19.116+0000 7fe0beffd700 1 -- 172.21.15.55:0/3386345908 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe0c00263a0 con 0x7fe0d00601e0 2023-11-15T01:50:19.122 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-15T01:50:19.117+0000 7fe0cf59e700 1 -- 172.21.15.55:0/3386345908 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0d0045ad0 con 0x7fe0d00601e0 2023-11-15T01:50:19.123 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-15T01:50:19.117+0000 7fe0beffd700 1 --2- 172.21.15.55:0/3386345908 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fe0b802ff60 0x7fe0b8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:50:19.123 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-15T01:50:19.117+0000 7fe0beffd700 1 -- 172.21.15.55:0/3386345908 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe0c00410a0 con 0x7fe0d00601e0 2023-11-15T01:50:19.123 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-15T01:50:19.117+0000 7fe0cdd9b700 1 --2- 172.21.15.55:0/3386345908 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fe0b802ff60 0x7fe0b8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:50:19.123 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-15T01:50:19.117+0000 7fe0cdd9b700 1 --2- 172.21.15.55:0/3386345908 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fe0b802ff60 0x7fe0b8032420 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7fe0c40096a0 tx=0x7fe0c4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:50:19.123 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-15T01:50:19.121+0000 7fe0beffd700 1 -- 172.21.15.55:0/3386345908 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe0c00126f0 con 0x7fe0d00601e0 2023-11-15T01:50:19.292 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-15T01:50:19.291+0000 7fe0cf59e700 1 -- 172.21.15.55:0/3386345908 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fe0d0107f30 con 0x7fe0d00601e0 2023-11-15T01:50:19.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:50:18 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:50:18.694028+0000 mgr.smithi055.szwngw (mgr.14164) 17 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:50:20.010 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-15T01:50:20.009+0000 7fe0beffd700 1 -- 172.21.15.55:0/3386345908 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v5) v1 ==== 124+0+0 (secure 0 0 0) 0x7fe0c0012330 con 0x7fe0d00601e0 2023-11-15T01:50:20.012 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-15T01:50:20.011+0000 7fe0cf59e700 1 -- 172.21.15.55:0/3386345908 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fe0b802ff60 msgr2=0x7fe0b8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:50:20.012 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-15T01:50:20.011+0000 7fe0cf59e700 1 --2- 172.21.15.55:0/3386345908 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fe0b802ff60 0x7fe0b8032420 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7fe0c40096a0 tx=0x7fe0c4006b40).stop 2023-11-15T01:50:20.013 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-15T01:50:20.011+0000 7fe0cf59e700 1 -- 172.21.15.55:0/3386345908 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe0d00601e0 msgr2=0x7fe0d0106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:50:20.013 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-15T01:50:20.011+0000 7fe0cf59e700 1 --2- 172.21.15.55:0/3386345908 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe0d00601e0 0x7fe0d0106500 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7fe0c0013510 tx=0x7fe0c000b040).stop 2023-11-15T01:50:20.013 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-15T01:50:20.012+0000 7fe0cf59e700 1 -- 172.21.15.55:0/3386345908 shutdown_connections 2023-11-15T01:50:20.013 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-15T01:50:20.012+0000 7fe0cf59e700 1 --2- 172.21.15.55:0/3386345908 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fe0b802ff60 0x7fe0b8032420 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:50:20.013 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-15T01:50:20.012+0000 7fe0cf59e700 1 --2- 172.21.15.55:0/3386345908 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe0d00601e0 0x7fe0d0106500 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:50:20.014 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-15T01:50:20.012+0000 7fe0cf59e700 1 -- 172.21.15.55:0/3386345908 >> 172.21.15.55:0/3386345908 conn(0x7fe0d00f09a0 msgr2=0x7fe0d00f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:50:20.014 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-15T01:50:20.012+0000 7fe0cf59e700 1 -- 172.21.15.55:0/3386345908 shutdown_connections 2023-11-15T01:50:20.014 INFO:teuthology.orchestra.run.smithi055.stderr:2023-11-15T01:50:20.012+0000 7fe0cf59e700 1 -- 172.21.15.55:0/3386345908 wait complete. 2023-11-15T01:50:20.015 INFO:teuthology.orchestra.run.smithi055.stderr:adjusted tunables profile to default 2023-11-15T01:50:20.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:50:19 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:50:19.292931+0000 mon.smithi055 (mon.0) 118 : audit [INF] from='client.? 172.21.15.55:0/3386345908' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-11-15T01:50:20.404 INFO:tasks.cephadm:Adding mon.smithi055 on smithi055 2023-11-15T01:50:20.405 INFO:tasks.cephadm:Adding mon.smithi137 on smithi137 2023-11-15T01:50:20.405 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph orch apply mon '2;smithi055:172.21.15.55=smithi055;smithi137:172.21.15.137=smithi137' 2023-11-15T01:50:21.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:50:21 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:50:20.010155+0000 mon.smithi055 (mon.0) 119 : audit [INF] from='client.? 172.21.15.55:0/3386345908' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-11-15T01:50:21.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:50:21 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:50:20.010209+0000 mon.smithi055 (mon.0) 120 : cluster [DBG] osdmap e5: 0 total, 0 up, 0 in 2023-11-15T01:50:21.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:50:21 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:50:20.694695+0000 mgr.smithi055.szwngw (mgr.14164) 18 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:50:21.510 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:21.507+0000 7f8ded94a700 1 -- 172.21.15.137:0/494896618 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8de80f6510 msgr2=0x7f8de80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:50:21.510 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:21.507+0000 7f8ded94a700 1 --2- 172.21.15.137:0/494896618 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8de80f6510 0x7f8de80f6930 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f8ddc009fb0 tx=0x7f8ddc009b30).stop 2023-11-15T01:50:21.511 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:21.507+0000 7f8ded94a700 1 -- 172.21.15.137:0/494896618 shutdown_connections 2023-11-15T01:50:21.511 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:21.507+0000 7f8ded94a700 1 --2- 172.21.15.137:0/494896618 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8de80f6510 0x7f8de80f6930 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:50:21.511 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:21.507+0000 7f8ded94a700 1 -- 172.21.15.137:0/494896618 >> 172.21.15.137:0/494896618 conn(0x7f8de80f1c70 msgr2=0x7f8de80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:50:21.511 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:21.507+0000 7f8ded94a700 1 -- 172.21.15.137:0/494896618 shutdown_connections 2023-11-15T01:50:21.511 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:21.507+0000 7f8ded94a700 1 -- 172.21.15.137:0/494896618 wait complete. 2023-11-15T01:50:21.512 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:21.508+0000 7f8ded94a700 1 Processor -- start 2023-11-15T01:50:21.512 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:21.508+0000 7f8ded94a700 1 -- start start 2023-11-15T01:50:21.512 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:21.508+0000 7f8ded94a700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8de80f6510 0x7f8de8106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:50:21.512 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:21.508+0000 7f8ded94a700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8de8106c20 con 0x7f8de80f6510 2023-11-15T01:50:21.513 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:21.508+0000 7f8de6ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8de80f6510 0x7f8de8106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:50:21.513 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:21.508+0000 7f8de6ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8de80f6510 0x7f8de8106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:58266/0 (socket says 172.21.15.137:58266) 2023-11-15T01:50:21.513 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:21.508+0000 7f8de6ffd700 1 -- 172.21.15.137:0/4264712774 learned_addr learned my addr 172.21.15.137:0/4264712774 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:50:21.514 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:21.509+0000 7f8de6ffd700 1 -- 172.21.15.137:0/4264712774 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ddc005040 con 0x7f8de80f6510 2023-11-15T01:50:21.515 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:21.509+0000 7f8de6ffd700 1 --2- 172.21.15.137:0/4264712774 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8de80f6510 0x7f8de8106730 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f8ddc013510 tx=0x7f8ddc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:50:21.515 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:21.509+0000 7f8dcf7fe700 1 -- 172.21.15.137:0/4264712774 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ddc009320 con 0x7f8de80f6510 2023-11-15T01:50:21.515 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:21.509+0000 7f8dcf7fe700 1 -- 172.21.15.137:0/4264712774 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8ddc013f00 con 0x7f8de80f6510 2023-11-15T01:50:21.515 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:21.509+0000 7f8ded94a700 1 -- 172.21.15.137:0/4264712774 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8de8106e20 con 0x7f8de80f6510 2023-11-15T01:50:21.516 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:21.509+0000 7f8dcf7fe700 1 -- 172.21.15.137:0/4264712774 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ddc014d90 con 0x7f8de80f6510 2023-11-15T01:50:21.516 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:21.509+0000 7f8ded94a700 1 -- 172.21.15.137:0/4264712774 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8de81072b0 con 0x7f8de80f6510 2023-11-15T01:50:21.516 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:21.510+0000 7f8dcf7fe700 1 -- 172.21.15.137:0/4264712774 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8ddc014420 con 0x7f8de80f6510 2023-11-15T01:50:21.516 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:21.510+0000 7f8ded94a700 1 -- 172.21.15.137:0/4264712774 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8de8045ad0 con 0x7f8de80f6510 2023-11-15T01:50:21.517 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:21.510+0000 7f8dcf7fe700 1 --2- 172.21.15.137:0/4264712774 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f8dd002ff60 0x7f8dd0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:50:21.517 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:21.510+0000 7f8dcf7fe700 1 -- 172.21.15.137:0/4264712774 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8ddc041000 con 0x7f8de80f6510 2023-11-15T01:50:21.517 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:21.510+0000 7f8de67fc700 1 --2- 172.21.15.137:0/4264712774 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f8dd002ff60 0x7f8dd0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:50:21.517 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:21.511+0000 7f8de67fc700 1 --2- 172.21.15.137:0/4264712774 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f8dd002ff60 0x7f8dd0032420 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f8dd80096a0 tx=0x7f8dd8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:50:21.518 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:21.515+0000 7f8dcf7fe700 1 -- 172.21.15.137:0/4264712774 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8ddc0095f0 con 0x7f8de80f6510 2023-11-15T01:50:21.693 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:21.691+0000 7f8ded94a700 1 -- 172.21.15.137:0/4264712774 --> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi055:172.21.15.55=smithi055;smithi137:172.21.15.137=smithi137", "target": ["mon-mgr", ""]}) v1 -- 0x7f8de80f2f20 con 0x7f8dd002ff60 2023-11-15T01:50:21.704 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:21.703+0000 7f8dcf7fe700 1 -- 172.21.15.137:0/4264712774 <== mgr.14164 v2:172.21.15.55:6800/157802929 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f8de80f2f20 con 0x7f8dd002ff60 2023-11-15T01:50:21.704 INFO:teuthology.orchestra.run.smithi137.stdout:Scheduled mon update... 2023-11-15T01:50:21.706 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:21.703+0000 7f8ded94a700 1 -- 172.21.15.137:0/4264712774 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f8dd002ff60 msgr2=0x7f8dd0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:50:21.706 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:21.703+0000 7f8ded94a700 1 --2- 172.21.15.137:0/4264712774 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f8dd002ff60 0x7f8dd0032420 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f8dd80096a0 tx=0x7f8dd8006b40).stop 2023-11-15T01:50:21.706 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:21.703+0000 7f8ded94a700 1 -- 172.21.15.137:0/4264712774 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8de80f6510 msgr2=0x7f8de8106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:50:21.707 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:21.703+0000 7f8ded94a700 1 --2- 172.21.15.137:0/4264712774 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8de80f6510 0x7f8de8106730 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f8ddc013510 tx=0x7f8ddc00b040).stop 2023-11-15T01:50:21.707 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:21.704+0000 7f8ded94a700 1 -- 172.21.15.137:0/4264712774 shutdown_connections 2023-11-15T01:50:21.707 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:21.704+0000 7f8ded94a700 1 --2- 172.21.15.137:0/4264712774 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f8dd002ff60 0x7f8dd0032420 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:50:21.707 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:21.704+0000 7f8ded94a700 1 --2- 172.21.15.137:0/4264712774 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8de80f6510 0x7f8de8106730 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:50:21.707 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:21.704+0000 7f8ded94a700 1 -- 172.21.15.137:0/4264712774 >> 172.21.15.137:0/4264712774 conn(0x7f8de80f1c70 msgr2=0x7f8de80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:50:21.708 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:21.704+0000 7f8ded94a700 1 -- 172.21.15.137:0/4264712774 shutdown_connections 2023-11-15T01:50:21.708 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:21.704+0000 7f8ded94a700 1 -- 172.21.15.137:0/4264712774 wait complete. 2023-11-15T01:50:22.086 DEBUG:teuthology.orchestra.run.smithi137:mon.smithi137> sudo journalctl -f -n 0 -u ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7@mon.smithi137.service 2023-11-15T01:50:22.089 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:50:22.089 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:50:22.117 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:-- Logs begin at Wed 2023-11-15 01:06:49 UTC. -- 2023-11-15T01:50:23.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:50:22 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:50:21.694177+0000 mgr.smithi055.szwngw (mgr.14164) 19 : audit [DBG] from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi055:172.21.15.55=smithi055;smithi137:172.21.15.137=smithi137", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T01:50:23.079 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:50:22 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cephadm 2023-11-15T01:50:21.695686+0000 mgr.smithi055.szwngw (mgr.14164) 20 : cephadm [INF] Saving service mon spec with placement smithi055:172.21.15.55=smithi055;smithi137:172.21.15.137=smithi137;count:2 2023-11-15T01:50:23.079 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:50:22 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:50:21.703554+0000 mon.smithi055 (mon.0) 121 : audit [INF] from='mgr.14164 172.21.15.55:0/59463371' entity='mgr.smithi055.szwngw' 2023-11-15T01:50:23.226 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:23.222+0000 7fb6888ed700 1 -- 172.21.15.137:0/3158698328 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb6840f5410 msgr2=0x7fb6840f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:50:23.226 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:23.222+0000 7fb6888ed700 1 --2- 172.21.15.137:0/3158698328 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb6840f5410 0x7fb6840f5830 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7fb674009fb0 tx=0x7fb674009b30).stop 2023-11-15T01:50:23.226 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:23.222+0000 7fb6888ed700 1 -- 172.21.15.137:0/3158698328 shutdown_connections 2023-11-15T01:50:23.226 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:23.222+0000 7fb6888ed700 1 --2- 172.21.15.137:0/3158698328 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb6840f5410 0x7fb6840f5830 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:50:23.227 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:23.222+0000 7fb6888ed700 1 -- 172.21.15.137:0/3158698328 >> 172.21.15.137:0/3158698328 conn(0x7fb6840f0b70 msgr2=0x7fb6840f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:50:23.227 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:23.222+0000 7fb6888ed700 1 -- 172.21.15.137:0/3158698328 shutdown_connections 2023-11-15T01:50:23.227 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:23.222+0000 7fb6888ed700 1 -- 172.21.15.137:0/3158698328 wait complete. 2023-11-15T01:50:23.227 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:23.223+0000 7fb6888ed700 1 Processor -- start 2023-11-15T01:50:23.227 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:23.223+0000 7fb6888ed700 1 -- start start 2023-11-15T01:50:23.228 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:23.223+0000 7fb6888ed700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb6840f5410 0x7fb6841093d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:50:23.228 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:23.223+0000 7fb6888ed700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb684060240 con 0x7fb6840f5410 2023-11-15T01:50:23.228 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:23.223+0000 7fb68259c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb6840f5410 0x7fb6841093d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:50:23.228 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:23.223+0000 7fb68259c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb6840f5410 0x7fb6841093d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:58270/0 (socket says 172.21.15.137:58270) 2023-11-15T01:50:23.228 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:23.223+0000 7fb68259c700 1 -- 172.21.15.137:0/3914715501 learned_addr learned my addr 172.21.15.137:0/3914715501 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:50:23.229 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:23.223+0000 7fb68259c700 1 -- 172.21.15.137:0/3914715501 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb674005040 con 0x7fb6840f5410 2023-11-15T01:50:23.229 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:23.224+0000 7fb68259c700 1 --2- 172.21.15.137:0/3914715501 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb6840f5410 0x7fb6841093d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7fb674013510 tx=0x7fb67400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:50:23.229 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:23.224+0000 7fb67affd700 1 -- 172.21.15.137:0/3914715501 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb674009320 con 0x7fb6840f5410 2023-11-15T01:50:23.229 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:23.224+0000 7fb6888ed700 1 -- 172.21.15.137:0/3914715501 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb684060440 con 0x7fb6840f5410 2023-11-15T01:50:23.229 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:23.224+0000 7fb67affd700 1 -- 172.21.15.137:0/3914715501 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb674013f00 con 0x7fb6840f5410 2023-11-15T01:50:23.230 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:23.224+0000 7fb67affd700 1 -- 172.21.15.137:0/3914715501 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb674014d90 con 0x7fb6840f5410 2023-11-15T01:50:23.230 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:23.224+0000 7fb6888ed700 1 -- 172.21.15.137:0/3914715501 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb684109a80 con 0x7fb6840f5410 2023-11-15T01:50:23.231 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:23.225+0000 7fb67affd700 1 -- 172.21.15.137:0/3914715501 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb674014420 con 0x7fb6840f5410 2023-11-15T01:50:23.231 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:23.225+0000 7fb6888ed700 1 -- 172.21.15.137:0/3914715501 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb684045ad0 con 0x7fb6840f5410 2023-11-15T01:50:23.231 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:23.225+0000 7fb67affd700 1 --2- 172.21.15.137:0/3914715501 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fb67002ff60 0x7fb670032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:50:23.231 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:23.225+0000 7fb67affd700 1 -- 172.21.15.137:0/3914715501 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb674041050 con 0x7fb6840f5410 2023-11-15T01:50:23.232 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:23.225+0000 7fb681d9b700 1 --2- 172.21.15.137:0/3914715501 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fb67002ff60 0x7fb670032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:50:23.232 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:23.225+0000 7fb681d9b700 1 --2- 172.21.15.137:0/3914715501 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fb67002ff60 0x7fb670032420 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fb66c0096a0 tx=0x7fb66c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:50:23.232 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:23.230+0000 7fb67affd700 1 -- 172.21.15.137:0/3914715501 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb6740095f0 con 0x7fb6840f5410 2023-11-15T01:50:23.435 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:23.433+0000 7fb6888ed700 1 -- 172.21.15.137:0/3914715501 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb68410a290 con 0x7fb6840f5410 2023-11-15T01:50:23.436 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:23.434+0000 7fb67affd700 1 -- 172.21.15.137:0/3914715501 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb67401db70 con 0x7fb6840f5410 2023-11-15T01:50:23.436 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:50:23.436 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:50:23.437 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:23.436+0000 7fb6888ed700 1 -- 172.21.15.137:0/3914715501 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fb67002ff60 msgr2=0x7fb670032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:50:23.438 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:23.436+0000 7fb6888ed700 1 --2- 172.21.15.137:0/3914715501 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fb67002ff60 0x7fb670032420 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fb66c0096a0 tx=0x7fb66c006b40).stop 2023-11-15T01:50:23.438 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:23.436+0000 7fb6888ed700 1 -- 172.21.15.137:0/3914715501 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb6840f5410 msgr2=0x7fb6841093d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:50:23.438 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:23.436+0000 7fb6888ed700 1 --2- 172.21.15.137:0/3914715501 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb6840f5410 0x7fb6841093d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7fb674013510 tx=0x7fb67400b040).stop 2023-11-15T01:50:23.438 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:23.436+0000 7fb6888ed700 1 -- 172.21.15.137:0/3914715501 shutdown_connections 2023-11-15T01:50:23.438 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:23.436+0000 7fb6888ed700 1 --2- 172.21.15.137:0/3914715501 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fb67002ff60 0x7fb670032420 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:50:23.439 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:23.436+0000 7fb6888ed700 1 --2- 172.21.15.137:0/3914715501 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb6840f5410 0x7fb6841093d0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:50:23.439 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:23.436+0000 7fb6888ed700 1 -- 172.21.15.137:0/3914715501 >> 172.21.15.137:0/3914715501 conn(0x7fb6840f0b70 msgr2=0x7fb6840f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:50:23.439 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:23.436+0000 7fb6888ed700 1 -- 172.21.15.137:0/3914715501 shutdown_connections 2023-11-15T01:50:23.439 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:23.436+0000 7fb6888ed700 1 -- 172.21.15.137:0/3914715501 wait complete. 2023-11-15T01:50:23.440 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:50:24.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:50:23 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:50:22.695008+0000 mgr.smithi055.szwngw (mgr.14164) 21 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:50:24.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:50:23 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:50:23.436019+0000 mon.smithi055 (mon.0) 122 : audit [DBG] from='client.? 172.21.15.137:0/3914715501' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:50:24.819 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:50:24.820 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:50:25.940 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:25.938+0000 7fe8886bd700 1 -- 172.21.15.137:0/2659071567 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe8800f5410 msgr2=0x7fe8800f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:50:25.940 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:25.938+0000 7fe8886bd700 1 --2- 172.21.15.137:0/2659071567 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe8800f5410 0x7fe8800f5830 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7fe870009fb0 tx=0x7fe870009b30).stop 2023-11-15T01:50:25.941 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:25.938+0000 7fe8886bd700 1 -- 172.21.15.137:0/2659071567 shutdown_connections 2023-11-15T01:50:25.941 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:25.938+0000 7fe8886bd700 1 --2- 172.21.15.137:0/2659071567 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe8800f5410 0x7fe8800f5830 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:50:25.941 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:25.938+0000 7fe8886bd700 1 -- 172.21.15.137:0/2659071567 >> 172.21.15.137:0/2659071567 conn(0x7fe8800f0b70 msgr2=0x7fe8800f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:50:25.941 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:25.938+0000 7fe8886bd700 1 -- 172.21.15.137:0/2659071567 shutdown_connections 2023-11-15T01:50:25.941 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:25.938+0000 7fe8886bd700 1 -- 172.21.15.137:0/2659071567 wait complete. 2023-11-15T01:50:25.942 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:25.939+0000 7fe8886bd700 1 Processor -- start 2023-11-15T01:50:25.942 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:25.939+0000 7fe8886bd700 1 -- start start 2023-11-15T01:50:25.942 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:25.939+0000 7fe8886bd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe8800f5410 0x7fe8801093d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:50:25.942 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:25.939+0000 7fe8886bd700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe880060240 con 0x7fe8800f5410 2023-11-15T01:50:25.942 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:25.939+0000 7fe886459700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe8800f5410 0x7fe8801093d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:50:25.943 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:25.939+0000 7fe886459700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe8800f5410 0x7fe8801093d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:58278/0 (socket says 172.21.15.137:58278) 2023-11-15T01:50:25.943 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:25.939+0000 7fe886459700 1 -- 172.21.15.137:0/3985452446 learned_addr learned my addr 172.21.15.137:0/3985452446 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:50:25.943 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:25.940+0000 7fe886459700 1 -- 172.21.15.137:0/3985452446 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe870005040 con 0x7fe8800f5410 2023-11-15T01:50:25.944 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:25.940+0000 7fe886459700 1 --2- 172.21.15.137:0/3985452446 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe8800f5410 0x7fe8801093d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7fe870013510 tx=0x7fe87000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:50:25.944 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:25.940+0000 7fe876ffd700 1 -- 172.21.15.137:0/3985452446 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe870009320 con 0x7fe8800f5410 2023-11-15T01:50:25.945 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:25.940+0000 7fe876ffd700 1 -- 172.21.15.137:0/3985452446 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe870013f00 con 0x7fe8800f5410 2023-11-15T01:50:25.945 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:25.940+0000 7fe8886bd700 1 -- 172.21.15.137:0/3985452446 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe880060440 con 0x7fe8800f5410 2023-11-15T01:50:25.945 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:25.940+0000 7fe876ffd700 1 -- 172.21.15.137:0/3985452446 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe870014d90 con 0x7fe8800f5410 2023-11-15T01:50:25.945 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:25.940+0000 7fe8886bd700 1 -- 172.21.15.137:0/3985452446 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe880109a80 con 0x7fe8800f5410 2023-11-15T01:50:25.945 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:25.941+0000 7fe876ffd700 1 -- 172.21.15.137:0/3985452446 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe8700095f0 con 0x7fe8800f5410 2023-11-15T01:50:25.946 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:25.941+0000 7fe8886bd700 1 -- 172.21.15.137:0/3985452446 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe880045ad0 con 0x7fe8800f5410 2023-11-15T01:50:25.946 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:25.941+0000 7fe876ffd700 1 --2- 172.21.15.137:0/3985452446 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fe86c02ff60 0x7fe86c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:50:25.946 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:25.941+0000 7fe876ffd700 1 -- 172.21.15.137:0/3985452446 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe870041080 con 0x7fe8800f5410 2023-11-15T01:50:25.946 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:25.941+0000 7fe885c58700 1 --2- 172.21.15.137:0/3985452446 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fe86c02ff60 0x7fe86c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:50:25.946 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:25.942+0000 7fe885c58700 1 --2- 172.21.15.137:0/3985452446 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fe86c02ff60 0x7fe86c032420 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fe87c0096a0 tx=0x7fe87c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:50:25.947 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:25.946+0000 7fe876ffd700 1 -- 172.21.15.137:0/3985452446 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe870014420 con 0x7fe8800f5410 2023-11-15T01:50:26.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:50:25 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:50:24.695376+0000 mgr.smithi055.szwngw (mgr.14164) 22 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:50:26.153 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:26.152+0000 7fe8886bd700 1 -- 172.21.15.137:0/3985452446 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe88010a2a0 con 0x7fe8800f5410 2023-11-15T01:50:26.154 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:26.153+0000 7fe876ffd700 1 -- 172.21.15.137:0/3985452446 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe870010230 con 0x7fe8800f5410 2023-11-15T01:50:26.154 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:50:26.154 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:50:26.155 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:26.155+0000 7fe8886bd700 1 -- 172.21.15.137:0/3985452446 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fe86c02ff60 msgr2=0x7fe86c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:50:26.156 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:26.155+0000 7fe8886bd700 1 --2- 172.21.15.137:0/3985452446 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fe86c02ff60 0x7fe86c032420 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fe87c0096a0 tx=0x7fe87c006b40).stop 2023-11-15T01:50:26.156 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:26.155+0000 7fe8886bd700 1 -- 172.21.15.137:0/3985452446 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe8800f5410 msgr2=0x7fe8801093d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:50:26.156 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:26.155+0000 7fe8886bd700 1 --2- 172.21.15.137:0/3985452446 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe8800f5410 0x7fe8801093d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7fe870013510 tx=0x7fe87000b040).stop 2023-11-15T01:50:26.156 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:26.155+0000 7fe8886bd700 1 -- 172.21.15.137:0/3985452446 shutdown_connections 2023-11-15T01:50:26.156 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:26.155+0000 7fe8886bd700 1 --2- 172.21.15.137:0/3985452446 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fe86c02ff60 0x7fe86c032420 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:50:26.157 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:26.155+0000 7fe8886bd700 1 --2- 172.21.15.137:0/3985452446 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe8800f5410 0x7fe8801093d0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:50:26.157 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:26.155+0000 7fe8886bd700 1 -- 172.21.15.137:0/3985452446 >> 172.21.15.137:0/3985452446 conn(0x7fe8800f0b70 msgr2=0x7fe8800f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:50:26.157 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:26.155+0000 7fe8886bd700 1 -- 172.21.15.137:0/3985452446 shutdown_connections 2023-11-15T01:50:26.157 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:26.155+0000 7fe8886bd700 1 -- 172.21.15.137:0/3985452446 wait complete. 2023-11-15T01:50:26.158 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:50:27.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:50:26 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:50:26.154081+0000 mon.smithi055 (mon.0) 123 : audit [DBG] from='client.? 172.21.15.137:0/3985452446' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:50:27.538 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:50:27.539 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:50:28.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:50:27 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:50:26.695742+0000 mgr.smithi055.szwngw (mgr.14164) 23 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:50:28.718 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:28.716+0000 7f858226b700 1 -- 172.21.15.137:0/195297431 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f857c0f6510 msgr2=0x7f857c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:50:28.719 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:28.716+0000 7f858226b700 1 --2- 172.21.15.137:0/195297431 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f857c0f6510 0x7f857c0f6930 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f8570009fb0 tx=0x7f8570009b30).stop 2023-11-15T01:50:28.719 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:28.716+0000 7f858226b700 1 -- 172.21.15.137:0/195297431 shutdown_connections 2023-11-15T01:50:28.719 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:28.716+0000 7f858226b700 1 --2- 172.21.15.137:0/195297431 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f857c0f6510 0x7f857c0f6930 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:50:28.719 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:28.716+0000 7f858226b700 1 -- 172.21.15.137:0/195297431 >> 172.21.15.137:0/195297431 conn(0x7f857c0f1c70 msgr2=0x7f857c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:50:28.720 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:28.716+0000 7f858226b700 1 -- 172.21.15.137:0/195297431 shutdown_connections 2023-11-15T01:50:28.720 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:28.716+0000 7f858226b700 1 -- 172.21.15.137:0/195297431 wait complete. 2023-11-15T01:50:28.720 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:28.716+0000 7f858226b700 1 Processor -- start 2023-11-15T01:50:28.720 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:28.717+0000 7f858226b700 1 -- start start 2023-11-15T01:50:28.720 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:28.717+0000 7f858226b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f857c0f6510 0x7f857c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:50:28.721 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:28.717+0000 7f858226b700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f857c106c20 con 0x7f857c0f6510 2023-11-15T01:50:28.721 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:28.717+0000 7f857bfff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f857c0f6510 0x7f857c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:50:28.721 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:28.717+0000 7f857bfff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f857c0f6510 0x7f857c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:40906/0 (socket says 172.21.15.137:40906) 2023-11-15T01:50:28.721 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:28.717+0000 7f857bfff700 1 -- 172.21.15.137:0/3738737552 learned_addr learned my addr 172.21.15.137:0/3738737552 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:50:28.721 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:28.717+0000 7f857bfff700 1 -- 172.21.15.137:0/3738737552 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8570005040 con 0x7f857c0f6510 2023-11-15T01:50:28.722 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:28.717+0000 7f857bfff700 1 --2- 172.21.15.137:0/3738737552 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f857c0f6510 0x7f857c106730 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f8570013510 tx=0x7f857000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:50:28.722 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:28.718+0000 7f8578ff9700 1 -- 172.21.15.137:0/3738737552 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8570009320 con 0x7f857c0f6510 2023-11-15T01:50:28.722 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:28.718+0000 7f858226b700 1 -- 172.21.15.137:0/3738737552 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f857c106e20 con 0x7f857c0f6510 2023-11-15T01:50:28.722 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:28.718+0000 7f8578ff9700 1 -- 172.21.15.137:0/3738737552 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8570013f00 con 0x7f857c0f6510 2023-11-15T01:50:28.722 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:28.718+0000 7f8578ff9700 1 -- 172.21.15.137:0/3738737552 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8570014d90 con 0x7f857c0f6510 2023-11-15T01:50:28.723 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:28.718+0000 7f858226b700 1 -- 172.21.15.137:0/3738737552 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f857c1072b0 con 0x7f857c0f6510 2023-11-15T01:50:28.724 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:28.719+0000 7f8578ff9700 1 -- 172.21.15.137:0/3738737552 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8570014420 con 0x7f857c0f6510 2023-11-15T01:50:28.724 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:28.719+0000 7f858226b700 1 -- 172.21.15.137:0/3738737552 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f857c045ad0 con 0x7f857c0f6510 2023-11-15T01:50:28.724 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:28.719+0000 7f8578ff9700 1 --2- 172.21.15.137:0/3738737552 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f856402ff60 0x7f8564032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:50:28.724 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:28.719+0000 7f8578ff9700 1 -- 172.21.15.137:0/3738737552 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8570041000 con 0x7f857c0f6510 2023-11-15T01:50:28.724 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:28.719+0000 7f857b7fe700 1 --2- 172.21.15.137:0/3738737552 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f856402ff60 0x7f8564032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:50:28.725 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:28.719+0000 7f857b7fe700 1 --2- 172.21.15.137:0/3738737552 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f856402ff60 0x7f8564032420 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f856c0096a0 tx=0x7f856c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:50:28.725 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:28.723+0000 7f8578ff9700 1 -- 172.21.15.137:0/3738737552 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f85700095f0 con 0x7f857c0f6510 2023-11-15T01:50:28.932 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:28.930+0000 7f858226b700 1 -- 172.21.15.137:0/3738737552 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f857c0fbf60 con 0x7f857c0f6510 2023-11-15T01:50:28.932 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:28.931+0000 7f8578ff9700 1 -- 172.21.15.137:0/3738737552 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f857001db70 con 0x7f857c0f6510 2023-11-15T01:50:28.933 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:50:28.933 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:50:28.934 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:28.932+0000 7f858226b700 1 -- 172.21.15.137:0/3738737552 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f856402ff60 msgr2=0x7f8564032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:50:28.935 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:28.932+0000 7f858226b700 1 --2- 172.21.15.137:0/3738737552 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f856402ff60 0x7f8564032420 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f856c0096a0 tx=0x7f856c006b40).stop 2023-11-15T01:50:28.935 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:28.932+0000 7f858226b700 1 -- 172.21.15.137:0/3738737552 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f857c0f6510 msgr2=0x7f857c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:50:28.935 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:28.932+0000 7f858226b700 1 --2- 172.21.15.137:0/3738737552 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f857c0f6510 0x7f857c106730 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f8570013510 tx=0x7f857000b040).stop 2023-11-15T01:50:28.936 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:28.932+0000 7f858226b700 1 -- 172.21.15.137:0/3738737552 shutdown_connections 2023-11-15T01:50:28.936 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:28.932+0000 7f858226b700 1 --2- 172.21.15.137:0/3738737552 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f856402ff60 0x7f8564032420 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:50:28.936 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:28.932+0000 7f858226b700 1 --2- 172.21.15.137:0/3738737552 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f857c0f6510 0x7f857c106730 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:50:28.936 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:28.932+0000 7f858226b700 1 -- 172.21.15.137:0/3738737552 >> 172.21.15.137:0/3738737552 conn(0x7f857c0f1c70 msgr2=0x7f857c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:50:28.937 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:28.932+0000 7f858226b700 1 -- 172.21.15.137:0/3738737552 shutdown_connections 2023-11-15T01:50:28.937 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:28.932+0000 7f858226b700 1 -- 172.21.15.137:0/3738737552 wait complete. 2023-11-15T01:50:28.937 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:50:30.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:50:29 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:50:28.696154+0000 mgr.smithi055.szwngw (mgr.14164) 24 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:50:30.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:50:29 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:50:28.931806+0000 mon.smithi055 (mon.0) 124 : audit [DBG] from='client.? 172.21.15.137:0/3738737552' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:50:30.309 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:50:30.310 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:50:31.504 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:31.503+0000 7fd34eaec700 1 -- 172.21.15.137:0/376729520 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd3480f5410 msgr2=0x7fd3480f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:50:31.505 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:31.503+0000 7fd34eaec700 1 --2- 172.21.15.137:0/376729520 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd3480f5410 0x7fd3480f5830 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7fd33c009fb0 tx=0x7fd33c009b30).stop 2023-11-15T01:50:31.505 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:31.503+0000 7fd34eaec700 1 -- 172.21.15.137:0/376729520 shutdown_connections 2023-11-15T01:50:31.505 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:31.503+0000 7fd34eaec700 1 --2- 172.21.15.137:0/376729520 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd3480f5410 0x7fd3480f5830 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:50:31.505 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:31.503+0000 7fd34eaec700 1 -- 172.21.15.137:0/376729520 >> 172.21.15.137:0/376729520 conn(0x7fd3480f0b70 msgr2=0x7fd3480f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:50:31.506 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:31.503+0000 7fd34eaec700 1 -- 172.21.15.137:0/376729520 shutdown_connections 2023-11-15T01:50:31.506 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:31.503+0000 7fd34eaec700 1 -- 172.21.15.137:0/376729520 wait complete. 2023-11-15T01:50:31.506 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:31.504+0000 7fd34eaec700 1 Processor -- start 2023-11-15T01:50:31.506 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:31.504+0000 7fd34eaec700 1 -- start start 2023-11-15T01:50:31.508 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:31.504+0000 7fd34eaec700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd3480f5410 0x7fd348106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:50:31.508 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:31.504+0000 7fd34eaec700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd348106c20 con 0x7fd3480f5410 2023-11-15T01:50:31.508 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:31.504+0000 7fd34c888700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd3480f5410 0x7fd348106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:50:31.509 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:31.504+0000 7fd34c888700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd3480f5410 0x7fd348106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:40924/0 (socket says 172.21.15.137:40924) 2023-11-15T01:50:31.509 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:31.504+0000 7fd34c888700 1 -- 172.21.15.137:0/1731619754 learned_addr learned my addr 172.21.15.137:0/1731619754 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:50:31.509 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:31.505+0000 7fd34c888700 1 -- 172.21.15.137:0/1731619754 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd33c005040 con 0x7fd3480f5410 2023-11-15T01:50:31.509 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:31.505+0000 7fd34c888700 1 --2- 172.21.15.137:0/1731619754 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd3480f5410 0x7fd348106730 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7fd33c013510 tx=0x7fd33c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:50:31.509 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:31.505+0000 7fd3457fa700 1 -- 172.21.15.137:0/1731619754 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd33c009320 con 0x7fd3480f5410 2023-11-15T01:50:31.510 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:31.505+0000 7fd34eaec700 1 -- 172.21.15.137:0/1731619754 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd348106e20 con 0x7fd3480f5410 2023-11-15T01:50:31.510 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:31.505+0000 7fd3457fa700 1 -- 172.21.15.137:0/1731619754 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd33c013f00 con 0x7fd3480f5410 2023-11-15T01:50:31.510 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:31.505+0000 7fd3457fa700 1 -- 172.21.15.137:0/1731619754 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd33c014d90 con 0x7fd3480f5410 2023-11-15T01:50:31.510 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:31.505+0000 7fd34eaec700 1 -- 172.21.15.137:0/1731619754 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3481072b0 con 0x7fd3480f5410 2023-11-15T01:50:31.511 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:31.506+0000 7fd3457fa700 1 -- 172.21.15.137:0/1731619754 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd33c0095f0 con 0x7fd3480f5410 2023-11-15T01:50:31.511 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:31.506+0000 7fd34eaec700 1 -- 172.21.15.137:0/1731619754 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd348045ad0 con 0x7fd3480f5410 2023-11-15T01:50:31.511 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:31.506+0000 7fd3457fa700 1 --2- 172.21.15.137:0/1731619754 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fd33802ff60 0x7fd338032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:50:31.511 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:31.506+0000 7fd3457fa700 1 -- 172.21.15.137:0/1731619754 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd33c041000 con 0x7fd3480f5410 2023-11-15T01:50:31.511 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:31.506+0000 7fd347fff700 1 --2- 172.21.15.137:0/1731619754 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fd33802ff60 0x7fd338032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:50:31.512 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:31.507+0000 7fd347fff700 1 --2- 172.21.15.137:0/1731619754 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fd33802ff60 0x7fd338032420 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fd3340096a0 tx=0x7fd334006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:50:31.512 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:31.511+0000 7fd3457fa700 1 -- 172.21.15.137:0/1731619754 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd33c01d7f0 con 0x7fd3480f5410 2023-11-15T01:50:31.717 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:31.716+0000 7fd34eaec700 1 -- 172.21.15.137:0/1731619754 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd3480fae60 con 0x7fd3480f5410 2023-11-15T01:50:31.718 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:31.717+0000 7fd3457fa700 1 -- 172.21.15.137:0/1731619754 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd33c010230 con 0x7fd3480f5410 2023-11-15T01:50:31.718 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:50:31.718 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:50:31.719 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:31.718+0000 7fd34eaec700 1 -- 172.21.15.137:0/1731619754 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fd33802ff60 msgr2=0x7fd338032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:50:31.720 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:31.718+0000 7fd34eaec700 1 --2- 172.21.15.137:0/1731619754 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fd33802ff60 0x7fd338032420 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fd3340096a0 tx=0x7fd334006b40).stop 2023-11-15T01:50:31.720 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:31.718+0000 7fd34eaec700 1 -- 172.21.15.137:0/1731619754 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd3480f5410 msgr2=0x7fd348106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:50:31.720 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:31.718+0000 7fd34eaec700 1 --2- 172.21.15.137:0/1731619754 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd3480f5410 0x7fd348106730 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7fd33c013510 tx=0x7fd33c00b040).stop 2023-11-15T01:50:31.720 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:31.719+0000 7fd34eaec700 1 -- 172.21.15.137:0/1731619754 shutdown_connections 2023-11-15T01:50:31.720 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:31.719+0000 7fd34eaec700 1 --2- 172.21.15.137:0/1731619754 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fd33802ff60 0x7fd338032420 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:50:31.721 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:31.719+0000 7fd34eaec700 1 --2- 172.21.15.137:0/1731619754 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd3480f5410 0x7fd348106730 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:50:31.721 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:31.719+0000 7fd34eaec700 1 -- 172.21.15.137:0/1731619754 >> 172.21.15.137:0/1731619754 conn(0x7fd3480f0b70 msgr2=0x7fd3480f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:50:31.721 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:31.719+0000 7fd34eaec700 1 -- 172.21.15.137:0/1731619754 shutdown_connections 2023-11-15T01:50:31.721 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:31.719+0000 7fd34eaec700 1 -- 172.21.15.137:0/1731619754 wait complete. 2023-11-15T01:50:31.722 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:50:31.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:50:31 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:50:30.696557+0000 mgr.smithi055.szwngw (mgr.14164) 25 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:50:32.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:50:32 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:50:31.718014+0000 mon.smithi055 (mon.0) 125 : audit [DBG] from='client.? 172.21.15.137:0/1731619754' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:50:33.347 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:50:33.348 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:50:33.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:50:33 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:50:32.696933+0000 mgr.smithi055.szwngw (mgr.14164) 26 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:50:34.526 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:34.524+0000 7f134f760700 1 -- 172.21.15.137:0/1599278876 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f13480f5410 msgr2=0x7f13480f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:50:34.527 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:34.524+0000 7f134f760700 1 --2- 172.21.15.137:0/1599278876 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f13480f5410 0x7f13480f5830 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f1338009fb0 tx=0x7f1338009b30).stop 2023-11-15T01:50:34.527 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:34.524+0000 7f134f760700 1 -- 172.21.15.137:0/1599278876 shutdown_connections 2023-11-15T01:50:34.527 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:34.524+0000 7f134f760700 1 --2- 172.21.15.137:0/1599278876 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f13480f5410 0x7f13480f5830 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:50:34.527 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:34.524+0000 7f134f760700 1 -- 172.21.15.137:0/1599278876 >> 172.21.15.137:0/1599278876 conn(0x7f13480f0b70 msgr2=0x7f13480f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:50:34.527 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:34.524+0000 7f134f760700 1 -- 172.21.15.137:0/1599278876 shutdown_connections 2023-11-15T01:50:34.528 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:34.524+0000 7f134f760700 1 -- 172.21.15.137:0/1599278876 wait complete. 2023-11-15T01:50:34.528 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:34.525+0000 7f134f760700 1 Processor -- start 2023-11-15T01:50:34.528 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:34.525+0000 7f134f760700 1 -- start start 2023-11-15T01:50:34.528 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:34.525+0000 7f134f760700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f13480f5410 0x7f1348109390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:50:34.528 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:34.525+0000 7f134f760700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1348060280 con 0x7f13480f5410 2023-11-15T01:50:34.529 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:34.525+0000 7f134d4fc700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f13480f5410 0x7f1348109390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:50:34.529 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:34.525+0000 7f134d4fc700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f13480f5410 0x7f1348109390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:40942/0 (socket says 172.21.15.137:40942) 2023-11-15T01:50:34.529 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:34.525+0000 7f134d4fc700 1 -- 172.21.15.137:0/4012530145 learned_addr learned my addr 172.21.15.137:0/4012530145 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:50:34.529 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:34.526+0000 7f134d4fc700 1 -- 172.21.15.137:0/4012530145 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1338005040 con 0x7f13480f5410 2023-11-15T01:50:34.529 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:34.526+0000 7f134d4fc700 1 --2- 172.21.15.137:0/4012530145 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f13480f5410 0x7f1348109390 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f1338014000 tx=0x7f133800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:50:34.531 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:34.526+0000 7f133dffb700 1 -- 172.21.15.137:0/4012530145 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f13380095f0 con 0x7f13480f5410 2023-11-15T01:50:34.531 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:34.526+0000 7f134f760700 1 -- 172.21.15.137:0/4012530145 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1348060480 con 0x7f13480f5410 2023-11-15T01:50:34.531 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:34.526+0000 7f133dffb700 1 -- 172.21.15.137:0/4012530145 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1338007e70 con 0x7f13480f5410 2023-11-15T01:50:34.532 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:34.526+0000 7f133dffb700 1 -- 172.21.15.137:0/4012530145 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1338014ea0 con 0x7f13480f5410 2023-11-15T01:50:34.532 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:34.526+0000 7f134f760700 1 -- 172.21.15.137:0/4012530145 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1348109ac0 con 0x7f13480f5410 2023-11-15T01:50:34.532 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:34.527+0000 7f133dffb700 1 -- 172.21.15.137:0/4012530145 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1338025070 con 0x7f13480f5410 2023-11-15T01:50:34.532 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:34.527+0000 7f134f760700 1 -- 172.21.15.137:0/4012530145 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1348045ad0 con 0x7f13480f5410 2023-11-15T01:50:34.532 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:34.527+0000 7f133dffb700 1 --2- 172.21.15.137:0/4012530145 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f133402ff60 0x7f1334032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:50:34.533 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:34.527+0000 7f133dffb700 1 -- 172.21.15.137:0/4012530145 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1338040970 con 0x7f13480f5410 2023-11-15T01:50:34.533 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:34.527+0000 7f134ccfb700 1 --2- 172.21.15.137:0/4012530145 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f133402ff60 0x7f1334032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:50:34.533 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:34.528+0000 7f134ccfb700 1 --2- 172.21.15.137:0/4012530145 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f133402ff60 0x7f1334032420 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f13440096a0 tx=0x7f1344006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:50:34.533 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:34.532+0000 7f133dffb700 1 -- 172.21.15.137:0/4012530145 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f13380079c0 con 0x7f13480f5410 2023-11-15T01:50:34.736 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:34.736+0000 7f134f760700 1 -- 172.21.15.137:0/4012530145 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f13480fae60 con 0x7f13480f5410 2023-11-15T01:50:34.737 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:34.736+0000 7f133dffb700 1 -- 172.21.15.137:0/4012530145 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f133800f300 con 0x7f13480f5410 2023-11-15T01:50:34.737 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:50:34.738 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:50:34.739 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:34.738+0000 7f134f760700 1 -- 172.21.15.137:0/4012530145 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f133402ff60 msgr2=0x7f1334032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:50:34.739 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:34.738+0000 7f134f760700 1 --2- 172.21.15.137:0/4012530145 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f133402ff60 0x7f1334032420 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f13440096a0 tx=0x7f1344006b40).stop 2023-11-15T01:50:34.739 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:34.738+0000 7f134f760700 1 -- 172.21.15.137:0/4012530145 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f13480f5410 msgr2=0x7f1348109390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:50:34.739 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:34.738+0000 7f134f760700 1 --2- 172.21.15.137:0/4012530145 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f13480f5410 0x7f1348109390 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f1338014000 tx=0x7f133800b040).stop 2023-11-15T01:50:34.740 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:34.738+0000 7f134f760700 1 -- 172.21.15.137:0/4012530145 shutdown_connections 2023-11-15T01:50:34.740 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:34.738+0000 7f134f760700 1 --2- 172.21.15.137:0/4012530145 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f133402ff60 0x7f1334032420 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:50:34.740 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:34.738+0000 7f134f760700 1 --2- 172.21.15.137:0/4012530145 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f13480f5410 0x7f1348109390 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:50:34.740 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:34.738+0000 7f134f760700 1 -- 172.21.15.137:0/4012530145 >> 172.21.15.137:0/4012530145 conn(0x7f13480f0b70 msgr2=0x7f13480f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:50:34.740 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:34.738+0000 7f134f760700 1 -- 172.21.15.137:0/4012530145 shutdown_connections 2023-11-15T01:50:34.741 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:34.738+0000 7f134f760700 1 -- 172.21.15.137:0/4012530145 wait complete. 2023-11-15T01:50:34.741 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:50:35.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:50:34 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:50:34.737477+0000 mon.smithi055 (mon.0) 126 : audit [DBG] from='client.? 172.21.15.137:0/4012530145' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:50:36.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:50:35 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:50:34.697254+0000 mgr.smithi055.szwngw (mgr.14164) 27 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:50:36.128 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:50:36.129 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:50:37.459 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:37.278+0000 7f116eea8700 1 -- 172.21.15.137:0/2861128216 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f11680f5410 msgr2=0x7f11680f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:50:37.459 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:37.278+0000 7f116eea8700 1 --2- 172.21.15.137:0/2861128216 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f11680f5410 0x7f11680f5830 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f115c009fb0 tx=0x7f115c009b30).stop 2023-11-15T01:50:37.460 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:37.278+0000 7f116eea8700 1 -- 172.21.15.137:0/2861128216 shutdown_connections 2023-11-15T01:50:37.460 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:37.278+0000 7f116eea8700 1 --2- 172.21.15.137:0/2861128216 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f11680f5410 0x7f11680f5830 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:50:37.460 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:37.278+0000 7f116eea8700 1 -- 172.21.15.137:0/2861128216 >> 172.21.15.137:0/2861128216 conn(0x7f11680f0b70 msgr2=0x7f11680f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:50:37.460 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:37.278+0000 7f116eea8700 1 -- 172.21.15.137:0/2861128216 shutdown_connections 2023-11-15T01:50:37.460 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:37.278+0000 7f116eea8700 1 -- 172.21.15.137:0/2861128216 wait complete. 2023-11-15T01:50:37.461 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:37.279+0000 7f116eea8700 1 Processor -- start 2023-11-15T01:50:37.461 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:37.279+0000 7f116eea8700 1 -- start start 2023-11-15T01:50:37.461 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:37.279+0000 7f116eea8700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f11680f5410 0x7f1168109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:50:37.461 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:37.279+0000 7f116eea8700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1168060240 con 0x7f11680f5410 2023-11-15T01:50:37.461 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:37.279+0000 7f116cc44700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f11680f5410 0x7f1168109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:50:37.462 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:37.279+0000 7f116cc44700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f11680f5410 0x7f1168109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:40954/0 (socket says 172.21.15.137:40954) 2023-11-15T01:50:37.462 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:37.279+0000 7f116cc44700 1 -- 172.21.15.137:0/483537427 learned_addr learned my addr 172.21.15.137:0/483537427 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:50:37.462 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:37.280+0000 7f116cc44700 1 -- 172.21.15.137:0/483537427 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f115c005040 con 0x7f11680f5410 2023-11-15T01:50:37.467 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:37.280+0000 7f116cc44700 1 --2- 172.21.15.137:0/483537427 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f11680f5410 0x7f1168109430 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f115c013510 tx=0x7f115c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:50:37.467 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:37.280+0000 7f11657fa700 1 -- 172.21.15.137:0/483537427 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f115c009320 con 0x7f11680f5410 2023-11-15T01:50:37.467 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:37.280+0000 7f116eea8700 1 -- 172.21.15.137:0/483537427 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1168060440 con 0x7f11680f5410 2023-11-15T01:50:37.468 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:37.280+0000 7f11657fa700 1 -- 172.21.15.137:0/483537427 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f115c013f00 con 0x7f11680f5410 2023-11-15T01:50:37.468 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:37.280+0000 7f11657fa700 1 -- 172.21.15.137:0/483537427 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f115c014d90 con 0x7f11680f5410 2023-11-15T01:50:37.468 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:37.280+0000 7f116eea8700 1 -- 172.21.15.137:0/483537427 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1168109ae0 con 0x7f11680f5410 2023-11-15T01:50:37.468 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:37.281+0000 7f11657fa700 1 -- 172.21.15.137:0/483537427 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f115c0095f0 con 0x7f11680f5410 2023-11-15T01:50:37.468 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:37.281+0000 7f116eea8700 1 -- 172.21.15.137:0/483537427 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1168045ad0 con 0x7f11680f5410 2023-11-15T01:50:37.469 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:37.281+0000 7f11657fa700 1 --2- 172.21.15.137:0/483537427 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f115802ff60 0x7f1158032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:50:37.469 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:37.281+0000 7f11657fa700 1 -- 172.21.15.137:0/483537427 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f115c041000 con 0x7f11680f5410 2023-11-15T01:50:37.469 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:37.281+0000 7f1167fff700 1 --2- 172.21.15.137:0/483537427 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f115802ff60 0x7f1158032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:50:37.469 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:37.282+0000 7f1167fff700 1 --2- 172.21.15.137:0/483537427 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f115802ff60 0x7f1158032420 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f11540096a0 tx=0x7f1154006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:50:37.469 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:37.286+0000 7f11657fa700 1 -- 172.21.15.137:0/483537427 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f115c01d7f0 con 0x7f11680f5410 2023-11-15T01:50:37.497 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:37.496+0000 7f116eea8700 1 -- 172.21.15.137:0/483537427 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f11680fae60 con 0x7f11680f5410 2023-11-15T01:50:37.498 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:37.497+0000 7f11657fa700 1 -- 172.21.15.137:0/483537427 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f115c010230 con 0x7f11680f5410 2023-11-15T01:50:37.498 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:50:37.498 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:50:37.500 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:37.499+0000 7f116eea8700 1 -- 172.21.15.137:0/483537427 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f115802ff60 msgr2=0x7f1158032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:50:37.500 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:37.499+0000 7f116eea8700 1 --2- 172.21.15.137:0/483537427 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f115802ff60 0x7f1158032420 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f11540096a0 tx=0x7f1154006b40).stop 2023-11-15T01:50:37.500 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:37.499+0000 7f116eea8700 1 -- 172.21.15.137:0/483537427 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f11680f5410 msgr2=0x7f1168109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:50:37.500 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:37.499+0000 7f116eea8700 1 --2- 172.21.15.137:0/483537427 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f11680f5410 0x7f1168109430 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f115c013510 tx=0x7f115c00b040).stop 2023-11-15T01:50:37.500 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:37.499+0000 7f116eea8700 1 -- 172.21.15.137:0/483537427 shutdown_connections 2023-11-15T01:50:37.501 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:37.499+0000 7f116eea8700 1 --2- 172.21.15.137:0/483537427 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f115802ff60 0x7f1158032420 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:50:37.501 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:37.499+0000 7f116eea8700 1 --2- 172.21.15.137:0/483537427 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f11680f5410 0x7f1168109430 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:50:37.501 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:37.499+0000 7f116eea8700 1 -- 172.21.15.137:0/483537427 >> 172.21.15.137:0/483537427 conn(0x7f11680f0b70 msgr2=0x7f11680f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:50:37.501 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:37.499+0000 7f116eea8700 1 -- 172.21.15.137:0/483537427 shutdown_connections 2023-11-15T01:50:37.501 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:37.499+0000 7f116eea8700 1 -- 172.21.15.137:0/483537427 wait complete. 2023-11-15T01:50:37.502 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:50:38.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:50:37 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:50:36.697594+0000 mgr.smithi055.szwngw (mgr.14164) 28 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:50:38.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:50:37 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:50:37.498138+0000 mon.smithi055 (mon.0) 127 : audit [DBG] from='client.? 172.21.15.137:0/483537427' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:50:40.018 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:50:40.019 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:50:40.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:50:39 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:50:38.697927+0000 mgr.smithi055.szwngw (mgr.14164) 29 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:50:41.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:50:41 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:50:40.698295+0000 mgr.smithi055.szwngw (mgr.14164) 30 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:50:44.079 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:50:43 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:50:42.698596+0000 mgr.smithi055.szwngw (mgr.14164) 31 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:50:44.079 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:50:43 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:50:42.847771+0000 mon.smithi055 (mon.0) 128 : audit [INF] from='mgr.14164 172.21.15.55:0/59463371' entity='mgr.smithi055.szwngw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi055.szwngw/mirror_snapshot_schedule"}]: dispatch 2023-11-15T01:50:44.079 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:50:43 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:50:42.851085+0000 mon.smithi055 (mon.0) 129 : audit [INF] from='mgr.14164 172.21.15.55:0/59463371' entity='mgr.smithi055.szwngw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi055.szwngw/trash_purge_schedule"}]: dispatch 2023-11-15T01:50:44.552 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:44.549+0000 7feaac1d2700 1 -- 172.21.15.137:0/3977683228 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feaa40f5410 msgr2=0x7feaa40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:50:44.552 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:44.549+0000 7feaac1d2700 1 --2- 172.21.15.137:0/3977683228 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feaa40f5410 0x7feaa40f5830 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7fea98009fb0 tx=0x7fea98009b30).stop 2023-11-15T01:50:44.552 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:44.549+0000 7feaac1d2700 1 -- 172.21.15.137:0/3977683228 shutdown_connections 2023-11-15T01:50:44.552 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:44.549+0000 7feaac1d2700 1 --2- 172.21.15.137:0/3977683228 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feaa40f5410 0x7feaa40f5830 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:50:44.553 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:44.549+0000 7feaac1d2700 1 -- 172.21.15.137:0/3977683228 >> 172.21.15.137:0/3977683228 conn(0x7feaa40f0b70 msgr2=0x7feaa40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:50:44.553 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:44.549+0000 7feaac1d2700 1 -- 172.21.15.137:0/3977683228 shutdown_connections 2023-11-15T01:50:44.553 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:44.549+0000 7feaac1d2700 1 -- 172.21.15.137:0/3977683228 wait complete. 2023-11-15T01:50:44.553 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:44.550+0000 7feaac1d2700 1 Processor -- start 2023-11-15T01:50:44.553 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:44.550+0000 7feaac1d2700 1 -- start start 2023-11-15T01:50:44.554 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:44.550+0000 7feaac1d2700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feaa40f5410 0x7feaa4109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:50:44.554 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:44.550+0000 7feaac1d2700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaa4060240 con 0x7feaa40f5410 2023-11-15T01:50:44.554 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:44.550+0000 7feaa9f6e700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feaa40f5410 0x7feaa4109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:50:44.554 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:44.550+0000 7feaa9f6e700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feaa40f5410 0x7feaa4109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:48730/0 (socket says 172.21.15.137:48730) 2023-11-15T01:50:44.554 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:44.550+0000 7feaa9f6e700 1 -- 172.21.15.137:0/339310651 learned_addr learned my addr 172.21.15.137:0/339310651 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:50:44.555 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:44.550+0000 7feaa9f6e700 1 -- 172.21.15.137:0/339310651 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea98005040 con 0x7feaa40f5410 2023-11-15T01:50:44.555 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:44.551+0000 7feaa9f6e700 1 --2- 172.21.15.137:0/339310651 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feaa40f5410 0x7feaa4109430 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fea98013510 tx=0x7fea9800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:50:44.555 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:44.551+0000 7fea967fc700 1 -- 172.21.15.137:0/339310651 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fea98009320 con 0x7feaa40f5410 2023-11-15T01:50:44.555 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:44.551+0000 7fea967fc700 1 -- 172.21.15.137:0/339310651 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fea98013f00 con 0x7feaa40f5410 2023-11-15T01:50:44.555 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:44.551+0000 7feaac1d2700 1 -- 172.21.15.137:0/339310651 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feaa4060440 con 0x7feaa40f5410 2023-11-15T01:50:44.556 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:44.551+0000 7fea967fc700 1 -- 172.21.15.137:0/339310651 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fea98014d90 con 0x7feaa40f5410 2023-11-15T01:50:44.556 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:44.551+0000 7feaac1d2700 1 -- 172.21.15.137:0/339310651 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feaa4109ae0 con 0x7feaa40f5410 2023-11-15T01:50:44.557 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:44.552+0000 7fea967fc700 1 -- 172.21.15.137:0/339310651 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fea980095f0 con 0x7feaa40f5410 2023-11-15T01:50:44.557 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:44.552+0000 7feaac1d2700 1 -- 172.21.15.137:0/339310651 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feaa4045ad0 con 0x7feaa40f5410 2023-11-15T01:50:44.558 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:44.552+0000 7fea967fc700 1 --2- 172.21.15.137:0/339310651 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fea9002ff60 0x7fea90032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:50:44.558 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:44.552+0000 7fea967fc700 1 -- 172.21.15.137:0/339310651 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fea98041000 con 0x7feaa40f5410 2023-11-15T01:50:44.558 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:44.552+0000 7feaa976d700 1 --2- 172.21.15.137:0/339310651 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fea9002ff60 0x7fea90032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:50:44.559 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:44.552+0000 7feaa976d700 1 --2- 172.21.15.137:0/339310651 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fea9002ff60 0x7fea90032420 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7feaa00096a0 tx=0x7feaa0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:50:44.559 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:44.557+0000 7fea967fc700 1 -- 172.21.15.137:0/339310651 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fea9801d7f0 con 0x7feaa40f5410 2023-11-15T01:50:44.761 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:44.760+0000 7feaac1d2700 1 -- 172.21.15.137:0/339310651 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7feaa40fae60 con 0x7feaa40f5410 2023-11-15T01:50:44.762 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:44.761+0000 7fea967fc700 1 -- 172.21.15.137:0/339310651 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fea98010230 con 0x7feaa40f5410 2023-11-15T01:50:44.762 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:50:44.763 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:50:44.764 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:44.762+0000 7feaac1d2700 1 -- 172.21.15.137:0/339310651 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fea9002ff60 msgr2=0x7fea90032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:50:44.764 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:44.762+0000 7feaac1d2700 1 --2- 172.21.15.137:0/339310651 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fea9002ff60 0x7fea90032420 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7feaa00096a0 tx=0x7feaa0006b40).stop 2023-11-15T01:50:44.764 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:44.762+0000 7feaac1d2700 1 -- 172.21.15.137:0/339310651 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feaa40f5410 msgr2=0x7feaa4109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:50:44.765 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:44.762+0000 7feaac1d2700 1 --2- 172.21.15.137:0/339310651 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feaa40f5410 0x7feaa4109430 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fea98013510 tx=0x7fea9800b040).stop 2023-11-15T01:50:44.765 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:44.762+0000 7feaac1d2700 1 -- 172.21.15.137:0/339310651 shutdown_connections 2023-11-15T01:50:44.765 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:44.762+0000 7feaac1d2700 1 --2- 172.21.15.137:0/339310651 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fea9002ff60 0x7fea90032420 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:50:44.765 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:44.762+0000 7feaac1d2700 1 --2- 172.21.15.137:0/339310651 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feaa40f5410 0x7feaa4109430 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:50:44.765 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:44.762+0000 7feaac1d2700 1 -- 172.21.15.137:0/339310651 >> 172.21.15.137:0/339310651 conn(0x7feaa40f0b70 msgr2=0x7feaa40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:50:44.766 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:44.762+0000 7feaac1d2700 1 -- 172.21.15.137:0/339310651 shutdown_connections 2023-11-15T01:50:44.766 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:44.762+0000 7feaac1d2700 1 -- 172.21.15.137:0/339310651 wait complete. 2023-11-15T01:50:44.766 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:50:46.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:50:45 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:50:44.698893+0000 mgr.smithi055.szwngw (mgr.14164) 32 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:50:46.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:50:45 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:50:44.761945+0000 mon.smithi055 (mon.0) 130 : audit [DBG] from='client.? 172.21.15.137:0/339310651' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:50:46.142 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:50:46.143 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:50:47.328 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:47.326+0000 7fe92b797700 1 -- 172.21.15.137:0/2693547006 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe9240f5410 msgr2=0x7fe9240f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:50:47.328 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:47.326+0000 7fe92b797700 1 --2- 172.21.15.137:0/2693547006 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe9240f5410 0x7fe9240f5830 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7fe918009fb0 tx=0x7fe918009b30).stop 2023-11-15T01:50:47.328 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:47.327+0000 7fe92b797700 1 -- 172.21.15.137:0/2693547006 shutdown_connections 2023-11-15T01:50:47.328 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:47.327+0000 7fe92b797700 1 --2- 172.21.15.137:0/2693547006 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe9240f5410 0x7fe9240f5830 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:50:47.329 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:47.327+0000 7fe92b797700 1 -- 172.21.15.137:0/2693547006 >> 172.21.15.137:0/2693547006 conn(0x7fe9240f0b70 msgr2=0x7fe9240f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:50:47.329 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:47.327+0000 7fe92b797700 1 -- 172.21.15.137:0/2693547006 shutdown_connections 2023-11-15T01:50:47.329 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:47.327+0000 7fe92b797700 1 -- 172.21.15.137:0/2693547006 wait complete. 2023-11-15T01:50:47.329 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:47.327+0000 7fe92b797700 1 Processor -- start 2023-11-15T01:50:47.329 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:47.327+0000 7fe92b797700 1 -- start start 2023-11-15T01:50:47.330 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:47.327+0000 7fe92b797700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe9240f5410 0x7fe924109400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:50:47.330 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:47.327+0000 7fe92b797700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe924060240 con 0x7fe9240f5410 2023-11-15T01:50:47.332 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:47.327+0000 7fe929533700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe9240f5410 0x7fe924109400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:50:47.332 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:47.327+0000 7fe929533700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe9240f5410 0x7fe924109400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:48746/0 (socket says 172.21.15.137:48746) 2023-11-15T01:50:47.332 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:47.328+0000 7fe929533700 1 -- 172.21.15.137:0/1680331159 learned_addr learned my addr 172.21.15.137:0/1680331159 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:50:47.332 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:47.328+0000 7fe929533700 1 -- 172.21.15.137:0/1680331159 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe918005040 con 0x7fe9240f5410 2023-11-15T01:50:47.332 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:47.328+0000 7fe929533700 1 --2- 172.21.15.137:0/1680331159 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe9240f5410 0x7fe924109400 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7fe918013510 tx=0x7fe91800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:50:47.333 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:47.328+0000 7fe915ffb700 1 -- 172.21.15.137:0/1680331159 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe918009320 con 0x7fe9240f5410 2023-11-15T01:50:47.333 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:47.328+0000 7fe92b797700 1 -- 172.21.15.137:0/1680331159 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe924060440 con 0x7fe9240f5410 2023-11-15T01:50:47.333 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:47.328+0000 7fe915ffb700 1 -- 172.21.15.137:0/1680331159 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe918013f00 con 0x7fe9240f5410 2023-11-15T01:50:47.333 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:47.328+0000 7fe915ffb700 1 -- 172.21.15.137:0/1680331159 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe918014d90 con 0x7fe9240f5410 2023-11-15T01:50:47.333 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:47.328+0000 7fe92b797700 1 -- 172.21.15.137:0/1680331159 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe924109ab0 con 0x7fe9240f5410 2023-11-15T01:50:47.334 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:47.329+0000 7fe915ffb700 1 -- 172.21.15.137:0/1680331159 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe9180263a0 con 0x7fe9240f5410 2023-11-15T01:50:47.334 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:47.329+0000 7fe915ffb700 1 --2- 172.21.15.137:0/1680331159 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fe91002ff60 0x7fe910032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:50:47.334 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:47.329+0000 7fe915ffb700 1 -- 172.21.15.137:0/1680331159 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe9180410c0 con 0x7fe9240f5410 2023-11-15T01:50:47.334 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:47.329+0000 7fe92b797700 1 -- 172.21.15.137:0/1680331159 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe924045ad0 con 0x7fe9240f5410 2023-11-15T01:50:47.334 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:47.330+0000 7fe928d32700 1 --2- 172.21.15.137:0/1680331159 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fe91002ff60 0x7fe910032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:50:47.335 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:47.330+0000 7fe928d32700 1 --2- 172.21.15.137:0/1680331159 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fe91002ff60 0x7fe910032420 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7fe9200096a0 tx=0x7fe920006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:50:47.335 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:47.334+0000 7fe915ffb700 1 -- 172.21.15.137:0/1680331159 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe918012920 con 0x7fe9240f5410 2023-11-15T01:50:47.541 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:47.540+0000 7fe92b797700 1 -- 172.21.15.137:0/1680331159 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe9240fae60 con 0x7fe9240f5410 2023-11-15T01:50:47.541 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:47.541+0000 7fe915ffb700 1 -- 172.21.15.137:0/1680331159 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe9240fae60 con 0x7fe9240f5410 2023-11-15T01:50:47.542 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:50:47.542 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:50:47.543 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:47.542+0000 7fe92b797700 1 -- 172.21.15.137:0/1680331159 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fe91002ff60 msgr2=0x7fe910032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:50:47.543 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:47.542+0000 7fe92b797700 1 --2- 172.21.15.137:0/1680331159 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fe91002ff60 0x7fe910032420 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7fe9200096a0 tx=0x7fe920006b40).stop 2023-11-15T01:50:47.543 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:47.542+0000 7fe92b797700 1 -- 172.21.15.137:0/1680331159 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe9240f5410 msgr2=0x7fe924109400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:50:47.543 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:47.542+0000 7fe92b797700 1 --2- 172.21.15.137:0/1680331159 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe9240f5410 0x7fe924109400 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7fe918013510 tx=0x7fe91800b040).stop 2023-11-15T01:50:47.544 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:47.542+0000 7fe92b797700 1 -- 172.21.15.137:0/1680331159 shutdown_connections 2023-11-15T01:50:47.544 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:47.542+0000 7fe92b797700 1 --2- 172.21.15.137:0/1680331159 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fe91002ff60 0x7fe910032420 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:50:47.544 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:47.542+0000 7fe92b797700 1 --2- 172.21.15.137:0/1680331159 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe9240f5410 0x7fe924109400 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:50:47.544 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:47.542+0000 7fe92b797700 1 -- 172.21.15.137:0/1680331159 >> 172.21.15.137:0/1680331159 conn(0x7fe9240f0b70 msgr2=0x7fe9240f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:50:47.544 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:47.542+0000 7fe92b797700 1 -- 172.21.15.137:0/1680331159 shutdown_connections 2023-11-15T01:50:47.545 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:47.542+0000 7fe92b797700 1 -- 172.21.15.137:0/1680331159 wait complete. 2023-11-15T01:50:47.545 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:50:48.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:50:47 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:50:46.699271+0000 mgr.smithi055.szwngw (mgr.14164) 33 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:50:48.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:50:47 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:50:47.541700+0000 mon.smithi055 (mon.0) 131 : audit [DBG] from='client.? 172.21.15.137:0/1680331159' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:50:48.927 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:50:48.928 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:50:50.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:50:49 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:50:48.699598+0000 mgr.smithi055.szwngw (mgr.14164) 34 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:50:51.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:50:51 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:50:50.699907+0000 mgr.smithi055.szwngw (mgr.14164) 35 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:50:52.121 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:51.993+0000 7fe017c30700 1 -- 172.21.15.137:0/1527145588 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe0100f5410 msgr2=0x7fe0100f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:50:52.121 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:51.993+0000 7fe017c30700 1 --2- 172.21.15.137:0/1527145588 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe0100f5410 0x7fe0100f5830 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7fe000009fb0 tx=0x7fe000009b30).stop 2023-11-15T01:50:52.121 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:51.993+0000 7fe017c30700 1 -- 172.21.15.137:0/1527145588 shutdown_connections 2023-11-15T01:50:52.122 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:51.993+0000 7fe017c30700 1 --2- 172.21.15.137:0/1527145588 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe0100f5410 0x7fe0100f5830 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:50:52.122 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:51.993+0000 7fe017c30700 1 -- 172.21.15.137:0/1527145588 >> 172.21.15.137:0/1527145588 conn(0x7fe0100f0b70 msgr2=0x7fe0100f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:50:52.122 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:51.993+0000 7fe017c30700 1 -- 172.21.15.137:0/1527145588 shutdown_connections 2023-11-15T01:50:52.122 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:51.993+0000 7fe017c30700 1 -- 172.21.15.137:0/1527145588 wait complete. 2023-11-15T01:50:52.123 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:51.994+0000 7fe017c30700 1 Processor -- start 2023-11-15T01:50:52.123 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:51.994+0000 7fe017c30700 1 -- start start 2023-11-15T01:50:52.123 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:51.994+0000 7fe017c30700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe0100f5410 0x7fe010106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:50:52.123 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:51.994+0000 7fe017c30700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe010106c10 con 0x7fe0100f5410 2023-11-15T01:50:52.123 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:51.994+0000 7fe0159cc700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe0100f5410 0x7fe010106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:50:52.124 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:51.994+0000 7fe0159cc700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe0100f5410 0x7fe010106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:59992/0 (socket says 172.21.15.137:59992) 2023-11-15T01:50:52.124 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:51.994+0000 7fe0159cc700 1 -- 172.21.15.137:0/2751864706 learned_addr learned my addr 172.21.15.137:0/2751864706 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:50:52.124 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:51.994+0000 7fe0159cc700 1 -- 172.21.15.137:0/2751864706 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe000005040 con 0x7fe0100f5410 2023-11-15T01:50:52.124 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:51.995+0000 7fe0159cc700 1 --2- 172.21.15.137:0/2751864706 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe0100f5410 0x7fe010106720 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7fe000013510 tx=0x7fe00000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:50:52.124 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:51.995+0000 7fe0067fc700 1 -- 172.21.15.137:0/2751864706 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe000009320 con 0x7fe0100f5410 2023-11-15T01:50:52.125 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:51.995+0000 7fe017c30700 1 -- 172.21.15.137:0/2751864706 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe010106e10 con 0x7fe0100f5410 2023-11-15T01:50:52.125 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:51.995+0000 7fe0067fc700 1 -- 172.21.15.137:0/2751864706 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe000013f00 con 0x7fe0100f5410 2023-11-15T01:50:52.125 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:51.995+0000 7fe0067fc700 1 -- 172.21.15.137:0/2751864706 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe000014d90 con 0x7fe0100f5410 2023-11-15T01:50:52.125 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:51.995+0000 7fe017c30700 1 -- 172.21.15.137:0/2751864706 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0101072a0 con 0x7fe0100f5410 2023-11-15T01:50:52.125 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:51.996+0000 7fe0067fc700 1 -- 172.21.15.137:0/2751864706 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe0000263a0 con 0x7fe0100f5410 2023-11-15T01:50:52.126 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:51.996+0000 7fe017c30700 1 -- 172.21.15.137:0/2751864706 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe010045ad0 con 0x7fe0100f5410 2023-11-15T01:50:52.126 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:51.996+0000 7fe0067fc700 1 --2- 172.21.15.137:0/2751864706 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fdffc02ff60 0x7fdffc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:50:52.126 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:51.996+0000 7fe0067fc700 1 -- 172.21.15.137:0/2751864706 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe0000412c0 con 0x7fe0100f5410 2023-11-15T01:50:52.126 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:51.996+0000 7fe0151cb700 1 --2- 172.21.15.137:0/2751864706 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fdffc02ff60 0x7fdffc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:50:52.126 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:51.996+0000 7fe0151cb700 1 --2- 172.21.15.137:0/2751864706 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fdffc02ff60 0x7fdffc032420 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7fe00c0096a0 tx=0x7fe00c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:50:52.127 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:52.001+0000 7fe0067fc700 1 -- 172.21.15.137:0/2751864706 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe00001cb50 con 0x7fe0100f5410 2023-11-15T01:50:52.217 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:52.216+0000 7fe017c30700 1 -- 172.21.15.137:0/2751864706 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe010108150 con 0x7fe0100f5410 2023-11-15T01:50:52.218 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:52.217+0000 7fe0067fc700 1 -- 172.21.15.137:0/2751864706 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe00001c970 con 0x7fe0100f5410 2023-11-15T01:50:52.218 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:50:52.218 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:50:52.220 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:52.219+0000 7fe017c30700 1 -- 172.21.15.137:0/2751864706 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fdffc02ff60 msgr2=0x7fdffc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:50:52.220 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:52.219+0000 7fe017c30700 1 --2- 172.21.15.137:0/2751864706 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fdffc02ff60 0x7fdffc032420 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7fe00c0096a0 tx=0x7fe00c006b40).stop 2023-11-15T01:50:52.221 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:52.219+0000 7fe017c30700 1 -- 172.21.15.137:0/2751864706 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe0100f5410 msgr2=0x7fe010106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:50:52.221 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:52.219+0000 7fe017c30700 1 --2- 172.21.15.137:0/2751864706 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe0100f5410 0x7fe010106720 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7fe000013510 tx=0x7fe00000b040).stop 2023-11-15T01:50:52.221 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:52.219+0000 7fe017c30700 1 -- 172.21.15.137:0/2751864706 shutdown_connections 2023-11-15T01:50:52.221 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:52.219+0000 7fe017c30700 1 --2- 172.21.15.137:0/2751864706 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fdffc02ff60 0x7fdffc032420 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:50:52.221 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:52.219+0000 7fe017c30700 1 --2- 172.21.15.137:0/2751864706 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe0100f5410 0x7fe010106720 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:50:52.222 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:52.219+0000 7fe017c30700 1 -- 172.21.15.137:0/2751864706 >> 172.21.15.137:0/2751864706 conn(0x7fe0100f0b70 msgr2=0x7fe01005fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:50:52.222 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:52.219+0000 7fe017c30700 1 -- 172.21.15.137:0/2751864706 shutdown_connections 2023-11-15T01:50:52.222 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:52.219+0000 7fe017c30700 1 -- 172.21.15.137:0/2751864706 wait complete. 2023-11-15T01:50:52.222 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:50:52.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:50:52 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:50:52.218221+0000 mon.smithi055 (mon.0) 132 : audit [DBG] from='client.? 172.21.15.137:0/2751864706' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:50:53.700 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:50:53.701 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:50:53.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:50:53 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:50:52.700345+0000 mgr.smithi055.szwngw (mgr.14164) 36 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:50:54.834 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:54.833+0000 7f1d987e4700 1 -- 172.21.15.137:0/4019688636 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1d900f5410 msgr2=0x7f1d900f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:50:54.834 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:54.833+0000 7f1d987e4700 1 --2- 172.21.15.137:0/4019688636 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1d900f5410 0x7f1d900f5830 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f1d8c009fb0 tx=0x7f1d8c009b30).stop 2023-11-15T01:50:54.835 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:54.833+0000 7f1d987e4700 1 -- 172.21.15.137:0/4019688636 shutdown_connections 2023-11-15T01:50:54.835 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:54.833+0000 7f1d987e4700 1 --2- 172.21.15.137:0/4019688636 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1d900f5410 0x7f1d900f5830 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:50:54.835 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:54.833+0000 7f1d987e4700 1 -- 172.21.15.137:0/4019688636 >> 172.21.15.137:0/4019688636 conn(0x7f1d900f0b70 msgr2=0x7f1d900f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:50:54.835 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:54.833+0000 7f1d987e4700 1 -- 172.21.15.137:0/4019688636 shutdown_connections 2023-11-15T01:50:54.835 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:54.833+0000 7f1d987e4700 1 -- 172.21.15.137:0/4019688636 wait complete. 2023-11-15T01:50:54.837 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:54.834+0000 7f1d987e4700 1 Processor -- start 2023-11-15T01:50:54.838 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:54.834+0000 7f1d987e4700 1 -- start start 2023-11-15T01:50:54.838 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:54.834+0000 7f1d987e4700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1d900f5410 0x7f1d90105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:50:54.838 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:54.834+0000 7f1d987e4700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d90105b20 con 0x7f1d900f5410 2023-11-15T01:50:54.838 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:54.835+0000 7f1d96580700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1d900f5410 0x7f1d90105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:50:54.839 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:54.835+0000 7f1d96580700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1d900f5410 0x7f1d90105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:60006/0 (socket says 172.21.15.137:60006) 2023-11-15T01:50:54.839 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:54.835+0000 7f1d96580700 1 -- 172.21.15.137:0/3520276288 learned_addr learned my addr 172.21.15.137:0/3520276288 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:50:54.839 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:54.835+0000 7f1d96580700 1 -- 172.21.15.137:0/3520276288 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d8c005040 con 0x7f1d900f5410 2023-11-15T01:50:54.839 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:54.835+0000 7f1d96580700 1 --2- 172.21.15.137:0/3520276288 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1d900f5410 0x7f1d90105630 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f1d8c013510 tx=0x7f1d8c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:50:54.839 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:54.835+0000 7f1d86ffd700 1 -- 172.21.15.137:0/3520276288 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1d8c009320 con 0x7f1d900f5410 2023-11-15T01:50:54.840 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:54.835+0000 7f1d987e4700 1 -- 172.21.15.137:0/3520276288 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d90105d20 con 0x7f1d900f5410 2023-11-15T01:50:54.840 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:54.835+0000 7f1d86ffd700 1 -- 172.21.15.137:0/3520276288 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1d8c013f00 con 0x7f1d900f5410 2023-11-15T01:50:54.840 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:54.835+0000 7f1d86ffd700 1 -- 172.21.15.137:0/3520276288 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1d8c014d90 con 0x7f1d900f5410 2023-11-15T01:50:54.840 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:54.835+0000 7f1d987e4700 1 -- 172.21.15.137:0/3520276288 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d901061b0 con 0x7f1d900f5410 2023-11-15T01:50:54.840 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:54.836+0000 7f1d86ffd700 1 -- 172.21.15.137:0/3520276288 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1d8c0095f0 con 0x7f1d900f5410 2023-11-15T01:50:54.840 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:54.836+0000 7f1d987e4700 1 -- 172.21.15.137:0/3520276288 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d90045ad0 con 0x7f1d900f5410 2023-11-15T01:50:54.841 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:54.836+0000 7f1d86ffd700 1 --2- 172.21.15.137:0/3520276288 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f1d7c02ff60 0x7f1d7c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:50:54.841 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:54.836+0000 7f1d86ffd700 1 -- 172.21.15.137:0/3520276288 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1d8c0410a0 con 0x7f1d900f5410 2023-11-15T01:50:54.841 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:54.836+0000 7f1d95d7f700 1 --2- 172.21.15.137:0/3520276288 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f1d7c02ff60 0x7f1d7c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:50:54.841 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:54.837+0000 7f1d95d7f700 1 --2- 172.21.15.137:0/3520276288 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f1d7c02ff60 0x7f1d7c032420 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f1d800096a0 tx=0x7f1d80006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:50:54.842 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:54.841+0000 7f1d86ffd700 1 -- 172.21.15.137:0/3520276288 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1d8c01d7f0 con 0x7f1d900f5410 2023-11-15T01:50:55.048 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:55.047+0000 7f1d987e4700 1 -- 172.21.15.137:0/3520276288 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1d900fae60 con 0x7f1d900f5410 2023-11-15T01:50:55.048 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:55.047+0000 7f1d86ffd700 1 -- 172.21.15.137:0/3520276288 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1d8c010230 con 0x7f1d900f5410 2023-11-15T01:50:55.049 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:50:55.049 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:50:55.050 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:55.049+0000 7f1d987e4700 1 -- 172.21.15.137:0/3520276288 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f1d7c02ff60 msgr2=0x7f1d7c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:50:55.050 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:55.049+0000 7f1d987e4700 1 --2- 172.21.15.137:0/3520276288 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f1d7c02ff60 0x7f1d7c032420 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f1d800096a0 tx=0x7f1d80006b40).stop 2023-11-15T01:50:55.050 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:55.049+0000 7f1d987e4700 1 -- 172.21.15.137:0/3520276288 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1d900f5410 msgr2=0x7f1d90105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:50:55.050 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:55.049+0000 7f1d987e4700 1 --2- 172.21.15.137:0/3520276288 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1d900f5410 0x7f1d90105630 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f1d8c013510 tx=0x7f1d8c00b040).stop 2023-11-15T01:50:55.051 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:55.049+0000 7f1d987e4700 1 -- 172.21.15.137:0/3520276288 shutdown_connections 2023-11-15T01:50:55.051 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:55.049+0000 7f1d987e4700 1 --2- 172.21.15.137:0/3520276288 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f1d7c02ff60 0x7f1d7c032420 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:50:55.051 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:55.049+0000 7f1d987e4700 1 --2- 172.21.15.137:0/3520276288 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1d900f5410 0x7f1d90105630 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:50:55.051 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:55.049+0000 7f1d987e4700 1 -- 172.21.15.137:0/3520276288 >> 172.21.15.137:0/3520276288 conn(0x7f1d900f0b70 msgr2=0x7f1d900f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:50:55.052 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:55.049+0000 7f1d987e4700 1 -- 172.21.15.137:0/3520276288 shutdown_connections 2023-11-15T01:50:55.052 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:55.049+0000 7f1d987e4700 1 -- 172.21.15.137:0/3520276288 wait complete. 2023-11-15T01:50:55.052 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:50:56.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:50:55 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:50:54.700741+0000 mgr.smithi055.szwngw (mgr.14164) 37 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:50:56.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:50:55 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:50:55.048529+0000 mon.smithi055 (mon.0) 133 : audit [DBG] from='client.? 172.21.15.137:0/3520276288' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:50:56.438 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:50:56.439 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:50:57.604 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:57.602+0000 7fb1ccb61700 1 -- 172.21.15.137:0/2192065726 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb1c80f5410 msgr2=0x7fb1c80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:50:57.604 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:57.602+0000 7fb1ccb61700 1 --2- 172.21.15.137:0/2192065726 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb1c80f5410 0x7fb1c80f5830 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7fb1bc009fb0 tx=0x7fb1bc009b30).stop 2023-11-15T01:50:57.605 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:57.602+0000 7fb1ccb61700 1 -- 172.21.15.137:0/2192065726 shutdown_connections 2023-11-15T01:50:57.605 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:57.602+0000 7fb1ccb61700 1 --2- 172.21.15.137:0/2192065726 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb1c80f5410 0x7fb1c80f5830 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:50:57.605 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:57.602+0000 7fb1ccb61700 1 -- 172.21.15.137:0/2192065726 >> 172.21.15.137:0/2192065726 conn(0x7fb1c80f0b70 msgr2=0x7fb1c80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:50:57.605 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:57.602+0000 7fb1ccb61700 1 -- 172.21.15.137:0/2192065726 shutdown_connections 2023-11-15T01:50:57.605 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:57.603+0000 7fb1ccb61700 1 -- 172.21.15.137:0/2192065726 wait complete. 2023-11-15T01:50:57.606 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:57.603+0000 7fb1ccb61700 1 Processor -- start 2023-11-15T01:50:57.606 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:57.603+0000 7fb1ccb61700 1 -- start start 2023-11-15T01:50:57.606 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:57.603+0000 7fb1ccb61700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb1c80f5410 0x7fb1c80fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:50:57.606 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:57.603+0000 7fb1ccb61700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1c80fb820 con 0x7fb1c80f5410 2023-11-15T01:50:57.606 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:57.603+0000 7fb1c659c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb1c80f5410 0x7fb1c80fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:50:57.607 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:57.603+0000 7fb1c659c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb1c80f5410 0x7fb1c80fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:60016/0 (socket says 172.21.15.137:60016) 2023-11-15T01:50:57.607 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:57.603+0000 7fb1c659c700 1 -- 172.21.15.137:0/4080441116 learned_addr learned my addr 172.21.15.137:0/4080441116 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:50:57.607 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:57.604+0000 7fb1c659c700 1 -- 172.21.15.137:0/4080441116 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1bc005040 con 0x7fb1c80f5410 2023-11-15T01:50:57.607 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:57.604+0000 7fb1c659c700 1 --2- 172.21.15.137:0/4080441116 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb1c80f5410 0x7fb1c80fd3a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fb1bc013510 tx=0x7fb1bc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:50:57.609 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:57.604+0000 7fb1b6ffd700 1 -- 172.21.15.137:0/4080441116 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb1bc009320 con 0x7fb1c80f5410 2023-11-15T01:50:57.609 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:57.604+0000 7fb1ccb61700 1 -- 172.21.15.137:0/4080441116 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1c80fb9c0 con 0x7fb1c80f5410 2023-11-15T01:50:57.609 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:57.604+0000 7fb1b6ffd700 1 -- 172.21.15.137:0/4080441116 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb1bc013f00 con 0x7fb1c80f5410 2023-11-15T01:50:57.609 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:57.604+0000 7fb1b6ffd700 1 -- 172.21.15.137:0/4080441116 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb1bc014d90 con 0x7fb1c80f5410 2023-11-15T01:50:57.609 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:57.604+0000 7fb1ccb61700 1 -- 172.21.15.137:0/4080441116 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1c80fbe50 con 0x7fb1c80f5410 2023-11-15T01:50:57.610 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:57.605+0000 7fb1b6ffd700 1 -- 172.21.15.137:0/4080441116 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb1bc0095f0 con 0x7fb1c80f5410 2023-11-15T01:50:57.610 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:57.605+0000 7fb1ccb61700 1 -- 172.21.15.137:0/4080441116 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1c8059100 con 0x7fb1c80f5410 2023-11-15T01:50:57.610 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:57.605+0000 7fb1b6ffd700 1 --2- 172.21.15.137:0/4080441116 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fb1b002ff60 0x7fb1b0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:50:57.610 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:57.605+0000 7fb1b6ffd700 1 -- 172.21.15.137:0/4080441116 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb1bc041000 con 0x7fb1c80f5410 2023-11-15T01:50:57.610 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:57.605+0000 7fb1c5d9b700 1 --2- 172.21.15.137:0/4080441116 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fb1b002ff60 0x7fb1b0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:50:57.611 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:57.606+0000 7fb1c5d9b700 1 --2- 172.21.15.137:0/4080441116 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fb1b002ff60 0x7fb1b0032420 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7fb1b80096a0 tx=0x7fb1b8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:50:57.611 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:57.610+0000 7fb1b6ffd700 1 -- 172.21.15.137:0/4080441116 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb1bc014420 con 0x7fb1c80f5410 2023-11-15T01:50:57.817 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:57.816+0000 7fb1ccb61700 1 -- 172.21.15.137:0/4080441116 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb1c80fae60 con 0x7fb1c80f5410 2023-11-15T01:50:57.818 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:57.817+0000 7fb1b6ffd700 1 -- 172.21.15.137:0/4080441116 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb1bc01d7f0 con 0x7fb1c80f5410 2023-11-15T01:50:57.818 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:50:57.819 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:50:57.819 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:57.819+0000 7fb1ccb61700 1 -- 172.21.15.137:0/4080441116 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fb1b002ff60 msgr2=0x7fb1b0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:50:57.820 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:57.819+0000 7fb1ccb61700 1 --2- 172.21.15.137:0/4080441116 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fb1b002ff60 0x7fb1b0032420 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7fb1b80096a0 tx=0x7fb1b8006b40).stop 2023-11-15T01:50:57.820 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:57.819+0000 7fb1ccb61700 1 -- 172.21.15.137:0/4080441116 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb1c80f5410 msgr2=0x7fb1c80fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:50:57.820 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:57.819+0000 7fb1ccb61700 1 --2- 172.21.15.137:0/4080441116 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb1c80f5410 0x7fb1c80fd3a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fb1bc013510 tx=0x7fb1bc00b040).stop 2023-11-15T01:50:57.820 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:57.819+0000 7fb1ccb61700 1 -- 172.21.15.137:0/4080441116 shutdown_connections 2023-11-15T01:50:57.820 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:57.819+0000 7fb1ccb61700 1 --2- 172.21.15.137:0/4080441116 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fb1b002ff60 0x7fb1b0032420 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:50:57.821 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:57.819+0000 7fb1ccb61700 1 --2- 172.21.15.137:0/4080441116 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb1c80f5410 0x7fb1c80fd3a0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:50:57.821 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:57.819+0000 7fb1ccb61700 1 -- 172.21.15.137:0/4080441116 >> 172.21.15.137:0/4080441116 conn(0x7fb1c80f0b70 msgr2=0x7fb1c805fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:50:57.821 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:57.819+0000 7fb1ccb61700 1 -- 172.21.15.137:0/4080441116 shutdown_connections 2023-11-15T01:50:57.821 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:50:57.819+0000 7fb1ccb61700 1 -- 172.21.15.137:0/4080441116 wait complete. 2023-11-15T01:50:57.822 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:50:58.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:50:57 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:50:56.701145+0000 mgr.smithi055.szwngw (mgr.14164) 38 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:50:59.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:50:58 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:50:57.818280+0000 mon.smithi055 (mon.0) 134 : audit [DBG] from='client.? 172.21.15.137:0/4080441116' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:50:59.146 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:50:59.147 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:51:00.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:50:59 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:50:58.701470+0000 mgr.smithi055.szwngw (mgr.14164) 39 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:51:00.322 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:00.321+0000 7ff6a3c11700 1 -- 172.21.15.137:0/4242945352 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff69c060280 msgr2=0x7ff69c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:51:00.323 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:00.321+0000 7ff6a3c11700 1 --2- 172.21.15.137:0/4242945352 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff69c060280 0x7ff69c0f90f0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7ff698009fb0 tx=0x7ff698009b30).stop 2023-11-15T01:51:00.323 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:00.321+0000 7ff6a3c11700 1 -- 172.21.15.137:0/4242945352 shutdown_connections 2023-11-15T01:51:00.323 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:00.321+0000 7ff6a3c11700 1 --2- 172.21.15.137:0/4242945352 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff69c060280 0x7ff69c0f90f0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:51:00.323 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:00.321+0000 7ff6a3c11700 1 -- 172.21.15.137:0/4242945352 >> 172.21.15.137:0/4242945352 conn(0x7ff69c0f0b70 msgr2=0x7ff69c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:51:00.324 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:00.321+0000 7ff6a3c11700 1 -- 172.21.15.137:0/4242945352 shutdown_connections 2023-11-15T01:51:00.324 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:00.321+0000 7ff6a3c11700 1 -- 172.21.15.137:0/4242945352 wait complete. 2023-11-15T01:51:00.324 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:00.322+0000 7ff6a3c11700 1 Processor -- start 2023-11-15T01:51:00.324 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:00.322+0000 7ff6a3c11700 1 -- start start 2023-11-15T01:51:00.324 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:00.322+0000 7ff6a3c11700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff69c060280 0x7ff69c1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:51:00.325 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:00.322+0000 7ff6a3c11700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff69c106bd0 con 0x7ff69c060280 2023-11-15T01:51:00.326 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:00.322+0000 7ff6a19ad700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff69c060280 0x7ff69c1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:51:00.327 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:00.322+0000 7ff6a19ad700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff69c060280 0x7ff69c1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:46944/0 (socket says 172.21.15.137:46944) 2023-11-15T01:51:00.327 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:00.322+0000 7ff6a19ad700 1 -- 172.21.15.137:0/3097144148 learned_addr learned my addr 172.21.15.137:0/3097144148 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:51:00.327 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:00.322+0000 7ff6a19ad700 1 -- 172.21.15.137:0/3097144148 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff698005040 con 0x7ff69c060280 2023-11-15T01:51:00.327 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:00.322+0000 7ff6a19ad700 1 --2- 172.21.15.137:0/3097144148 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff69c060280 0x7ff69c1066e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7ff698013510 tx=0x7ff69800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:51:00.327 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:00.323+0000 7ff6927fc700 1 -- 172.21.15.137:0/3097144148 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff698009320 con 0x7ff69c060280 2023-11-15T01:51:00.328 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:00.323+0000 7ff6a3c11700 1 -- 172.21.15.137:0/3097144148 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff69c106dd0 con 0x7ff69c060280 2023-11-15T01:51:00.328 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:00.323+0000 7ff6927fc700 1 -- 172.21.15.137:0/3097144148 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff698013f00 con 0x7ff69c060280 2023-11-15T01:51:00.328 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:00.323+0000 7ff6927fc700 1 -- 172.21.15.137:0/3097144148 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff698014dc0 con 0x7ff69c060280 2023-11-15T01:51:00.328 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:00.323+0000 7ff6a3c11700 1 -- 172.21.15.137:0/3097144148 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff69c107260 con 0x7ff69c060280 2023-11-15T01:51:00.328 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:00.323+0000 7ff6927fc700 1 -- 172.21.15.137:0/3097144148 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff698014420 con 0x7ff69c060280 2023-11-15T01:51:00.329 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:00.323+0000 7ff6a3c11700 1 -- 172.21.15.137:0/3097144148 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff69c045ad0 con 0x7ff69c060280 2023-11-15T01:51:00.329 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:00.324+0000 7ff6927fc700 1 --2- 172.21.15.137:0/3097144148 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7ff68802ff60 0x7ff688032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:51:00.329 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:00.324+0000 7ff6927fc700 1 -- 172.21.15.137:0/3097144148 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff6980410d0 con 0x7ff69c060280 2023-11-15T01:51:00.329 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:00.324+0000 7ff6a11ac700 1 --2- 172.21.15.137:0/3097144148 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7ff68802ff60 0x7ff688032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:51:00.329 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:00.324+0000 7ff6a11ac700 1 --2- 172.21.15.137:0/3097144148 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7ff68802ff60 0x7ff688032420 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7ff68c0096a0 tx=0x7ff68c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:51:00.331 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:00.328+0000 7ff6927fc700 1 -- 172.21.15.137:0/3097144148 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff6980095f0 con 0x7ff69c060280 2023-11-15T01:51:00.535 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:00.534+0000 7ff6a3c11700 1 -- 172.21.15.137:0/3097144148 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff69c108110 con 0x7ff69c060280 2023-11-15T01:51:00.536 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:00.535+0000 7ff6927fc700 1 -- 172.21.15.137:0/3097144148 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff69801cb10 con 0x7ff69c060280 2023-11-15T01:51:00.536 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:51:00.536 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:51:00.537 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:00.536+0000 7ff6a3c11700 1 -- 172.21.15.137:0/3097144148 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7ff68802ff60 msgr2=0x7ff688032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:51:00.537 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:00.536+0000 7ff6a3c11700 1 --2- 172.21.15.137:0/3097144148 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7ff68802ff60 0x7ff688032420 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7ff68c0096a0 tx=0x7ff68c006b40).stop 2023-11-15T01:51:00.537 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:00.536+0000 7ff6a3c11700 1 -- 172.21.15.137:0/3097144148 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff69c060280 msgr2=0x7ff69c1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:51:00.538 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:00.536+0000 7ff6a3c11700 1 --2- 172.21.15.137:0/3097144148 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff69c060280 0x7ff69c1066e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7ff698013510 tx=0x7ff69800b040).stop 2023-11-15T01:51:00.538 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:00.536+0000 7ff6a3c11700 1 -- 172.21.15.137:0/3097144148 shutdown_connections 2023-11-15T01:51:00.538 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:00.536+0000 7ff6a3c11700 1 --2- 172.21.15.137:0/3097144148 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7ff68802ff60 0x7ff688032420 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:51:00.538 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:00.536+0000 7ff6a3c11700 1 --2- 172.21.15.137:0/3097144148 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff69c060280 0x7ff69c1066e0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:51:00.538 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:00.536+0000 7ff6a3c11700 1 -- 172.21.15.137:0/3097144148 >> 172.21.15.137:0/3097144148 conn(0x7ff69c0f0b70 msgr2=0x7ff69c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:51:00.539 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:00.536+0000 7ff6a3c11700 1 -- 172.21.15.137:0/3097144148 shutdown_connections 2023-11-15T01:51:00.539 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:00.536+0000 7ff6a3c11700 1 -- 172.21.15.137:0/3097144148 wait complete. 2023-11-15T01:51:00.540 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:51:01.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:51:01 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:51:00.535935+0000 mon.smithi055 (mon.0) 135 : audit [DBG] from='client.? 172.21.15.137:0/3097144148' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:51:01.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:51:01 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:51:00.701819+0000 mgr.smithi055.szwngw (mgr.14164) 40 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:51:01.908 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:51:01.908 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:51:03.193 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:03.190+0000 7f9b82585700 1 -- 172.21.15.137:0/2807073603 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9b7c0f51a0 msgr2=0x7f9b7c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:51:03.193 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:03.190+0000 7f9b82585700 1 --2- 172.21.15.137:0/2807073603 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9b7c0f51a0 0x7f9b7c0f55c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f9b6c009fb0 tx=0x7f9b6c009b30).stop 2023-11-15T01:51:03.193 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:03.190+0000 7f9b82585700 1 -- 172.21.15.137:0/2807073603 shutdown_connections 2023-11-15T01:51:03.193 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:03.190+0000 7f9b82585700 1 --2- 172.21.15.137:0/2807073603 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9b7c0f51a0 0x7f9b7c0f55c0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:51:03.194 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:03.190+0000 7f9b82585700 1 -- 172.21.15.137:0/2807073603 >> 172.21.15.137:0/2807073603 conn(0x7f9b7c0f09a0 msgr2=0x7f9b7c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:51:03.194 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:03.191+0000 7f9b82585700 1 -- 172.21.15.137:0/2807073603 shutdown_connections 2023-11-15T01:51:03.194 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:03.191+0000 7f9b82585700 1 -- 172.21.15.137:0/2807073603 wait complete. 2023-11-15T01:51:03.194 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:03.191+0000 7f9b82585700 1 Processor -- start 2023-11-15T01:51:03.195 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:03.191+0000 7f9b82585700 1 -- start start 2023-11-15T01:51:03.195 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:03.191+0000 7f9b82585700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9b7c0f51a0 0x7f9b7c1091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:51:03.195 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:03.191+0000 7f9b82585700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b7c060220 con 0x7f9b7c0f51a0 2023-11-15T01:51:03.195 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:03.191+0000 7f9b81583700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9b7c0f51a0 0x7f9b7c1091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:51:03.195 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:03.191+0000 7f9b81583700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9b7c0f51a0 0x7f9b7c1091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:46952/0 (socket says 172.21.15.137:46952) 2023-11-15T01:51:03.195 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:03.191+0000 7f9b81583700 1 -- 172.21.15.137:0/4194404558 learned_addr learned my addr 172.21.15.137:0/4194404558 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:51:03.196 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:03.192+0000 7f9b81583700 1 -- 172.21.15.137:0/4194404558 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b6c005040 con 0x7f9b7c0f51a0 2023-11-15T01:51:03.196 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:03.192+0000 7f9b81583700 1 --2- 172.21.15.137:0/4194404558 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9b7c0f51a0 0x7f9b7c1091d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f9b6c013510 tx=0x7f9b6c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:51:03.196 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:03.192+0000 7f9b71ffb700 1 -- 172.21.15.137:0/4194404558 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b6c009320 con 0x7f9b7c0f51a0 2023-11-15T01:51:03.196 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:03.192+0000 7f9b82585700 1 -- 172.21.15.137:0/4194404558 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b7c060420 con 0x7f9b7c0f51a0 2023-11-15T01:51:03.196 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:03.192+0000 7f9b71ffb700 1 -- 172.21.15.137:0/4194404558 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9b6c013f00 con 0x7f9b7c0f51a0 2023-11-15T01:51:03.197 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:03.192+0000 7f9b71ffb700 1 -- 172.21.15.137:0/4194404558 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b6c014dc0 con 0x7f9b7c0f51a0 2023-11-15T01:51:03.197 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:03.192+0000 7f9b82585700 1 -- 172.21.15.137:0/4194404558 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b7c109880 con 0x7f9b7c0f51a0 2023-11-15T01:51:03.198 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:03.193+0000 7f9b71ffb700 1 -- 172.21.15.137:0/4194404558 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9b6c01d3d0 con 0x7f9b7c0f51a0 2023-11-15T01:51:03.198 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:03.193+0000 7f9b71ffb700 1 --2- 172.21.15.137:0/4194404558 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f9b6802ff60 0x7f9b68032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:51:03.198 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:03.193+0000 7f9b71ffb700 1 -- 172.21.15.137:0/4194404558 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9b6c041050 con 0x7f9b7c0f51a0 2023-11-15T01:51:03.199 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:03.193+0000 7f9b82585700 1 -- 172.21.15.137:0/4194404558 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b7c059100 con 0x7f9b7c0f51a0 2023-11-15T01:51:03.199 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:03.193+0000 7f9b80d82700 1 --2- 172.21.15.137:0/4194404558 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f9b6802ff60 0x7f9b68032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:51:03.199 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:03.194+0000 7f9b80d82700 1 --2- 172.21.15.137:0/4194404558 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f9b6802ff60 0x7f9b68032420 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f9b780096a0 tx=0x7f9b78006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:51:03.200 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:03.198+0000 7f9b71ffb700 1 -- 172.21.15.137:0/4194404558 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9b6c012920 con 0x7f9b7c0f51a0 2023-11-15T01:51:03.404 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:03.403+0000 7f9b82585700 1 -- 172.21.15.137:0/4194404558 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9b7c045ad0 con 0x7f9b7c0f51a0 2023-11-15T01:51:03.405 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:03.404+0000 7f9b71ffb700 1 -- 172.21.15.137:0/4194404558 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9b6c012330 con 0x7f9b7c0f51a0 2023-11-15T01:51:03.405 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:51:03.405 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:51:03.406 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:03.406+0000 7f9b82585700 1 -- 172.21.15.137:0/4194404558 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f9b6802ff60 msgr2=0x7f9b68032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:51:03.407 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:03.406+0000 7f9b82585700 1 --2- 172.21.15.137:0/4194404558 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f9b6802ff60 0x7f9b68032420 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f9b780096a0 tx=0x7f9b78006b40).stop 2023-11-15T01:51:03.407 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:03.406+0000 7f9b82585700 1 -- 172.21.15.137:0/4194404558 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9b7c0f51a0 msgr2=0x7f9b7c1091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:51:03.407 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:03.406+0000 7f9b82585700 1 --2- 172.21.15.137:0/4194404558 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9b7c0f51a0 0x7f9b7c1091d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f9b6c013510 tx=0x7f9b6c00b040).stop 2023-11-15T01:51:03.408 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:03.406+0000 7f9b82585700 1 -- 172.21.15.137:0/4194404558 shutdown_connections 2023-11-15T01:51:03.408 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:03.406+0000 7f9b82585700 1 --2- 172.21.15.137:0/4194404558 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f9b6802ff60 0x7f9b68032420 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:51:03.408 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:03.406+0000 7f9b82585700 1 --2- 172.21.15.137:0/4194404558 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9b7c0f51a0 0x7f9b7c1091d0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:51:03.408 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:03.406+0000 7f9b82585700 1 -- 172.21.15.137:0/4194404558 >> 172.21.15.137:0/4194404558 conn(0x7f9b7c0f09a0 msgr2=0x7f9b7c0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:51:03.408 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:03.406+0000 7f9b82585700 1 -- 172.21.15.137:0/4194404558 shutdown_connections 2023-11-15T01:51:03.409 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:03.406+0000 7f9b82585700 1 -- 172.21.15.137:0/4194404558 wait complete. 2023-11-15T01:51:03.409 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:51:04.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:51:03 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:51:02.702058+0000 mgr.smithi055.szwngw (mgr.14164) 41 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:51:04.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:51:03 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:51:03.405112+0000 mon.smithi055 (mon.0) 136 : audit [DBG] from='client.? 172.21.15.137:0/4194404558' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:51:04.782 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:51:04.783 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:51:05.951 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:05.949+0000 7fd85f59e700 1 -- 172.21.15.137:0/3876436902 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd8600f51a0 msgr2=0x7fd8600f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:51:05.951 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:05.949+0000 7fd85f59e700 1 --2- 172.21.15.137:0/3876436902 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd8600f51a0 0x7fd8600f55c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7fd850009fb0 tx=0x7fd850009b30).stop 2023-11-15T01:51:05.951 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:05.949+0000 7fd85f59e700 1 -- 172.21.15.137:0/3876436902 shutdown_connections 2023-11-15T01:51:05.952 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:05.949+0000 7fd85f59e700 1 --2- 172.21.15.137:0/3876436902 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd8600f51a0 0x7fd8600f55c0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:51:05.952 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:05.949+0000 7fd85f59e700 1 -- 172.21.15.137:0/3876436902 >> 172.21.15.137:0/3876436902 conn(0x7fd8600f09a0 msgr2=0x7fd8600f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:51:05.952 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:05.949+0000 7fd85f59e700 1 -- 172.21.15.137:0/3876436902 shutdown_connections 2023-11-15T01:51:05.952 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:05.949+0000 7fd85f59e700 1 -- 172.21.15.137:0/3876436902 wait complete. 2023-11-15T01:51:05.952 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:05.950+0000 7fd85f59e700 1 Processor -- start 2023-11-15T01:51:05.953 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:05.950+0000 7fd85f59e700 1 -- start start 2023-11-15T01:51:05.953 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:05.950+0000 7fd85f59e700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd8600f51a0 0x7fd8601053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:51:05.953 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:05.950+0000 7fd85f59e700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8601058a0 con 0x7fd8600f51a0 2023-11-15T01:51:05.953 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:05.950+0000 7fd85e59c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd8600f51a0 0x7fd8601053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:51:05.953 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:05.950+0000 7fd85e59c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd8600f51a0 0x7fd8601053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:46974/0 (socket says 172.21.15.137:46974) 2023-11-15T01:51:05.954 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:05.950+0000 7fd85e59c700 1 -- 172.21.15.137:0/1625327197 learned_addr learned my addr 172.21.15.137:0/1625327197 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:51:05.954 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:05.950+0000 7fd85e59c700 1 -- 172.21.15.137:0/1625327197 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd850005040 con 0x7fd8600f51a0 2023-11-15T01:51:05.954 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:05.950+0000 7fd85e59c700 1 --2- 172.21.15.137:0/1625327197 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd8600f51a0 0x7fd8601053b0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7fd850013510 tx=0x7fd85000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:51:05.955 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:05.951+0000 7fd84effd700 1 -- 172.21.15.137:0/1625327197 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd850009320 con 0x7fd8600f51a0 2023-11-15T01:51:05.955 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:05.951+0000 7fd85f59e700 1 -- 172.21.15.137:0/1625327197 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd860105aa0 con 0x7fd8600f51a0 2023-11-15T01:51:05.957 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:05.951+0000 7fd84effd700 1 -- 172.21.15.137:0/1625327197 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd850013f00 con 0x7fd8600f51a0 2023-11-15T01:51:05.957 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:05.951+0000 7fd84effd700 1 -- 172.21.15.137:0/1625327197 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd850014dc0 con 0x7fd8600f51a0 2023-11-15T01:51:05.957 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:05.951+0000 7fd85f59e700 1 -- 172.21.15.137:0/1625327197 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd860105f30 con 0x7fd8600f51a0 2023-11-15T01:51:05.958 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:05.951+0000 7fd84effd700 1 -- 172.21.15.137:0/1625327197 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd85000f460 con 0x7fd8600f51a0 2023-11-15T01:51:05.958 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:05.952+0000 7fd84effd700 1 --2- 172.21.15.137:0/1625327197 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fd84802ff60 0x7fd848032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:51:05.958 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:05.952+0000 7fd84effd700 1 -- 172.21.15.137:0/1625327197 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd8500430e0 con 0x7fd8600f51a0 2023-11-15T01:51:05.958 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:05.952+0000 7fd85f59e700 1 -- 172.21.15.137:0/1625327197 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd860045ad0 con 0x7fd8600f51a0 2023-11-15T01:51:05.958 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:05.952+0000 7fd85dd9b700 1 --2- 172.21.15.137:0/1625327197 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fd84802ff60 0x7fd848032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:51:05.959 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:05.952+0000 7fd85dd9b700 1 --2- 172.21.15.137:0/1625327197 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fd84802ff60 0x7fd848032420 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7fd8540096a0 tx=0x7fd854006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:51:05.959 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:05.957+0000 7fd84effd700 1 -- 172.21.15.137:0/1625327197 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd8500283a0 con 0x7fd8600f51a0 2023-11-15T01:51:06.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:51:05 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:51:04.702499+0000 mgr.smithi055.szwngw (mgr.14164) 42 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:51:06.166 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:06.165+0000 7fd85f59e700 1 -- 172.21.15.137:0/1625327197 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd860106de0 con 0x7fd8600f51a0 2023-11-15T01:51:06.167 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:06.166+0000 7fd84effd700 1 -- 172.21.15.137:0/1625327197 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd85002d030 con 0x7fd8600f51a0 2023-11-15T01:51:06.167 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:51:06.167 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:51:06.168 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:06.167+0000 7fd85f59e700 1 -- 172.21.15.137:0/1625327197 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fd84802ff60 msgr2=0x7fd848032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:51:06.168 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:06.167+0000 7fd85f59e700 1 --2- 172.21.15.137:0/1625327197 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fd84802ff60 0x7fd848032420 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7fd8540096a0 tx=0x7fd854006b40).stop 2023-11-15T01:51:06.168 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:06.167+0000 7fd85f59e700 1 -- 172.21.15.137:0/1625327197 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd8600f51a0 msgr2=0x7fd8601053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:51:06.168 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:06.167+0000 7fd85f59e700 1 --2- 172.21.15.137:0/1625327197 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd8600f51a0 0x7fd8601053b0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7fd850013510 tx=0x7fd85000b040).stop 2023-11-15T01:51:06.169 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:06.167+0000 7fd85f59e700 1 -- 172.21.15.137:0/1625327197 shutdown_connections 2023-11-15T01:51:06.169 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:06.167+0000 7fd85f59e700 1 --2- 172.21.15.137:0/1625327197 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fd84802ff60 0x7fd848032420 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:51:06.169 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:06.167+0000 7fd85f59e700 1 --2- 172.21.15.137:0/1625327197 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd8600f51a0 0x7fd8601053b0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:51:06.169 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:06.167+0000 7fd85f59e700 1 -- 172.21.15.137:0/1625327197 >> 172.21.15.137:0/1625327197 conn(0x7fd8600f09a0 msgr2=0x7fd8600f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:51:06.169 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:06.167+0000 7fd85f59e700 1 -- 172.21.15.137:0/1625327197 shutdown_connections 2023-11-15T01:51:06.170 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:06.167+0000 7fd85f59e700 1 -- 172.21.15.137:0/1625327197 wait complete. 2023-11-15T01:51:06.170 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:51:07.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:51:06 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:51:06.166812+0000 mon.smithi055 (mon.0) 137 : audit [DBG] from='client.? 172.21.15.137:0/1625327197' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:51:07.560 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:51:07.561 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:51:08.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:51:07 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:51:06.702848+0000 mgr.smithi055.szwngw (mgr.14164) 43 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:51:08.711 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:08.709+0000 7f24a6f13700 1 -- 172.21.15.137:0/513864093 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f24a00f5330 msgr2=0x7f24a00f5750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:51:08.736 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:08.709+0000 7f24a6f13700 1 --2- 172.21.15.137:0/513864093 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f24a00f5330 0x7f24a00f5750 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f2494009fb0 tx=0x7f2494009b30).stop 2023-11-15T01:51:08.736 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:08.709+0000 7f24a6f13700 1 -- 172.21.15.137:0/513864093 shutdown_connections 2023-11-15T01:51:08.737 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:08.709+0000 7f24a6f13700 1 --2- 172.21.15.137:0/513864093 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f24a00f5330 0x7f24a00f5750 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:51:08.737 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:08.709+0000 7f24a6f13700 1 -- 172.21.15.137:0/513864093 >> 172.21.15.137:0/513864093 conn(0x7f24a00f0a90 msgr2=0x7f24a00f2eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:51:08.737 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:08.709+0000 7f24a6f13700 1 -- 172.21.15.137:0/513864093 shutdown_connections 2023-11-15T01:51:08.737 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:08.709+0000 7f24a6f13700 1 -- 172.21.15.137:0/513864093 wait complete. 2023-11-15T01:51:08.737 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:08.710+0000 7f24a6f13700 1 Processor -- start 2023-11-15T01:51:08.738 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:08.710+0000 7f24a6f13700 1 -- start start 2023-11-15T01:51:08.738 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:08.710+0000 7f24a6f13700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f24a00f5330 0x7f24a01058d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:51:08.738 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:08.710+0000 7f24a6f13700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24a0105dc0 con 0x7f24a00f5330 2023-11-15T01:51:08.738 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:08.710+0000 7f24a4caf700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f24a00f5330 0x7f24a01058d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:51:08.738 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:08.710+0000 7f24a4caf700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f24a00f5330 0x7f24a01058d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:49966/0 (socket says 172.21.15.137:49966) 2023-11-15T01:51:08.739 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:08.710+0000 7f24a4caf700 1 -- 172.21.15.137:0/2098626273 learned_addr learned my addr 172.21.15.137:0/2098626273 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:51:08.740 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:08.710+0000 7f24a4caf700 1 -- 172.21.15.137:0/2098626273 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2494005040 con 0x7f24a00f5330 2023-11-15T01:51:08.741 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:08.711+0000 7f24a4caf700 1 --2- 172.21.15.137:0/2098626273 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f24a00f5330 0x7f24a01058d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f2494013510 tx=0x7f249400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:51:08.741 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:08.711+0000 7f249d7fa700 1 -- 172.21.15.137:0/2098626273 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2494009320 con 0x7f24a00f5330 2023-11-15T01:51:08.741 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:08.711+0000 7f249d7fa700 1 -- 172.21.15.137:0/2098626273 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2494013f00 con 0x7f24a00f5330 2023-11-15T01:51:08.741 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:08.711+0000 7f24a6f13700 1 -- 172.21.15.137:0/2098626273 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24a0105fc0 con 0x7f24a00f5330 2023-11-15T01:51:08.741 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:08.711+0000 7f249d7fa700 1 -- 172.21.15.137:0/2098626273 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2494014d90 con 0x7f24a00f5330 2023-11-15T01:51:08.742 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:08.711+0000 7f24a6f13700 1 -- 172.21.15.137:0/2098626273 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24a0106450 con 0x7f24a00f5330 2023-11-15T01:51:08.742 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:08.712+0000 7f249d7fa700 1 -- 172.21.15.137:0/2098626273 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f24940095f0 con 0x7f24a00f5330 2023-11-15T01:51:08.742 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:08.712+0000 7f24a6f13700 1 -- 172.21.15.137:0/2098626273 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24a0045ad0 con 0x7f24a00f5330 2023-11-15T01:51:08.743 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:08.712+0000 7f249d7fa700 1 --2- 172.21.15.137:0/2098626273 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f249002ff60 0x7f2490032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:51:08.743 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:08.712+0000 7f249d7fa700 1 -- 172.21.15.137:0/2098626273 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2494041000 con 0x7f24a00f5330 2023-11-15T01:51:08.743 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:08.712+0000 7f249ffff700 1 --2- 172.21.15.137:0/2098626273 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f249002ff60 0x7f2490032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:51:08.743 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:08.712+0000 7f249ffff700 1 --2- 172.21.15.137:0/2098626273 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f249002ff60 0x7f2490032420 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f248c0096a0 tx=0x7f248c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:51:08.744 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:08.717+0000 7f249d7fa700 1 -- 172.21.15.137:0/2098626273 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f249401d7f0 con 0x7f24a00f5330 2023-11-15T01:51:08.922 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:08.921+0000 7f24a6f13700 1 -- 172.21.15.137:0/2098626273 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f24a005ea50 con 0x7f24a00f5330 2023-11-15T01:51:08.923 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:08.922+0000 7f249d7fa700 1 -- 172.21.15.137:0/2098626273 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2494010230 con 0x7f24a00f5330 2023-11-15T01:51:08.923 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:51:08.924 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:51:08.924 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:08.924+0000 7f24a6f13700 1 -- 172.21.15.137:0/2098626273 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f249002ff60 msgr2=0x7f2490032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:51:08.925 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:08.924+0000 7f24a6f13700 1 --2- 172.21.15.137:0/2098626273 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f249002ff60 0x7f2490032420 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f248c0096a0 tx=0x7f248c006b40).stop 2023-11-15T01:51:08.925 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:08.924+0000 7f24a6f13700 1 -- 172.21.15.137:0/2098626273 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f24a00f5330 msgr2=0x7f24a01058d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:51:08.925 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:08.924+0000 7f24a6f13700 1 --2- 172.21.15.137:0/2098626273 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f24a00f5330 0x7f24a01058d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f2494013510 tx=0x7f249400b040).stop 2023-11-15T01:51:08.925 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:08.924+0000 7f24a6f13700 1 -- 172.21.15.137:0/2098626273 shutdown_connections 2023-11-15T01:51:08.926 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:08.924+0000 7f24a6f13700 1 --2- 172.21.15.137:0/2098626273 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f249002ff60 0x7f2490032420 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:51:08.927 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:08.924+0000 7f24a6f13700 1 --2- 172.21.15.137:0/2098626273 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f24a00f5330 0x7f24a01058d0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:51:08.927 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:08.924+0000 7f24a6f13700 1 -- 172.21.15.137:0/2098626273 >> 172.21.15.137:0/2098626273 conn(0x7f24a00f0a90 msgr2=0x7f24a00f1530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:51:08.927 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:08.924+0000 7f24a6f13700 1 -- 172.21.15.137:0/2098626273 shutdown_connections 2023-11-15T01:51:08.928 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:08.924+0000 7f24a6f13700 1 -- 172.21.15.137:0/2098626273 wait complete. 2023-11-15T01:51:08.928 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:51:10.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:51:09 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:51:08.703098+0000 mgr.smithi055.szwngw (mgr.14164) 44 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:51:10.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:51:09 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:51:08.923328+0000 mon.smithi055 (mon.0) 138 : audit [DBG] from='client.? 172.21.15.137:0/2098626273' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:51:10.310 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:51:10.311 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:51:11.480 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:11.478+0000 7facb7e88700 1 -- 172.21.15.137:0/481770560 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7facb00f5410 msgr2=0x7facb00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:51:11.481 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:11.478+0000 7facb7e88700 1 --2- 172.21.15.137:0/481770560 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7facb00f5410 0x7facb00f5830 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7faca0009fb0 tx=0x7faca0009b30).stop 2023-11-15T01:51:11.481 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:11.478+0000 7facb7e88700 1 -- 172.21.15.137:0/481770560 shutdown_connections 2023-11-15T01:51:11.481 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:11.478+0000 7facb7e88700 1 --2- 172.21.15.137:0/481770560 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7facb00f5410 0x7facb00f5830 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:51:11.481 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:11.478+0000 7facb7e88700 1 -- 172.21.15.137:0/481770560 >> 172.21.15.137:0/481770560 conn(0x7facb00f0b70 msgr2=0x7facb00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:51:11.482 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:11.478+0000 7facb7e88700 1 -- 172.21.15.137:0/481770560 shutdown_connections 2023-11-15T01:51:11.482 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:11.478+0000 7facb7e88700 1 -- 172.21.15.137:0/481770560 wait complete. 2023-11-15T01:51:11.482 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:11.478+0000 7facb7e88700 1 Processor -- start 2023-11-15T01:51:11.482 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:11.478+0000 7facb7e88700 1 -- start start 2023-11-15T01:51:11.482 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:11.479+0000 7facb7e88700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7facb0106820 0x7facb0106c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:51:11.483 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:11.479+0000 7facb7e88700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facb0107130 con 0x7facb0106820 2023-11-15T01:51:11.483 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:11.479+0000 7facb5c24700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7facb0106820 0x7facb0106c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:51:11.483 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:11.479+0000 7facb5c24700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7facb0106820 0x7facb0106c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:49988/0 (socket says 172.21.15.137:49988) 2023-11-15T01:51:11.483 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:11.479+0000 7facb5c24700 1 -- 172.21.15.137:0/260082377 learned_addr learned my addr 172.21.15.137:0/260082377 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:51:11.483 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:11.479+0000 7facb5c24700 1 -- 172.21.15.137:0/260082377 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faca0005040 con 0x7facb0106820 2023-11-15T01:51:11.484 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:11.479+0000 7facb5c24700 1 --2- 172.21.15.137:0/260082377 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7facb0106820 0x7facb0106c40 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7faca0009f80 tx=0x7faca0009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:51:11.484 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:11.480+0000 7faca67fc700 1 -- 172.21.15.137:0/260082377 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faca00136c0 con 0x7facb0106820 2023-11-15T01:51:11.484 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:11.480+0000 7facb7e88700 1 -- 172.21.15.137:0/260082377 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facb0107330 con 0x7facb0106820 2023-11-15T01:51:11.484 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:11.480+0000 7faca67fc700 1 -- 172.21.15.137:0/260082377 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faca0013e70 con 0x7facb0106820 2023-11-15T01:51:11.484 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:11.480+0000 7faca67fc700 1 -- 172.21.15.137:0/260082377 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faca0014420 con 0x7facb0106820 2023-11-15T01:51:11.485 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:11.480+0000 7facb7e88700 1 -- 172.21.15.137:0/260082377 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facb0107e80 con 0x7facb0106820 2023-11-15T01:51:11.486 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:11.481+0000 7faca67fc700 1 -- 172.21.15.137:0/260082377 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7faca0025060 con 0x7facb0106820 2023-11-15T01:51:11.486 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:11.481+0000 7facb7e88700 1 -- 172.21.15.137:0/260082377 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facb0059100 con 0x7facb0106820 2023-11-15T01:51:11.486 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:11.481+0000 7faca67fc700 1 --2- 172.21.15.137:0/260082377 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fac9c02ff60 0x7fac9c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:51:11.486 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:11.481+0000 7faca67fc700 1 -- 172.21.15.137:0/260082377 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7faca0040e00 con 0x7facb0106820 2023-11-15T01:51:11.486 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:11.481+0000 7facb5423700 1 --2- 172.21.15.137:0/260082377 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fac9c02ff60 0x7fac9c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:51:11.487 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:11.481+0000 7facb5423700 1 --2- 172.21.15.137:0/260082377 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fac9c02ff60 0x7fac9c032420 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7facac0096a0 tx=0x7facac006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:51:11.487 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:11.486+0000 7faca67fc700 1 -- 172.21.15.137:0/260082377 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7faca001d3a0 con 0x7facb0106820 2023-11-15T01:51:11.692 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:11.692+0000 7facb7e88700 1 -- 172.21.15.137:0/260082377 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7facb0045ad0 con 0x7facb0106820 2023-11-15T01:51:11.693 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:11.692+0000 7faca67fc700 1 -- 172.21.15.137:0/260082377 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7faca002d2d0 con 0x7facb0106820 2023-11-15T01:51:11.693 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:51:11.694 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:51:11.698 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:11.694+0000 7facb7e88700 1 -- 172.21.15.137:0/260082377 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fac9c02ff60 msgr2=0x7fac9c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:51:11.698 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:11.694+0000 7facb7e88700 1 --2- 172.21.15.137:0/260082377 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fac9c02ff60 0x7fac9c032420 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7facac0096a0 tx=0x7facac006b40).stop 2023-11-15T01:51:11.698 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:11.694+0000 7facb7e88700 1 -- 172.21.15.137:0/260082377 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7facb0106820 msgr2=0x7facb0106c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:51:11.698 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:11.694+0000 7facb7e88700 1 --2- 172.21.15.137:0/260082377 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7facb0106820 0x7facb0106c40 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7faca0009f80 tx=0x7faca0009b30).stop 2023-11-15T01:51:11.699 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:11.694+0000 7facb7e88700 1 -- 172.21.15.137:0/260082377 shutdown_connections 2023-11-15T01:51:11.699 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:11.694+0000 7facb7e88700 1 --2- 172.21.15.137:0/260082377 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fac9c02ff60 0x7fac9c032420 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:51:11.699 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:11.694+0000 7facb7e88700 1 --2- 172.21.15.137:0/260082377 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7facb0106820 0x7facb0106c40 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:51:11.699 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:11.694+0000 7facb7e88700 1 -- 172.21.15.137:0/260082377 >> 172.21.15.137:0/260082377 conn(0x7facb00f0b70 msgr2=0x7facb005fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:51:11.700 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:11.694+0000 7facb7e88700 1 -- 172.21.15.137:0/260082377 shutdown_connections 2023-11-15T01:51:11.700 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:11.694+0000 7facb7e88700 1 -- 172.21.15.137:0/260082377 wait complete. 2023-11-15T01:51:11.700 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:51:11.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:51:11 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:51:10.703455+0000 mgr.smithi055.szwngw (mgr.14164) 45 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:51:12.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:51:12 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:51:11.693462+0000 mon.smithi055 (mon.0) 139 : audit [DBG] from='client.? 172.21.15.137:0/260082377' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:51:13.099 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:51:13.100 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:51:13.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:51:13 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:51:12.703836+0000 mgr.smithi055.szwngw (mgr.14164) 46 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:51:14.298 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:14.296+0000 7f44b1ae9700 1 -- 172.21.15.137:0/4203714845 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f44ac0f5410 msgr2=0x7f44ac0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:51:14.299 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:14.296+0000 7f44b1ae9700 1 --2- 172.21.15.137:0/4203714845 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f44ac0f5410 0x7f44ac0f5830 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f44a0009fb0 tx=0x7f44a0009b30).stop 2023-11-15T01:51:14.299 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:14.296+0000 7f44b1ae9700 1 -- 172.21.15.137:0/4203714845 shutdown_connections 2023-11-15T01:51:14.299 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:14.296+0000 7f44b1ae9700 1 --2- 172.21.15.137:0/4203714845 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f44ac0f5410 0x7f44ac0f5830 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:51:14.300 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:14.296+0000 7f44b1ae9700 1 -- 172.21.15.137:0/4203714845 >> 172.21.15.137:0/4203714845 conn(0x7f44ac0f0b70 msgr2=0x7f44ac0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:51:14.300 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:14.296+0000 7f44b1ae9700 1 -- 172.21.15.137:0/4203714845 shutdown_connections 2023-11-15T01:51:14.300 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:14.296+0000 7f44b1ae9700 1 -- 172.21.15.137:0/4203714845 wait complete. 2023-11-15T01:51:14.302 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:14.297+0000 7f44b1ae9700 1 Processor -- start 2023-11-15T01:51:14.303 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:14.297+0000 7f44b1ae9700 1 -- start start 2023-11-15T01:51:14.303 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:14.298+0000 7f44b1ae9700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f44ac0f5410 0x7f44ac1055d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:51:14.303 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:14.298+0000 7f44b1ae9700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44ac105ac0 con 0x7f44ac0f5410 2023-11-15T01:51:14.303 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:14.298+0000 7f44ab7fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f44ac0f5410 0x7f44ac1055d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:51:14.304 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:14.298+0000 7f44ab7fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f44ac0f5410 0x7f44ac1055d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:50016/0 (socket says 172.21.15.137:50016) 2023-11-15T01:51:14.304 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:14.298+0000 7f44ab7fe700 1 -- 172.21.15.137:0/1203902142 learned_addr learned my addr 172.21.15.137:0/1203902142 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:51:14.304 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:14.298+0000 7f44ab7fe700 1 -- 172.21.15.137:0/1203902142 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44a0005040 con 0x7f44ac0f5410 2023-11-15T01:51:14.304 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:14.298+0000 7f44ab7fe700 1 --2- 172.21.15.137:0/1203902142 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f44ac0f5410 0x7f44ac1055d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f44a0013510 tx=0x7f44a000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:51:14.304 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:14.299+0000 7f4493fff700 1 -- 172.21.15.137:0/1203902142 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f44a0009320 con 0x7f44ac0f5410 2023-11-15T01:51:14.305 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:14.299+0000 7f44b1ae9700 1 -- 172.21.15.137:0/1203902142 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44ac105cc0 con 0x7f44ac0f5410 2023-11-15T01:51:14.305 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:14.299+0000 7f4493fff700 1 -- 172.21.15.137:0/1203902142 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f44a0013f00 con 0x7f44ac0f5410 2023-11-15T01:51:14.306 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:14.299+0000 7f4493fff700 1 -- 172.21.15.137:0/1203902142 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f44a0014d90 con 0x7f44ac0f5410 2023-11-15T01:51:14.306 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:14.299+0000 7f44b1ae9700 1 -- 172.21.15.137:0/1203902142 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44ac1060d0 con 0x7f44ac0f5410 2023-11-15T01:51:14.306 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:14.300+0000 7f4493fff700 1 -- 172.21.15.137:0/1203902142 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f44a00263a0 con 0x7f44ac0f5410 2023-11-15T01:51:14.306 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:14.300+0000 7f4493fff700 1 --2- 172.21.15.137:0/1203902142 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f44940321c0 0x7f4494034680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:51:14.306 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:14.300+0000 7f4493fff700 1 -- 172.21.15.137:0/1203902142 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f44a00412b0 con 0x7f44ac0f5410 2023-11-15T01:51:14.307 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:14.300+0000 7f44b1ae9700 1 -- 172.21.15.137:0/1203902142 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44ac059100 con 0x7f44ac0f5410 2023-11-15T01:51:14.307 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:14.300+0000 7f44aaffd700 1 --2- 172.21.15.137:0/1203902142 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f44940321c0 0x7f4494034680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:51:14.307 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:14.301+0000 7f44aaffd700 1 --2- 172.21.15.137:0/1203902142 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f44940321c0 0x7f4494034680 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f449c0096a0 tx=0x7f449c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:51:14.308 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:14.305+0000 7f4493fff700 1 -- 172.21.15.137:0/1203902142 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f44a001cb10 con 0x7f44ac0f5410 2023-11-15T01:51:14.512 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:14.511+0000 7f44b1ae9700 1 -- 172.21.15.137:0/1203902142 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f44ac106dd0 con 0x7f44ac0f5410 2023-11-15T01:51:14.513 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:14.511+0000 7f4493fff700 1 -- 172.21.15.137:0/1203902142 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f44a002b030 con 0x7f44ac0f5410 2023-11-15T01:51:14.513 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:51:14.514 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:51:14.515 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:14.513+0000 7f44b1ae9700 1 -- 172.21.15.137:0/1203902142 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f44940321c0 msgr2=0x7f4494034680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:51:14.515 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:14.513+0000 7f44b1ae9700 1 --2- 172.21.15.137:0/1203902142 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f44940321c0 0x7f4494034680 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f449c0096a0 tx=0x7f449c006b40).stop 2023-11-15T01:51:14.515 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:14.513+0000 7f44b1ae9700 1 -- 172.21.15.137:0/1203902142 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f44ac0f5410 msgr2=0x7f44ac1055d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:51:14.515 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:14.513+0000 7f44b1ae9700 1 --2- 172.21.15.137:0/1203902142 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f44ac0f5410 0x7f44ac1055d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f44a0013510 tx=0x7f44a000b040).stop 2023-11-15T01:51:14.516 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:14.513+0000 7f44b1ae9700 1 -- 172.21.15.137:0/1203902142 shutdown_connections 2023-11-15T01:51:14.516 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:14.513+0000 7f44b1ae9700 1 --2- 172.21.15.137:0/1203902142 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f44940321c0 0x7f4494034680 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:51:14.516 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:14.513+0000 7f44b1ae9700 1 --2- 172.21.15.137:0/1203902142 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f44ac0f5410 0x7f44ac1055d0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:51:14.516 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:14.513+0000 7f44b1ae9700 1 -- 172.21.15.137:0/1203902142 >> 172.21.15.137:0/1203902142 conn(0x7f44ac0f0b70 msgr2=0x7f44ac0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:51:14.516 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:14.513+0000 7f44b1ae9700 1 -- 172.21.15.137:0/1203902142 shutdown_connections 2023-11-15T01:51:14.517 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:14.513+0000 7f44b1ae9700 1 -- 172.21.15.137:0/1203902142 wait complete. 2023-11-15T01:51:14.517 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:51:14.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:51:14 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:51:14.513384+0000 mon.smithi055 (mon.0) 140 : audit [DBG] from='client.? 172.21.15.137:0/1203902142' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:51:15.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:51:15 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:51:14.704158+0000 mgr.smithi055.szwngw (mgr.14164) 47 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:51:15.913 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:51:15.914 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:51:17.075 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:17.071+0000 7fe2271a6700 1 -- 172.21.15.137:0/1190724630 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe2200f51a0 msgr2=0x7fe2200f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:51:17.076 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:17.071+0000 7fe2271a6700 1 --2- 172.21.15.137:0/1190724630 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe2200f51a0 0x7fe2200f55c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7fe210009fb0 tx=0x7fe210009b30).stop 2023-11-15T01:51:17.076 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:17.071+0000 7fe2271a6700 1 -- 172.21.15.137:0/1190724630 shutdown_connections 2023-11-15T01:51:17.076 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:17.071+0000 7fe2271a6700 1 --2- 172.21.15.137:0/1190724630 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe2200f51a0 0x7fe2200f55c0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:51:17.076 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:17.072+0000 7fe2271a6700 1 -- 172.21.15.137:0/1190724630 >> 172.21.15.137:0/1190724630 conn(0x7fe2200f09a0 msgr2=0x7fe2200f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:51:17.077 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:17.072+0000 7fe2271a6700 1 -- 172.21.15.137:0/1190724630 shutdown_connections 2023-11-15T01:51:17.077 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:17.072+0000 7fe2271a6700 1 -- 172.21.15.137:0/1190724630 wait complete. 2023-11-15T01:51:17.077 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:17.072+0000 7fe2271a6700 1 Processor -- start 2023-11-15T01:51:17.077 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:17.072+0000 7fe2271a6700 1 -- start start 2023-11-15T01:51:17.077 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:17.072+0000 7fe2271a6700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe2200f51a0 0x7fe220109170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:51:17.078 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:17.072+0000 7fe2271a6700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe220060280 con 0x7fe2200f51a0 2023-11-15T01:51:17.078 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:17.073+0000 7fe2261a4700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe2200f51a0 0x7fe220109170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:51:17.079 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:17.073+0000 7fe2261a4700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe2200f51a0 0x7fe220109170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:50036/0 (socket says 172.21.15.137:50036) 2023-11-15T01:51:17.079 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:17.073+0000 7fe2261a4700 1 -- 172.21.15.137:0/1972861279 learned_addr learned my addr 172.21.15.137:0/1972861279 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:51:17.079 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:17.073+0000 7fe2261a4700 1 -- 172.21.15.137:0/1972861279 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe210005040 con 0x7fe2200f51a0 2023-11-15T01:51:17.079 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:17.073+0000 7fe2261a4700 1 --2- 172.21.15.137:0/1972861279 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe2200f51a0 0x7fe220109170 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7fe210013510 tx=0x7fe21000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:51:17.080 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:17.073+0000 7fe216ffd700 1 -- 172.21.15.137:0/1972861279 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe210009320 con 0x7fe2200f51a0 2023-11-15T01:51:17.080 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:17.073+0000 7fe2271a6700 1 -- 172.21.15.137:0/1972861279 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe220060480 con 0x7fe2200f51a0 2023-11-15T01:51:17.080 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:17.073+0000 7fe216ffd700 1 -- 172.21.15.137:0/1972861279 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe210013f00 con 0x7fe2200f51a0 2023-11-15T01:51:17.080 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:17.073+0000 7fe216ffd700 1 -- 172.21.15.137:0/1972861279 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe210014dc0 con 0x7fe2200f51a0 2023-11-15T01:51:17.080 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:17.073+0000 7fe2271a6700 1 -- 172.21.15.137:0/1972861279 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2201098a0 con 0x7fe2200f51a0 2023-11-15T01:51:17.082 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:17.074+0000 7fe2271a6700 1 -- 172.21.15.137:0/1972861279 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe220045ad0 con 0x7fe2200f51a0 2023-11-15T01:51:17.082 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:17.074+0000 7fe216ffd700 1 -- 172.21.15.137:0/1972861279 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe2100095f0 con 0x7fe2200f51a0 2023-11-15T01:51:17.082 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:17.074+0000 7fe216ffd700 1 --2- 172.21.15.137:0/1972861279 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fe20c02ff60 0x7fe20c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:51:17.083 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:17.074+0000 7fe216ffd700 1 -- 172.21.15.137:0/1972861279 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe210041020 con 0x7fe2200f51a0 2023-11-15T01:51:17.083 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:17.074+0000 7fe2259a3700 1 --2- 172.21.15.137:0/1972861279 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fe20c02ff60 0x7fe20c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:51:17.083 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:17.075+0000 7fe2259a3700 1 --2- 172.21.15.137:0/1972861279 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fe20c02ff60 0x7fe20c032420 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fe21c0096a0 tx=0x7fe21c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:51:17.083 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:17.080+0000 7fe216ffd700 1 -- 172.21.15.137:0/1972861279 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe210014420 con 0x7fe2200f51a0 2023-11-15T01:51:17.287 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:17.285+0000 7fe2271a6700 1 -- 172.21.15.137:0/1972861279 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe2200fabe0 con 0x7fe2200f51a0 2023-11-15T01:51:17.288 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:17.286+0000 7fe216ffd700 1 -- 172.21.15.137:0/1972861279 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe21001c920 con 0x7fe2200f51a0 2023-11-15T01:51:17.288 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:51:17.288 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:51:17.289 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:17.287+0000 7fe2271a6700 1 -- 172.21.15.137:0/1972861279 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fe20c02ff60 msgr2=0x7fe20c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:51:17.289 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:17.287+0000 7fe2271a6700 1 --2- 172.21.15.137:0/1972861279 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fe20c02ff60 0x7fe20c032420 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fe21c0096a0 tx=0x7fe21c006b40).stop 2023-11-15T01:51:17.290 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:17.287+0000 7fe2271a6700 1 -- 172.21.15.137:0/1972861279 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe2200f51a0 msgr2=0x7fe220109170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:51:17.290 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:17.287+0000 7fe2271a6700 1 --2- 172.21.15.137:0/1972861279 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe2200f51a0 0x7fe220109170 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7fe210013510 tx=0x7fe21000b040).stop 2023-11-15T01:51:17.290 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:17.287+0000 7fe2271a6700 1 -- 172.21.15.137:0/1972861279 shutdown_connections 2023-11-15T01:51:17.290 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:17.287+0000 7fe2271a6700 1 --2- 172.21.15.137:0/1972861279 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fe20c02ff60 0x7fe20c032420 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:51:17.290 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:17.287+0000 7fe2271a6700 1 --2- 172.21.15.137:0/1972861279 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe2200f51a0 0x7fe220109170 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:51:17.291 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:17.287+0000 7fe2271a6700 1 -- 172.21.15.137:0/1972861279 >> 172.21.15.137:0/1972861279 conn(0x7fe2200f09a0 msgr2=0x7fe2200f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:51:17.291 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:17.287+0000 7fe2271a6700 1 -- 172.21.15.137:0/1972861279 shutdown_connections 2023-11-15T01:51:17.291 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:17.287+0000 7fe2271a6700 1 -- 172.21.15.137:0/1972861279 wait complete. 2023-11-15T01:51:17.291 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:51:18.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:51:17 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:51:16.704449+0000 mgr.smithi055.szwngw (mgr.14164) 48 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:51:18.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:51:17 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:51:17.287862+0000 mon.smithi055 (mon.0) 141 : audit [DBG] from='client.? 172.21.15.137:0/1972861279' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:51:18.683 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:51:18.683 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:51:19.835 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:19.833+0000 7f4ff8039700 1 -- 172.21.15.137:0/4271089646 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4ff00f5410 msgr2=0x7f4ff00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:51:19.836 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:19.834+0000 7f4ff8039700 1 --2- 172.21.15.137:0/4271089646 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4ff00f5410 0x7f4ff00f5830 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f4fe0009fb0 tx=0x7f4fe0009b30).stop 2023-11-15T01:51:19.836 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:19.834+0000 7f4ff8039700 1 -- 172.21.15.137:0/4271089646 shutdown_connections 2023-11-15T01:51:19.836 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:19.834+0000 7f4ff8039700 1 --2- 172.21.15.137:0/4271089646 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4ff00f5410 0x7f4ff00f5830 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:51:19.837 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:19.834+0000 7f4ff8039700 1 -- 172.21.15.137:0/4271089646 >> 172.21.15.137:0/4271089646 conn(0x7f4ff00f0b70 msgr2=0x7f4ff00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:51:19.837 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:19.834+0000 7f4ff8039700 1 -- 172.21.15.137:0/4271089646 shutdown_connections 2023-11-15T01:51:19.837 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:19.834+0000 7f4ff8039700 1 -- 172.21.15.137:0/4271089646 wait complete. 2023-11-15T01:51:19.840 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:19.834+0000 7f4ff8039700 1 Processor -- start 2023-11-15T01:51:19.840 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:19.834+0000 7f4ff8039700 1 -- start start 2023-11-15T01:51:19.841 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:19.834+0000 7f4ff8039700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4ff00f5410 0x7f4ff00fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:51:19.841 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:19.834+0000 7f4ff8039700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ff00fb820 con 0x7f4ff00f5410 2023-11-15T01:51:19.841 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:19.835+0000 7f4ff5dd5700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4ff00f5410 0x7f4ff00fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:51:19.841 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:19.835+0000 7f4ff5dd5700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4ff00f5410 0x7f4ff00fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:45808/0 (socket says 172.21.15.137:45808) 2023-11-15T01:51:19.842 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:19.835+0000 7f4ff5dd5700 1 -- 172.21.15.137:0/2291964469 learned_addr learned my addr 172.21.15.137:0/2291964469 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:51:19.842 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:19.835+0000 7f4ff5dd5700 1 -- 172.21.15.137:0/2291964469 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fe0005040 con 0x7f4ff00f5410 2023-11-15T01:51:19.842 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:19.835+0000 7f4ff5dd5700 1 --2- 172.21.15.137:0/2291964469 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4ff00f5410 0x7f4ff00fd3a0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f4fe0013510 tx=0x7f4fe000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:51:19.842 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:19.835+0000 7f4fe67fc700 1 -- 172.21.15.137:0/2291964469 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4fe0009320 con 0x7f4ff00f5410 2023-11-15T01:51:19.843 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:19.835+0000 7f4fe67fc700 1 -- 172.21.15.137:0/2291964469 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4fe0013f00 con 0x7f4ff00f5410 2023-11-15T01:51:19.843 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:19.835+0000 7f4ff8039700 1 -- 172.21.15.137:0/2291964469 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ff00fb9c0 con 0x7f4ff00f5410 2023-11-15T01:51:19.843 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:19.835+0000 7f4fe67fc700 1 -- 172.21.15.137:0/2291964469 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4fe0014d90 con 0x7f4ff00f5410 2023-11-15T01:51:19.843 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:19.835+0000 7f4ff8039700 1 -- 172.21.15.137:0/2291964469 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ff00fbe50 con 0x7f4ff00f5410 2023-11-15T01:51:19.844 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:19.836+0000 7f4fe67fc700 1 -- 172.21.15.137:0/2291964469 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4fe00095f0 con 0x7f4ff00f5410 2023-11-15T01:51:19.844 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:19.836+0000 7f4ff8039700 1 -- 172.21.15.137:0/2291964469 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ff0059100 con 0x7f4ff00f5410 2023-11-15T01:51:19.844 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:19.836+0000 7f4fe67fc700 1 --2- 172.21.15.137:0/2291964469 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f4fdc02ff60 0x7f4fdc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:51:19.844 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:19.836+0000 7f4fe67fc700 1 -- 172.21.15.137:0/2291964469 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4fe00410c0 con 0x7f4ff00f5410 2023-11-15T01:51:19.844 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:19.837+0000 7f4ff55d4700 1 --2- 172.21.15.137:0/2291964469 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f4fdc02ff60 0x7f4fdc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:51:19.845 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:19.837+0000 7f4ff55d4700 1 --2- 172.21.15.137:0/2291964469 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f4fdc02ff60 0x7f4fdc032420 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f4fec0096a0 tx=0x7f4fec006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:51:19.845 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:19.842+0000 7f4fe67fc700 1 -- 172.21.15.137:0/2291964469 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4fe0014420 con 0x7f4ff00f5410 2023-11-15T01:51:20.050 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:20.049+0000 7f4ff8039700 1 -- 172.21.15.137:0/2291964469 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4ff00fae60 con 0x7f4ff00f5410 2023-11-15T01:51:20.051 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:20.049+0000 7f4fe67fc700 1 -- 172.21.15.137:0/2291964469 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4fe0010230 con 0x7f4ff00f5410 2023-11-15T01:51:20.051 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:51:20.052 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:51:20.052 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:20.051+0000 7f4ff8039700 1 -- 172.21.15.137:0/2291964469 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f4fdc02ff60 msgr2=0x7f4fdc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:51:20.053 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:20.051+0000 7f4ff8039700 1 --2- 172.21.15.137:0/2291964469 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f4fdc02ff60 0x7f4fdc032420 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f4fec0096a0 tx=0x7f4fec006b40).stop 2023-11-15T01:51:20.053 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:20.051+0000 7f4ff8039700 1 -- 172.21.15.137:0/2291964469 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4ff00f5410 msgr2=0x7f4ff00fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:51:20.053 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:20.051+0000 7f4ff8039700 1 --2- 172.21.15.137:0/2291964469 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4ff00f5410 0x7f4ff00fd3a0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f4fe0013510 tx=0x7f4fe000b040).stop 2023-11-15T01:51:20.053 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:20.051+0000 7f4ff8039700 1 -- 172.21.15.137:0/2291964469 shutdown_connections 2023-11-15T01:51:20.054 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:20.051+0000 7f4ff8039700 1 --2- 172.21.15.137:0/2291964469 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f4fdc02ff60 0x7f4fdc032420 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:51:20.054 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:20.051+0000 7f4ff8039700 1 --2- 172.21.15.137:0/2291964469 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4ff00f5410 0x7f4ff00fd3a0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:51:20.054 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:20.051+0000 7f4ff8039700 1 -- 172.21.15.137:0/2291964469 >> 172.21.15.137:0/2291964469 conn(0x7f4ff00f0b70 msgr2=0x7f4ff005fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:51:20.054 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:20.051+0000 7f4ff8039700 1 -- 172.21.15.137:0/2291964469 shutdown_connections 2023-11-15T01:51:20.054 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:20.051+0000 7f4ff8039700 1 -- 172.21.15.137:0/2291964469 wait complete. 2023-11-15T01:51:20.055 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:51:20.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:51:19 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:51:18.704765+0000 mgr.smithi055.szwngw (mgr.14164) 49 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:51:21.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:51:20 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:51:20.051261+0000 mon.smithi055 (mon.0) 142 : audit [DBG] from='client.? 172.21.15.137:0/2291964469' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:51:21.447 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:51:21.448 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:51:22.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:51:21 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:51:20.705112+0000 mgr.smithi055.szwngw (mgr.14164) 50 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:51:22.642 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:22.639+0000 7fbc91c06700 1 -- 172.21.15.137:0/3497142519 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbc8c0f5410 msgr2=0x7fbc8c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:51:22.642 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:22.639+0000 7fbc91c06700 1 --2- 172.21.15.137:0/3497142519 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbc8c0f5410 0x7fbc8c0f5830 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7fbc7c009fb0 tx=0x7fbc7c009b30).stop 2023-11-15T01:51:22.643 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:22.639+0000 7fbc91c06700 1 -- 172.21.15.137:0/3497142519 shutdown_connections 2023-11-15T01:51:22.643 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:22.639+0000 7fbc91c06700 1 --2- 172.21.15.137:0/3497142519 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbc8c0f5410 0x7fbc8c0f5830 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:51:22.643 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:22.639+0000 7fbc91c06700 1 -- 172.21.15.137:0/3497142519 >> 172.21.15.137:0/3497142519 conn(0x7fbc8c0f0b70 msgr2=0x7fbc8c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:51:22.643 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:22.640+0000 7fbc91c06700 1 -- 172.21.15.137:0/3497142519 shutdown_connections 2023-11-15T01:51:22.644 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:22.640+0000 7fbc91c06700 1 -- 172.21.15.137:0/3497142519 wait complete. 2023-11-15T01:51:22.644 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:22.640+0000 7fbc91c06700 1 Processor -- start 2023-11-15T01:51:22.644 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:22.640+0000 7fbc91c06700 1 -- start start 2023-11-15T01:51:22.644 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:22.640+0000 7fbc91c06700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbc8c0f5410 0x7fbc8c105600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:51:22.644 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:22.640+0000 7fbc91c06700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc8c105af0 con 0x7fbc8c0f5410 2023-11-15T01:51:22.645 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:22.641+0000 7fbc8b7fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbc8c0f5410 0x7fbc8c105600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:51:22.645 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:22.641+0000 7fbc8b7fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbc8c0f5410 0x7fbc8c105600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:45836/0 (socket says 172.21.15.137:45836) 2023-11-15T01:51:22.645 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:22.641+0000 7fbc8b7fe700 1 -- 172.21.15.137:0/1591843644 learned_addr learned my addr 172.21.15.137:0/1591843644 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:51:22.645 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:22.641+0000 7fbc8b7fe700 1 -- 172.21.15.137:0/1591843644 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc7c005040 con 0x7fbc8c0f5410 2023-11-15T01:51:22.645 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:22.641+0000 7fbc8b7fe700 1 --2- 172.21.15.137:0/1591843644 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbc8c0f5410 0x7fbc8c105600 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7fbc7c013510 tx=0x7fbc7c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:51:22.646 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:22.641+0000 7fbc73fff700 1 -- 172.21.15.137:0/1591843644 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbc7c009320 con 0x7fbc8c0f5410 2023-11-15T01:51:22.646 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:22.641+0000 7fbc91c06700 1 -- 172.21.15.137:0/1591843644 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc8c105cf0 con 0x7fbc8c0f5410 2023-11-15T01:51:22.646 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:22.641+0000 7fbc73fff700 1 -- 172.21.15.137:0/1591843644 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbc7c013f00 con 0x7fbc8c0f5410 2023-11-15T01:51:22.646 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:22.641+0000 7fbc73fff700 1 -- 172.21.15.137:0/1591843644 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbc7c014d90 con 0x7fbc8c0f5410 2023-11-15T01:51:22.646 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:22.641+0000 7fbc91c06700 1 -- 172.21.15.137:0/1591843644 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc8c106180 con 0x7fbc8c0f5410 2023-11-15T01:51:22.647 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:22.642+0000 7fbc73fff700 1 -- 172.21.15.137:0/1591843644 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbc7c0095f0 con 0x7fbc8c0f5410 2023-11-15T01:51:22.648 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:22.642+0000 7fbc91c06700 1 -- 172.21.15.137:0/1591843644 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc8c045ad0 con 0x7fbc8c0f5410 2023-11-15T01:51:22.648 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:22.642+0000 7fbc73fff700 1 --2- 172.21.15.137:0/1591843644 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fbc7402ff60 0x7fbc74032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:51:22.648 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:22.642+0000 7fbc73fff700 1 -- 172.21.15.137:0/1591843644 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbc7c040fe0 con 0x7fbc8c0f5410 2023-11-15T01:51:22.648 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:22.642+0000 7fbc8affd700 1 --2- 172.21.15.137:0/1591843644 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fbc7402ff60 0x7fbc74032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:51:22.649 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:22.643+0000 7fbc8affd700 1 --2- 172.21.15.137:0/1591843644 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fbc7402ff60 0x7fbc74032420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fbc800096a0 tx=0x7fbc80006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:51:22.649 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:22.647+0000 7fbc73fff700 1 -- 172.21.15.137:0/1591843644 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbc7c01d7f0 con 0x7fbc8c0f5410 2023-11-15T01:51:22.855 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:22.854+0000 7fbc91c06700 1 -- 172.21.15.137:0/1591843644 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbc8c0fae60 con 0x7fbc8c0f5410 2023-11-15T01:51:22.855 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:22.855+0000 7fbc73fff700 1 -- 172.21.15.137:0/1591843644 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbc7c010230 con 0x7fbc8c0f5410 2023-11-15T01:51:22.856 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:51:22.856 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:51:22.857 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:22.856+0000 7fbc91c06700 1 -- 172.21.15.137:0/1591843644 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fbc7402ff60 msgr2=0x7fbc74032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:51:22.857 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:22.856+0000 7fbc91c06700 1 --2- 172.21.15.137:0/1591843644 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fbc7402ff60 0x7fbc74032420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fbc800096a0 tx=0x7fbc80006b40).stop 2023-11-15T01:51:22.857 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:22.856+0000 7fbc91c06700 1 -- 172.21.15.137:0/1591843644 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbc8c0f5410 msgr2=0x7fbc8c105600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:51:22.858 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:22.856+0000 7fbc91c06700 1 --2- 172.21.15.137:0/1591843644 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbc8c0f5410 0x7fbc8c105600 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7fbc7c013510 tx=0x7fbc7c00b040).stop 2023-11-15T01:51:22.858 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:22.856+0000 7fbc91c06700 1 -- 172.21.15.137:0/1591843644 shutdown_connections 2023-11-15T01:51:22.858 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:22.856+0000 7fbc91c06700 1 --2- 172.21.15.137:0/1591843644 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fbc7402ff60 0x7fbc74032420 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:51:22.858 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:22.856+0000 7fbc91c06700 1 --2- 172.21.15.137:0/1591843644 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbc8c0f5410 0x7fbc8c105600 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:51:22.858 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:22.856+0000 7fbc91c06700 1 -- 172.21.15.137:0/1591843644 >> 172.21.15.137:0/1591843644 conn(0x7fbc8c0f0b70 msgr2=0x7fbc8c0f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:51:22.859 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:22.856+0000 7fbc91c06700 1 -- 172.21.15.137:0/1591843644 shutdown_connections 2023-11-15T01:51:22.859 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:22.856+0000 7fbc91c06700 1 -- 172.21.15.137:0/1591843644 wait complete. 2023-11-15T01:51:22.859 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:51:24.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:51:23 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:51:22.705485+0000 mgr.smithi055.szwngw (mgr.14164) 51 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:51:24.079 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:51:23 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:51:22.855671+0000 mon.smithi055 (mon.0) 143 : audit [DBG] from='client.? 172.21.15.137:0/1591843644' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:51:24.263 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:51:24.264 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:51:25.438 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:25.436+0000 7fd37ce70700 1 -- 172.21.15.137:0/1976458254 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd3780f5410 msgr2=0x7fd3780f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:51:25.439 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:25.436+0000 7fd37ce70700 1 --2- 172.21.15.137:0/1976458254 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd3780f5410 0x7fd3780f5830 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7fd360009fb0 tx=0x7fd360009b30).stop 2023-11-15T01:51:25.439 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:25.436+0000 7fd37ce70700 1 -- 172.21.15.137:0/1976458254 shutdown_connections 2023-11-15T01:51:25.439 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:25.436+0000 7fd37ce70700 1 --2- 172.21.15.137:0/1976458254 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd3780f5410 0x7fd3780f5830 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:51:25.439 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:25.436+0000 7fd37ce70700 1 -- 172.21.15.137:0/1976458254 >> 172.21.15.137:0/1976458254 conn(0x7fd3780f0b70 msgr2=0x7fd3780f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:51:25.440 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:25.436+0000 7fd37ce70700 1 -- 172.21.15.137:0/1976458254 shutdown_connections 2023-11-15T01:51:25.440 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:25.436+0000 7fd37ce70700 1 -- 172.21.15.137:0/1976458254 wait complete. 2023-11-15T01:51:25.440 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:25.437+0000 7fd37ce70700 1 Processor -- start 2023-11-15T01:51:25.440 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:25.437+0000 7fd37ce70700 1 -- start start 2023-11-15T01:51:25.440 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:25.437+0000 7fd37ce70700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd3780f5410 0x7fd378106710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:51:25.441 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:25.437+0000 7fd37ce70700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd378106c00 con 0x7fd3780f5410 2023-11-15T01:51:25.441 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:25.437+0000 7fd37659c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd3780f5410 0x7fd378106710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:51:25.441 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:25.437+0000 7fd37659c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd3780f5410 0x7fd378106710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:45856/0 (socket says 172.21.15.137:45856) 2023-11-15T01:51:25.441 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:25.437+0000 7fd37659c700 1 -- 172.21.15.137:0/3332651641 learned_addr learned my addr 172.21.15.137:0/3332651641 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:51:25.441 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:25.437+0000 7fd37659c700 1 -- 172.21.15.137:0/3332651641 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd360005040 con 0x7fd3780f5410 2023-11-15T01:51:25.442 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:25.438+0000 7fd37659c700 1 --2- 172.21.15.137:0/3332651641 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd3780f5410 0x7fd378106710 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7fd360013510 tx=0x7fd36000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:51:25.443 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:25.438+0000 7fd36effd700 1 -- 172.21.15.137:0/3332651641 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd360009320 con 0x7fd3780f5410 2023-11-15T01:51:25.443 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:25.438+0000 7fd37ce70700 1 -- 172.21.15.137:0/3332651641 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd378106e00 con 0x7fd3780f5410 2023-11-15T01:51:25.444 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:25.438+0000 7fd36effd700 1 -- 172.21.15.137:0/3332651641 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd360013f00 con 0x7fd3780f5410 2023-11-15T01:51:25.444 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:25.438+0000 7fd36effd700 1 -- 172.21.15.137:0/3332651641 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd360014d90 con 0x7fd3780f5410 2023-11-15T01:51:25.444 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:25.438+0000 7fd37ce70700 1 -- 172.21.15.137:0/3332651641 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd378107290 con 0x7fd3780f5410 2023-11-15T01:51:25.444 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:25.439+0000 7fd36effd700 1 -- 172.21.15.137:0/3332651641 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd3600095f0 con 0x7fd3780f5410 2023-11-15T01:51:25.445 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:25.439+0000 7fd37ce70700 1 -- 172.21.15.137:0/3332651641 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd378045ad0 con 0x7fd3780f5410 2023-11-15T01:51:25.445 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:25.439+0000 7fd36effd700 1 --2- 172.21.15.137:0/3332651641 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fd36402ff60 0x7fd364032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:51:25.445 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:25.439+0000 7fd36effd700 1 -- 172.21.15.137:0/3332651641 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd360041000 con 0x7fd3780f5410 2023-11-15T01:51:25.445 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:25.439+0000 7fd375d9b700 1 --2- 172.21.15.137:0/3332651641 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fd36402ff60 0x7fd364032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:51:25.445 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:25.439+0000 7fd375d9b700 1 --2- 172.21.15.137:0/3332651641 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fd36402ff60 0x7fd364032420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7fd3680096a0 tx=0x7fd368006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:51:25.446 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:25.444+0000 7fd36effd700 1 -- 172.21.15.137:0/3332651641 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd360014420 con 0x7fd3780f5410 2023-11-15T01:51:25.652 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:25.652+0000 7fd37ce70700 1 -- 172.21.15.137:0/3332651641 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd378108140 con 0x7fd3780f5410 2023-11-15T01:51:25.653 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:25.652+0000 7fd36effd700 1 -- 172.21.15.137:0/3332651641 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd360010230 con 0x7fd3780f5410 2023-11-15T01:51:25.654 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:51:25.654 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:51:25.655 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:25.654+0000 7fd37ce70700 1 -- 172.21.15.137:0/3332651641 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fd36402ff60 msgr2=0x7fd364032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:51:25.655 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:25.654+0000 7fd37ce70700 1 --2- 172.21.15.137:0/3332651641 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fd36402ff60 0x7fd364032420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7fd3680096a0 tx=0x7fd368006b40).stop 2023-11-15T01:51:25.655 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:25.654+0000 7fd37ce70700 1 -- 172.21.15.137:0/3332651641 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd3780f5410 msgr2=0x7fd378106710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:51:25.655 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:25.654+0000 7fd37ce70700 1 --2- 172.21.15.137:0/3332651641 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd3780f5410 0x7fd378106710 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7fd360013510 tx=0x7fd36000b040).stop 2023-11-15T01:51:25.656 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:25.654+0000 7fd37ce70700 1 -- 172.21.15.137:0/3332651641 shutdown_connections 2023-11-15T01:51:25.656 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:25.654+0000 7fd37ce70700 1 --2- 172.21.15.137:0/3332651641 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fd36402ff60 0x7fd364032420 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:51:25.656 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:25.654+0000 7fd37ce70700 1 --2- 172.21.15.137:0/3332651641 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd3780f5410 0x7fd378106710 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:51:25.656 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:25.654+0000 7fd37ce70700 1 -- 172.21.15.137:0/3332651641 >> 172.21.15.137:0/3332651641 conn(0x7fd3780f0b70 msgr2=0x7fd37805fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:51:25.656 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:25.654+0000 7fd37ce70700 1 -- 172.21.15.137:0/3332651641 shutdown_connections 2023-11-15T01:51:25.657 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:25.654+0000 7fd37ce70700 1 -- 172.21.15.137:0/3332651641 wait complete. 2023-11-15T01:51:25.657 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:51:26.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:51:25 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:51:24.705890+0000 mgr.smithi055.szwngw (mgr.14164) 52 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:51:26.079 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:51:25 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:51:25.653412+0000 mon.smithi055 (mon.0) 144 : audit [DBG] from='client.? 172.21.15.137:0/3332651641' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:51:27.065 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:51:27.066 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:51:28.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:51:27 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:51:26.706185+0000 mgr.smithi055.szwngw (mgr.14164) 53 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:51:28.205 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:28.202+0000 7feb08d7b700 1 -- 172.21.15.137:0/760254199 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feb040f53f0 msgr2=0x7feb040f5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:51:28.205 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:28.202+0000 7feb08d7b700 1 --2- 172.21.15.137:0/760254199 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feb040f53f0 0x7feb040f5810 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7feaf4009fb0 tx=0x7feaf4009b30).stop 2023-11-15T01:51:28.205 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:28.203+0000 7feb08d7b700 1 -- 172.21.15.137:0/760254199 shutdown_connections 2023-11-15T01:51:28.206 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:28.203+0000 7feb08d7b700 1 --2- 172.21.15.137:0/760254199 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feb040f53f0 0x7feb040f5810 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:51:28.206 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:28.203+0000 7feb08d7b700 1 -- 172.21.15.137:0/760254199 >> 172.21.15.137:0/760254199 conn(0x7feb040f0b70 msgr2=0x7feb040f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:51:28.206 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:28.203+0000 7feb08d7b700 1 -- 172.21.15.137:0/760254199 shutdown_connections 2023-11-15T01:51:28.206 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:28.203+0000 7feb08d7b700 1 -- 172.21.15.137:0/760254199 wait complete. 2023-11-15T01:51:28.207 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:28.203+0000 7feb08d7b700 1 Processor -- start 2023-11-15T01:51:28.207 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:28.203+0000 7feb08d7b700 1 -- start start 2023-11-15T01:51:28.207 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:28.203+0000 7feb08d7b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feb040f53f0 0x7feb04109400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:51:28.207 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:28.203+0000 7feb08d7b700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb04060220 con 0x7feb040f53f0 2023-11-15T01:51:28.207 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:28.204+0000 7feb0259c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feb040f53f0 0x7feb04109400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:51:28.208 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:28.204+0000 7feb0259c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feb040f53f0 0x7feb04109400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:45870/0 (socket says 172.21.15.137:45870) 2023-11-15T01:51:28.208 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:28.204+0000 7feb0259c700 1 -- 172.21.15.137:0/3690892909 learned_addr learned my addr 172.21.15.137:0/3690892909 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:51:28.208 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:28.204+0000 7feb0259c700 1 -- 172.21.15.137:0/3690892909 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaf4005040 con 0x7feb040f53f0 2023-11-15T01:51:28.208 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:28.204+0000 7feb0259c700 1 --2- 172.21.15.137:0/3690892909 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feb040f53f0 0x7feb04109400 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7feaf4013510 tx=0x7feaf400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:51:28.208 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:28.204+0000 7feafaffd700 1 -- 172.21.15.137:0/3690892909 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feaf4009320 con 0x7feb040f53f0 2023-11-15T01:51:28.209 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:28.204+0000 7feb08d7b700 1 -- 172.21.15.137:0/3690892909 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb04060420 con 0x7feb040f53f0 2023-11-15T01:51:28.209 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:28.204+0000 7feafaffd700 1 -- 172.21.15.137:0/3690892909 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7feaf4013f00 con 0x7feb040f53f0 2023-11-15T01:51:28.209 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:28.204+0000 7feafaffd700 1 -- 172.21.15.137:0/3690892909 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feaf4014d90 con 0x7feb040f53f0 2023-11-15T01:51:28.209 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:28.204+0000 7feb08d7b700 1 -- 172.21.15.137:0/3690892909 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb04109ab0 con 0x7feb040f53f0 2023-11-15T01:51:28.210 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:28.205+0000 7feafaffd700 1 -- 172.21.15.137:0/3690892909 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7feaf40263a0 con 0x7feb040f53f0 2023-11-15T01:51:28.210 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:28.205+0000 7feafaffd700 1 --2- 172.21.15.137:0/3690892909 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7feaf002ff60 0x7feaf0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:51:28.210 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:28.205+0000 7feafaffd700 1 -- 172.21.15.137:0/3690892909 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7feaf4040ff0 con 0x7feb040f53f0 2023-11-15T01:51:28.211 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:28.205+0000 7feb08d7b700 1 -- 172.21.15.137:0/3690892909 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb04059100 con 0x7feb040f53f0 2023-11-15T01:51:28.211 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:28.205+0000 7feb01d9b700 1 --2- 172.21.15.137:0/3690892909 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7feaf002ff60 0x7feaf0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:51:28.211 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:28.206+0000 7feb01d9b700 1 --2- 172.21.15.137:0/3690892909 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7feaf002ff60 0x7feaf0032420 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7feaec0096a0 tx=0x7feaec006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:51:28.211 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:28.210+0000 7feafaffd700 1 -- 172.21.15.137:0/3690892909 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7feaf4012920 con 0x7feb040f53f0 2023-11-15T01:51:28.419 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:28.418+0000 7feb08d7b700 1 -- 172.21.15.137:0/3690892909 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7feb040fae40 con 0x7feb040f53f0 2023-11-15T01:51:28.420 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:28.419+0000 7feafaffd700 1 -- 172.21.15.137:0/3690892909 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7feaf402b020 con 0x7feb040f53f0 2023-11-15T01:51:28.420 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:51:28.420 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:51:28.421 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:28.420+0000 7feb08d7b700 1 -- 172.21.15.137:0/3690892909 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7feaf002ff60 msgr2=0x7feaf0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:51:28.421 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:28.420+0000 7feb08d7b700 1 --2- 172.21.15.137:0/3690892909 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7feaf002ff60 0x7feaf0032420 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7feaec0096a0 tx=0x7feaec006b40).stop 2023-11-15T01:51:28.422 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:28.420+0000 7feb08d7b700 1 -- 172.21.15.137:0/3690892909 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feb040f53f0 msgr2=0x7feb04109400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:51:28.422 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:28.420+0000 7feb08d7b700 1 --2- 172.21.15.137:0/3690892909 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feb040f53f0 0x7feb04109400 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7feaf4013510 tx=0x7feaf400b040).stop 2023-11-15T01:51:28.422 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:28.420+0000 7feb08d7b700 1 -- 172.21.15.137:0/3690892909 shutdown_connections 2023-11-15T01:51:28.422 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:28.420+0000 7feb08d7b700 1 --2- 172.21.15.137:0/3690892909 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7feaf002ff60 0x7feaf0032420 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:51:28.422 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:28.420+0000 7feb08d7b700 1 --2- 172.21.15.137:0/3690892909 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feb040f53f0 0x7feb04109400 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:51:28.423 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:28.420+0000 7feb08d7b700 1 -- 172.21.15.137:0/3690892909 >> 172.21.15.137:0/3690892909 conn(0x7feb040f0b70 msgr2=0x7feb040f1640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:51:28.423 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:28.420+0000 7feb08d7b700 1 -- 172.21.15.137:0/3690892909 shutdown_connections 2023-11-15T01:51:28.423 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:28.420+0000 7feb08d7b700 1 -- 172.21.15.137:0/3690892909 wait complete. 2023-11-15T01:51:28.423 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:51:29.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:51:28 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:51:28.420080+0000 mon.smithi055 (mon.0) 145 : audit [DBG] from='client.? 172.21.15.137:0/3690892909' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:51:29.823 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:51:29.824 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:51:30.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:51:29 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:51:28.706528+0000 mgr.smithi055.szwngw (mgr.14164) 54 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:51:31.003 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:31.002+0000 7faa69bc0700 1 -- 172.21.15.137:0/121823904 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faa640f51a0 msgr2=0x7faa640f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:51:31.003 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:31.002+0000 7faa69bc0700 1 --2- 172.21.15.137:0/121823904 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faa640f51a0 0x7faa640f55c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7faa58009fb0 tx=0x7faa58009b30).stop 2023-11-15T01:51:31.004 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:31.002+0000 7faa69bc0700 1 -- 172.21.15.137:0/121823904 shutdown_connections 2023-11-15T01:51:31.004 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:31.002+0000 7faa69bc0700 1 --2- 172.21.15.137:0/121823904 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faa640f51a0 0x7faa640f55c0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:51:31.004 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:31.002+0000 7faa69bc0700 1 -- 172.21.15.137:0/121823904 >> 172.21.15.137:0/121823904 conn(0x7faa640f09a0 msgr2=0x7faa640f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:51:31.004 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:31.002+0000 7faa69bc0700 1 -- 172.21.15.137:0/121823904 shutdown_connections 2023-11-15T01:51:31.004 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:31.002+0000 7faa69bc0700 1 -- 172.21.15.137:0/121823904 wait complete. 2023-11-15T01:51:31.005 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:31.002+0000 7faa69bc0700 1 Processor -- start 2023-11-15T01:51:31.005 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:31.002+0000 7faa69bc0700 1 -- start start 2023-11-15T01:51:31.005 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:31.003+0000 7faa69bc0700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faa640f51a0 0x7faa641064c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:51:31.005 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:31.003+0000 7faa69bc0700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa641069b0 con 0x7faa640f51a0 2023-11-15T01:51:31.007 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:31.003+0000 7faa68bbe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faa640f51a0 0x7faa641064c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:51:31.007 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:31.003+0000 7faa68bbe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faa640f51a0 0x7faa641064c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:39278/0 (socket says 172.21.15.137:39278) 2023-11-15T01:51:31.007 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:31.003+0000 7faa68bbe700 1 -- 172.21.15.137:0/3553016614 learned_addr learned my addr 172.21.15.137:0/3553016614 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:51:31.007 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:31.003+0000 7faa68bbe700 1 -- 172.21.15.137:0/3553016614 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa58005040 con 0x7faa640f51a0 2023-11-15T01:51:31.008 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:31.003+0000 7faa68bbe700 1 --2- 172.21.15.137:0/3553016614 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faa640f51a0 0x7faa641064c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7faa58013510 tx=0x7faa5800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:51:31.008 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:31.004+0000 7faa617fa700 1 -- 172.21.15.137:0/3553016614 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faa58009320 con 0x7faa640f51a0 2023-11-15T01:51:31.008 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:31.004+0000 7faa69bc0700 1 -- 172.21.15.137:0/3553016614 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa64106bb0 con 0x7faa640f51a0 2023-11-15T01:51:31.008 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:31.004+0000 7faa617fa700 1 -- 172.21.15.137:0/3553016614 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faa58013f00 con 0x7faa640f51a0 2023-11-15T01:51:31.008 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:31.004+0000 7faa617fa700 1 -- 172.21.15.137:0/3553016614 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faa58014dc0 con 0x7faa640f51a0 2023-11-15T01:51:31.009 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:31.004+0000 7faa69bc0700 1 -- 172.21.15.137:0/3553016614 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa64107040 con 0x7faa640f51a0 2023-11-15T01:51:31.009 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:31.005+0000 7faa617fa700 1 -- 172.21.15.137:0/3553016614 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7faa580095f0 con 0x7faa640f51a0 2023-11-15T01:51:31.009 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:31.005+0000 7faa69bc0700 1 -- 172.21.15.137:0/3553016614 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa64045ad0 con 0x7faa640f51a0 2023-11-15T01:51:31.009 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:31.005+0000 7faa617fa700 1 --2- 172.21.15.137:0/3553016614 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7faa5402ff60 0x7faa54032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:51:31.009 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:31.005+0000 7faa617fa700 1 -- 172.21.15.137:0/3553016614 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7faa58041040 con 0x7faa640f51a0 2023-11-15T01:51:31.010 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:31.005+0000 7faa63fff700 1 --2- 172.21.15.137:0/3553016614 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7faa5402ff60 0x7faa54032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:51:31.010 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:31.005+0000 7faa63fff700 1 --2- 172.21.15.137:0/3553016614 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7faa5402ff60 0x7faa54032420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7faa500096a0 tx=0x7faa50006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:51:31.011 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:31.010+0000 7faa617fa700 1 -- 172.21.15.137:0/3553016614 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7faa58014420 con 0x7faa640f51a0 2023-11-15T01:51:31.217 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:31.216+0000 7faa69bc0700 1 -- 172.21.15.137:0/3553016614 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7faa64107ef0 con 0x7faa640f51a0 2023-11-15T01:51:31.218 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:31.217+0000 7faa617fa700 1 -- 172.21.15.137:0/3553016614 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7faa58010230 con 0x7faa640f51a0 2023-11-15T01:51:31.218 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:51:31.218 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:51:31.219 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:31.219+0000 7faa69bc0700 1 -- 172.21.15.137:0/3553016614 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7faa5402ff60 msgr2=0x7faa54032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:51:31.219 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:31.219+0000 7faa69bc0700 1 --2- 172.21.15.137:0/3553016614 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7faa5402ff60 0x7faa54032420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7faa500096a0 tx=0x7faa50006b40).stop 2023-11-15T01:51:31.220 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:31.219+0000 7faa69bc0700 1 -- 172.21.15.137:0/3553016614 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faa640f51a0 msgr2=0x7faa641064c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:51:31.220 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:31.219+0000 7faa69bc0700 1 --2- 172.21.15.137:0/3553016614 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faa640f51a0 0x7faa641064c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7faa58013510 tx=0x7faa5800b040).stop 2023-11-15T01:51:31.220 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:31.219+0000 7faa69bc0700 1 -- 172.21.15.137:0/3553016614 shutdown_connections 2023-11-15T01:51:31.220 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:31.219+0000 7faa69bc0700 1 --2- 172.21.15.137:0/3553016614 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7faa5402ff60 0x7faa54032420 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:51:31.220 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:31.219+0000 7faa69bc0700 1 --2- 172.21.15.137:0/3553016614 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faa640f51a0 0x7faa641064c0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:51:31.221 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:31.219+0000 7faa69bc0700 1 -- 172.21.15.137:0/3553016614 >> 172.21.15.137:0/3553016614 conn(0x7faa640f09a0 msgr2=0x7faa640f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:51:31.221 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:31.219+0000 7faa69bc0700 1 -- 172.21.15.137:0/3553016614 shutdown_connections 2023-11-15T01:51:31.221 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:31.219+0000 7faa69bc0700 1 -- 172.21.15.137:0/3553016614 wait complete. 2023-11-15T01:51:31.221 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:51:31.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:51:31 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:51:30.706926+0000 mgr.smithi055.szwngw (mgr.14164) 55 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:51:31.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:51:31 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:51:31.218026+0000 mon.smithi055 (mon.0) 146 : audit [DBG] from='client.? 172.21.15.137:0/3553016614' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:51:32.608 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:51:32.609 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:51:33.782 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:33.781+0000 7f5ba5160700 1 -- 172.21.15.137:0/3369790117 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5ba00f6510 msgr2=0x7f5ba00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:51:33.783 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:33.781+0000 7f5ba5160700 1 --2- 172.21.15.137:0/3369790117 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5ba00f6510 0x7f5ba00f6930 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f5b94009fb0 tx=0x7f5b94009b30).stop 2023-11-15T01:51:33.783 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:33.781+0000 7f5ba5160700 1 -- 172.21.15.137:0/3369790117 shutdown_connections 2023-11-15T01:51:33.783 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:33.781+0000 7f5ba5160700 1 --2- 172.21.15.137:0/3369790117 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5ba00f6510 0x7f5ba00f6930 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:51:33.783 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:33.781+0000 7f5ba5160700 1 -- 172.21.15.137:0/3369790117 >> 172.21.15.137:0/3369790117 conn(0x7f5ba00f1c70 msgr2=0x7f5ba00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:51:33.783 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:33.781+0000 7f5ba5160700 1 -- 172.21.15.137:0/3369790117 shutdown_connections 2023-11-15T01:51:33.784 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:33.781+0000 7f5ba5160700 1 -- 172.21.15.137:0/3369790117 wait complete. 2023-11-15T01:51:33.784 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:33.782+0000 7f5ba5160700 1 Processor -- start 2023-11-15T01:51:33.784 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:33.782+0000 7f5ba5160700 1 -- start start 2023-11-15T01:51:33.784 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:33.782+0000 7f5ba5160700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5ba00f6510 0x7f5ba0106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:51:33.784 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:33.782+0000 7f5ba5160700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ba0106c20 con 0x7f5ba00f6510 2023-11-15T01:51:33.786 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:33.782+0000 7f5b9ed9d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5ba00f6510 0x7f5ba0106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:51:33.786 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:33.782+0000 7f5b9ed9d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5ba00f6510 0x7f5ba0106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:39300/0 (socket says 172.21.15.137:39300) 2023-11-15T01:51:33.786 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:33.782+0000 7f5b9ed9d700 1 -- 172.21.15.137:0/3905430409 learned_addr learned my addr 172.21.15.137:0/3905430409 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:51:33.787 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:33.782+0000 7f5b9ed9d700 1 -- 172.21.15.137:0/3905430409 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b94005040 con 0x7f5ba00f6510 2023-11-15T01:51:33.787 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:33.782+0000 7f5b9ed9d700 1 --2- 172.21.15.137:0/3905430409 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5ba00f6510 0x7f5ba0106730 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f5b94013510 tx=0x7f5b9400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:51:33.787 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:33.783+0000 7f5b877fe700 1 -- 172.21.15.137:0/3905430409 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b94009320 con 0x7f5ba00f6510 2023-11-15T01:51:33.787 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:33.783+0000 7f5ba5160700 1 -- 172.21.15.137:0/3905430409 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ba0106e20 con 0x7f5ba00f6510 2023-11-15T01:51:33.787 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:33.783+0000 7f5b877fe700 1 -- 172.21.15.137:0/3905430409 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5b94013f00 con 0x7f5ba00f6510 2023-11-15T01:51:33.788 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:33.783+0000 7f5b877fe700 1 -- 172.21.15.137:0/3905430409 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b94014d90 con 0x7f5ba00f6510 2023-11-15T01:51:33.788 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:33.783+0000 7f5ba5160700 1 -- 172.21.15.137:0/3905430409 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ba01072b0 con 0x7f5ba00f6510 2023-11-15T01:51:33.788 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:33.783+0000 7f5b877fe700 1 -- 172.21.15.137:0/3905430409 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5b940263a0 con 0x7f5ba00f6510 2023-11-15T01:51:33.788 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:33.783+0000 7f5ba5160700 1 -- 172.21.15.137:0/3905430409 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ba0045ad0 con 0x7f5ba00f6510 2023-11-15T01:51:33.788 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:33.784+0000 7f5b877fe700 1 --2- 172.21.15.137:0/3905430409 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f5b8802ffb0 0x7f5b88032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:51:33.789 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:33.784+0000 7f5b877fe700 1 -- 172.21.15.137:0/3905430409 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5b94041330 con 0x7f5ba00f6510 2023-11-15T01:51:33.789 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:33.784+0000 7f5b9e59c700 1 --2- 172.21.15.137:0/3905430409 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f5b8802ffb0 0x7f5b88032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:51:33.789 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:33.784+0000 7f5b9e59c700 1 --2- 172.21.15.137:0/3905430409 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f5b8802ffb0 0x7f5b88032470 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f5b900096a0 tx=0x7f5b90006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:51:33.789 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:33.788+0000 7f5b877fe700 1 -- 172.21.15.137:0/3905430409 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5b9401cb10 con 0x7f5ba00f6510 2023-11-15T01:51:33.995 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:33.995+0000 7f5ba5160700 1 -- 172.21.15.137:0/3905430409 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5ba00fbf60 con 0x7f5ba00f6510 2023-11-15T01:51:33.996 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:33.995+0000 7f5b877fe700 1 -- 172.21.15.137:0/3905430409 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5b9401cb10 con 0x7f5ba00f6510 2023-11-15T01:51:33.996 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:51:33.997 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:51:33.998 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:33.997+0000 7f5ba5160700 1 -- 172.21.15.137:0/3905430409 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f5b8802ffb0 msgr2=0x7f5b88032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:51:33.998 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:33.997+0000 7f5ba5160700 1 --2- 172.21.15.137:0/3905430409 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f5b8802ffb0 0x7f5b88032470 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f5b900096a0 tx=0x7f5b90006b40).stop 2023-11-15T01:51:33.998 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:33.997+0000 7f5ba5160700 1 -- 172.21.15.137:0/3905430409 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5ba00f6510 msgr2=0x7f5ba0106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:51:33.998 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:33.997+0000 7f5ba5160700 1 --2- 172.21.15.137:0/3905430409 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5ba00f6510 0x7f5ba0106730 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f5b94013510 tx=0x7f5b9400b040).stop 2023-11-15T01:51:33.999 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:33.997+0000 7f5ba5160700 1 -- 172.21.15.137:0/3905430409 shutdown_connections 2023-11-15T01:51:33.999 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:33.997+0000 7f5ba5160700 1 --2- 172.21.15.137:0/3905430409 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f5b8802ffb0 0x7f5b88032470 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:51:33.999 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:33.997+0000 7f5ba5160700 1 --2- 172.21.15.137:0/3905430409 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5ba00f6510 0x7f5ba0106730 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:51:33.999 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:33.997+0000 7f5ba5160700 1 -- 172.21.15.137:0/3905430409 >> 172.21.15.137:0/3905430409 conn(0x7f5ba00f1c70 msgr2=0x7f5ba00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:51:33.999 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:33.997+0000 7f5ba5160700 1 -- 172.21.15.137:0/3905430409 shutdown_connections 2023-11-15T01:51:34.000 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:33.997+0000 7f5ba5160700 1 -- 172.21.15.137:0/3905430409 wait complete. 2023-11-15T01:51:34.000 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:51:34.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:51:33 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:51:32.707281+0000 mgr.smithi055.szwngw (mgr.14164) 56 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:51:35.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:51:34 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:51:33.996654+0000 mon.smithi055 (mon.0) 147 : audit [DBG] from='client.? 172.21.15.137:0/3905430409' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:51:35.383 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:51:35.383 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:51:36.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:51:35 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:51:34.707653+0000 mgr.smithi055.szwngw (mgr.14164) 57 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:51:36.581 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:36.580+0000 7fcff160f700 1 -- 172.21.15.137:0/2489403778 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcfec0f5410 msgr2=0x7fcfec0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:51:36.582 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:36.580+0000 7fcff160f700 1 --2- 172.21.15.137:0/2489403778 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcfec0f5410 0x7fcfec0f5830 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7fcfe0009fb0 tx=0x7fcfe0009b30).stop 2023-11-15T01:51:36.582 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:36.581+0000 7fcff160f700 1 -- 172.21.15.137:0/2489403778 shutdown_connections 2023-11-15T01:51:36.582 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:36.581+0000 7fcff160f700 1 --2- 172.21.15.137:0/2489403778 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcfec0f5410 0x7fcfec0f5830 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:51:36.582 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:36.581+0000 7fcff160f700 1 -- 172.21.15.137:0/2489403778 >> 172.21.15.137:0/2489403778 conn(0x7fcfec0f0b70 msgr2=0x7fcfec0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:51:36.583 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:36.581+0000 7fcff160f700 1 -- 172.21.15.137:0/2489403778 shutdown_connections 2023-11-15T01:51:36.583 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:36.581+0000 7fcff160f700 1 -- 172.21.15.137:0/2489403778 wait complete. 2023-11-15T01:51:36.585 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:36.581+0000 7fcff160f700 1 Processor -- start 2023-11-15T01:51:36.585 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:36.581+0000 7fcff160f700 1 -- start start 2023-11-15T01:51:36.585 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:36.582+0000 7fcff160f700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcfec0f5410 0x7fcfec105600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:51:36.586 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:36.582+0000 7fcff160f700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfec105af0 con 0x7fcfec0f5410 2023-11-15T01:51:36.586 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:36.582+0000 7fcfeaffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcfec0f5410 0x7fcfec105600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:51:36.586 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:36.582+0000 7fcfeaffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcfec0f5410 0x7fcfec105600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:39312/0 (socket says 172.21.15.137:39312) 2023-11-15T01:51:36.586 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:36.582+0000 7fcfeaffd700 1 -- 172.21.15.137:0/891513105 learned_addr learned my addr 172.21.15.137:0/891513105 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:51:36.586 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:36.582+0000 7fcfeaffd700 1 -- 172.21.15.137:0/891513105 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfe0005040 con 0x7fcfec0f5410 2023-11-15T01:51:36.587 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:36.583+0000 7fcfeaffd700 1 --2- 172.21.15.137:0/891513105 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcfec0f5410 0x7fcfec105600 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7fcfe0013510 tx=0x7fcfe000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:51:36.587 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:36.583+0000 7fcfd37fe700 1 -- 172.21.15.137:0/891513105 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcfe0009320 con 0x7fcfec0f5410 2023-11-15T01:51:36.587 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:36.583+0000 7fcfd37fe700 1 -- 172.21.15.137:0/891513105 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcfe0013f00 con 0x7fcfec0f5410 2023-11-15T01:51:36.587 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:36.583+0000 7fcff160f700 1 -- 172.21.15.137:0/891513105 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfec105cf0 con 0x7fcfec0f5410 2023-11-15T01:51:36.587 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:36.583+0000 7fcfd37fe700 1 -- 172.21.15.137:0/891513105 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcfe0014d90 con 0x7fcfec0f5410 2023-11-15T01:51:36.588 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:36.583+0000 7fcff160f700 1 -- 172.21.15.137:0/891513105 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfec106180 con 0x7fcfec0f5410 2023-11-15T01:51:36.588 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:36.584+0000 7fcfd37fe700 1 -- 172.21.15.137:0/891513105 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcfe00095f0 con 0x7fcfec0f5410 2023-11-15T01:51:36.588 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:36.584+0000 7fcff160f700 1 -- 172.21.15.137:0/891513105 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfec045ad0 con 0x7fcfec0f5410 2023-11-15T01:51:36.588 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:36.584+0000 7fcfd37fe700 1 --2- 172.21.15.137:0/891513105 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fcfd402ff60 0x7fcfd4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:51:36.588 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:36.584+0000 7fcfd37fe700 1 -- 172.21.15.137:0/891513105 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcfe0041000 con 0x7fcfec0f5410 2023-11-15T01:51:36.588 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:36.584+0000 7fcfea7fc700 1 --2- 172.21.15.137:0/891513105 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fcfd402ff60 0x7fcfd4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:51:36.589 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:36.584+0000 7fcfea7fc700 1 --2- 172.21.15.137:0/891513105 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fcfd402ff60 0x7fcfd4032420 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fcfdc0096a0 tx=0x7fcfdc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:51:36.589 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:36.589+0000 7fcfd37fe700 1 -- 172.21.15.137:0/891513105 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcfe0014420 con 0x7fcfec0f5410 2023-11-15T01:51:36.794 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:36.794+0000 7fcff160f700 1 -- 172.21.15.137:0/891513105 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcfec0fae60 con 0x7fcfec0f5410 2023-11-15T01:51:36.795 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:36.795+0000 7fcfd37fe700 1 -- 172.21.15.137:0/891513105 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcfe001d7f0 con 0x7fcfec0f5410 2023-11-15T01:51:36.796 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:51:36.796 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:51:36.797 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:36.796+0000 7fcff160f700 1 -- 172.21.15.137:0/891513105 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fcfd402ff60 msgr2=0x7fcfd4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:51:36.797 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:36.796+0000 7fcff160f700 1 --2- 172.21.15.137:0/891513105 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fcfd402ff60 0x7fcfd4032420 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fcfdc0096a0 tx=0x7fcfdc006b40).stop 2023-11-15T01:51:36.797 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:36.796+0000 7fcff160f700 1 -- 172.21.15.137:0/891513105 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcfec0f5410 msgr2=0x7fcfec105600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:51:36.798 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:36.796+0000 7fcff160f700 1 --2- 172.21.15.137:0/891513105 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcfec0f5410 0x7fcfec105600 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7fcfe0013510 tx=0x7fcfe000b040).stop 2023-11-15T01:51:36.798 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:36.796+0000 7fcff160f700 1 -- 172.21.15.137:0/891513105 shutdown_connections 2023-11-15T01:51:36.798 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:36.796+0000 7fcff160f700 1 --2- 172.21.15.137:0/891513105 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fcfd402ff60 0x7fcfd4032420 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:51:36.798 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:36.796+0000 7fcff160f700 1 --2- 172.21.15.137:0/891513105 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcfec0f5410 0x7fcfec105600 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:51:36.798 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:36.796+0000 7fcff160f700 1 -- 172.21.15.137:0/891513105 >> 172.21.15.137:0/891513105 conn(0x7fcfec0f0b70 msgr2=0x7fcfec0f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:51:36.799 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:36.796+0000 7fcff160f700 1 -- 172.21.15.137:0/891513105 shutdown_connections 2023-11-15T01:51:36.799 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:36.796+0000 7fcff160f700 1 -- 172.21.15.137:0/891513105 wait complete. 2023-11-15T01:51:36.799 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:51:38.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:51:37 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:51:36.707973+0000 mgr.smithi055.szwngw (mgr.14164) 58 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:51:38.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:51:37 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:51:36.795709+0000 mon.smithi055 (mon.0) 148 : audit [DBG] from='client.? 172.21.15.137:0/891513105' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:51:38.188 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:51:38.189 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:51:39.355 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:39.352+0000 7fe043986700 1 -- 172.21.15.137:0/2394640673 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe03c0f53f0 msgr2=0x7fe03c0f5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:51:39.355 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:39.352+0000 7fe043986700 1 --2- 172.21.15.137:0/2394640673 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe03c0f53f0 0x7fe03c0f5810 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7fe02c009fb0 tx=0x7fe02c009b30).stop 2023-11-15T01:51:39.356 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:39.352+0000 7fe043986700 1 -- 172.21.15.137:0/2394640673 shutdown_connections 2023-11-15T01:51:39.356 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:39.352+0000 7fe043986700 1 --2- 172.21.15.137:0/2394640673 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe03c0f53f0 0x7fe03c0f5810 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:51:39.356 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:39.352+0000 7fe043986700 1 -- 172.21.15.137:0/2394640673 >> 172.21.15.137:0/2394640673 conn(0x7fe03c0f0b70 msgr2=0x7fe03c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:51:39.356 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:39.353+0000 7fe043986700 1 -- 172.21.15.137:0/2394640673 shutdown_connections 2023-11-15T01:51:39.357 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:39.353+0000 7fe043986700 1 -- 172.21.15.137:0/2394640673 wait complete. 2023-11-15T01:51:39.357 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:39.353+0000 7fe043986700 1 Processor -- start 2023-11-15T01:51:39.357 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:39.353+0000 7fe043986700 1 -- start start 2023-11-15T01:51:39.357 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:39.353+0000 7fe043986700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe03c0f53f0 0x7fe03c1093e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:51:39.357 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:39.353+0000 7fe043986700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe03c060240 con 0x7fe03c0f53f0 2023-11-15T01:51:39.358 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:39.354+0000 7fe041722700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe03c0f53f0 0x7fe03c1093e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:51:39.358 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:39.354+0000 7fe041722700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe03c0f53f0 0x7fe03c1093e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:34720/0 (socket says 172.21.15.137:34720) 2023-11-15T01:51:39.358 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:39.354+0000 7fe041722700 1 -- 172.21.15.137:0/4191539469 learned_addr learned my addr 172.21.15.137:0/4191539469 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:51:39.358 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:39.354+0000 7fe041722700 1 -- 172.21.15.137:0/4191539469 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe02c005040 con 0x7fe03c0f53f0 2023-11-15T01:51:39.358 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:39.354+0000 7fe041722700 1 --2- 172.21.15.137:0/4191539469 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe03c0f53f0 0x7fe03c1093e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7fe02c013510 tx=0x7fe02c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:51:39.359 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:39.354+0000 7fe031ffb700 1 -- 172.21.15.137:0/4191539469 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe02c009320 con 0x7fe03c0f53f0 2023-11-15T01:51:39.359 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:39.354+0000 7fe043986700 1 -- 172.21.15.137:0/4191539469 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe03c060440 con 0x7fe03c0f53f0 2023-11-15T01:51:39.359 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:39.354+0000 7fe031ffb700 1 -- 172.21.15.137:0/4191539469 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe02c013f00 con 0x7fe03c0f53f0 2023-11-15T01:51:39.359 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:39.354+0000 7fe031ffb700 1 -- 172.21.15.137:0/4191539469 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe02c014d90 con 0x7fe03c0f53f0 2023-11-15T01:51:39.359 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:39.355+0000 7fe043986700 1 -- 172.21.15.137:0/4191539469 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe03c109a90 con 0x7fe03c0f53f0 2023-11-15T01:51:39.361 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:39.355+0000 7fe031ffb700 1 -- 172.21.15.137:0/4191539469 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe02c0095f0 con 0x7fe03c0f53f0 2023-11-15T01:51:39.361 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:39.355+0000 7fe043986700 1 -- 172.21.15.137:0/4191539469 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe03c045ad0 con 0x7fe03c0f53f0 2023-11-15T01:51:39.361 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:39.355+0000 7fe031ffb700 1 --2- 172.21.15.137:0/4191539469 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fe02802ff60 0x7fe028032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:51:39.361 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:39.356+0000 7fe031ffb700 1 -- 172.21.15.137:0/4191539469 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe02c041000 con 0x7fe03c0f53f0 2023-11-15T01:51:39.361 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:39.356+0000 7fe040f21700 1 --2- 172.21.15.137:0/4191539469 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fe02802ff60 0x7fe028032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:51:39.362 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:39.356+0000 7fe040f21700 1 --2- 172.21.15.137:0/4191539469 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fe02802ff60 0x7fe028032420 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7fe0380096a0 tx=0x7fe038006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:51:39.362 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:39.360+0000 7fe031ffb700 1 -- 172.21.15.137:0/4191539469 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe02c014420 con 0x7fe03c0f53f0 2023-11-15T01:51:39.569 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:39.568+0000 7fe043986700 1 -- 172.21.15.137:0/4191539469 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe03c10a2b0 con 0x7fe03c0f53f0 2023-11-15T01:51:39.570 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:39.569+0000 7fe031ffb700 1 -- 172.21.15.137:0/4191539469 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe02c01d7f0 con 0x7fe03c0f53f0 2023-11-15T01:51:39.570 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:51:39.571 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:51:39.572 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:39.571+0000 7fe043986700 1 -- 172.21.15.137:0/4191539469 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fe02802ff60 msgr2=0x7fe028032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:51:39.572 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:39.571+0000 7fe043986700 1 --2- 172.21.15.137:0/4191539469 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fe02802ff60 0x7fe028032420 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7fe0380096a0 tx=0x7fe038006b40).stop 2023-11-15T01:51:39.572 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:39.571+0000 7fe043986700 1 -- 172.21.15.137:0/4191539469 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe03c0f53f0 msgr2=0x7fe03c1093e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:51:39.572 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:39.571+0000 7fe043986700 1 --2- 172.21.15.137:0/4191539469 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe03c0f53f0 0x7fe03c1093e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7fe02c013510 tx=0x7fe02c00b040).stop 2023-11-15T01:51:39.573 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:39.571+0000 7fe043986700 1 -- 172.21.15.137:0/4191539469 shutdown_connections 2023-11-15T01:51:39.573 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:39.571+0000 7fe043986700 1 --2- 172.21.15.137:0/4191539469 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fe02802ff60 0x7fe028032420 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:51:39.573 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:39.571+0000 7fe043986700 1 --2- 172.21.15.137:0/4191539469 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe03c0f53f0 0x7fe03c1093e0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:51:39.573 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:39.571+0000 7fe043986700 1 -- 172.21.15.137:0/4191539469 >> 172.21.15.137:0/4191539469 conn(0x7fe03c0f0b70 msgr2=0x7fe03c0f1640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:51:39.573 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:39.571+0000 7fe043986700 1 -- 172.21.15.137:0/4191539469 shutdown_connections 2023-11-15T01:51:39.574 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:39.571+0000 7fe043986700 1 -- 172.21.15.137:0/4191539469 wait complete. 2023-11-15T01:51:39.574 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:51:40.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:51:39 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:51:38.708327+0000 mgr.smithi055.szwngw (mgr.14164) 59 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:51:40.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:51:39 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:51:39.570265+0000 mon.smithi055 (mon.0) 149 : audit [DBG] from='client.? 172.21.15.137:0/4191539469' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:51:40.976 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:51:40.976 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:51:41.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:51:41 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:51:40.708709+0000 mgr.smithi055.szwngw (mgr.14164) 60 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:51:42.139 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:42.137+0000 7fc48d108700 1 -- 172.21.15.137:0/599740763 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc4880f5410 msgr2=0x7fc4880f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:51:42.140 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:42.137+0000 7fc48d108700 1 --2- 172.21.15.137:0/599740763 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc4880f5410 0x7fc4880f5830 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7fc47c009fb0 tx=0x7fc47c009b30).stop 2023-11-15T01:51:42.140 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:42.137+0000 7fc48d108700 1 -- 172.21.15.137:0/599740763 shutdown_connections 2023-11-15T01:51:42.140 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:42.137+0000 7fc48d108700 1 --2- 172.21.15.137:0/599740763 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc4880f5410 0x7fc4880f5830 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:51:42.140 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:42.137+0000 7fc48d108700 1 -- 172.21.15.137:0/599740763 >> 172.21.15.137:0/599740763 conn(0x7fc4880f0b70 msgr2=0x7fc4880f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:51:42.140 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:42.137+0000 7fc48d108700 1 -- 172.21.15.137:0/599740763 shutdown_connections 2023-11-15T01:51:42.141 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:42.137+0000 7fc48d108700 1 -- 172.21.15.137:0/599740763 wait complete. 2023-11-15T01:51:42.141 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:42.138+0000 7fc48d108700 1 Processor -- start 2023-11-15T01:51:42.141 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:42.138+0000 7fc48d108700 1 -- start start 2023-11-15T01:51:42.141 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:42.138+0000 7fc48d108700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc4880f5410 0x7fc488109400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:51:42.141 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:42.138+0000 7fc48d108700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc488060240 con 0x7fc4880f5410 2023-11-15T01:51:42.142 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:42.139+0000 7fc486d9d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc4880f5410 0x7fc488109400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:51:42.142 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:42.139+0000 7fc486d9d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc4880f5410 0x7fc488109400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:34732/0 (socket says 172.21.15.137:34732) 2023-11-15T01:51:42.142 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:42.139+0000 7fc486d9d700 1 -- 172.21.15.137:0/1893297941 learned_addr learned my addr 172.21.15.137:0/1893297941 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:51:42.142 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:42.139+0000 7fc486d9d700 1 -- 172.21.15.137:0/1893297941 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc47c005040 con 0x7fc4880f5410 2023-11-15T01:51:42.143 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:42.139+0000 7fc486d9d700 1 --2- 172.21.15.137:0/1893297941 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc4880f5410 0x7fc488109400 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7fc47c013510 tx=0x7fc47c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:51:42.144 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:42.139+0000 7fc46f7fe700 1 -- 172.21.15.137:0/1893297941 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc47c009320 con 0x7fc4880f5410 2023-11-15T01:51:42.144 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:42.139+0000 7fc48d108700 1 -- 172.21.15.137:0/1893297941 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc488060440 con 0x7fc4880f5410 2023-11-15T01:51:42.144 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:42.139+0000 7fc46f7fe700 1 -- 172.21.15.137:0/1893297941 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc47c013f00 con 0x7fc4880f5410 2023-11-15T01:51:42.144 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:42.139+0000 7fc48d108700 1 -- 172.21.15.137:0/1893297941 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc488109ab0 con 0x7fc4880f5410 2023-11-15T01:51:42.144 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:42.139+0000 7fc46f7fe700 1 -- 172.21.15.137:0/1893297941 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc47c014d90 con 0x7fc4880f5410 2023-11-15T01:51:42.145 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:42.140+0000 7fc46f7fe700 1 -- 172.21.15.137:0/1893297941 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc47c0263a0 con 0x7fc4880f5410 2023-11-15T01:51:42.145 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:42.140+0000 7fc46f7fe700 1 --2- 172.21.15.137:0/1893297941 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fc47002ff60 0x7fc470032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:51:42.145 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:42.140+0000 7fc46f7fe700 1 -- 172.21.15.137:0/1893297941 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc47c0410e0 con 0x7fc4880f5410 2023-11-15T01:51:42.145 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:42.140+0000 7fc48d108700 1 -- 172.21.15.137:0/1893297941 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc488045ad0 con 0x7fc4880f5410 2023-11-15T01:51:42.145 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:42.141+0000 7fc48659c700 1 --2- 172.21.15.137:0/1893297941 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fc47002ff60 0x7fc470032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:51:42.146 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:42.141+0000 7fc48659c700 1 --2- 172.21.15.137:0/1893297941 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fc47002ff60 0x7fc470032420 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fc4780096a0 tx=0x7fc478006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:51:42.146 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:42.145+0000 7fc46f7fe700 1 -- 172.21.15.137:0/1893297941 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc47c012920 con 0x7fc4880f5410 2023-11-15T01:51:42.351 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:42.350+0000 7fc48d108700 1 -- 172.21.15.137:0/1893297941 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc4880fae60 con 0x7fc4880f5410 2023-11-15T01:51:42.351 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:42.351+0000 7fc46f7fe700 1 -- 172.21.15.137:0/1893297941 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc4880fae60 con 0x7fc4880f5410 2023-11-15T01:51:42.352 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:51:42.352 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:51:42.353 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:42.352+0000 7fc48d108700 1 -- 172.21.15.137:0/1893297941 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fc47002ff60 msgr2=0x7fc470032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:51:42.353 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:42.352+0000 7fc48d108700 1 --2- 172.21.15.137:0/1893297941 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fc47002ff60 0x7fc470032420 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fc4780096a0 tx=0x7fc478006b40).stop 2023-11-15T01:51:42.353 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:42.352+0000 7fc48d108700 1 -- 172.21.15.137:0/1893297941 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc4880f5410 msgr2=0x7fc488109400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:51:42.354 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:42.352+0000 7fc48d108700 1 --2- 172.21.15.137:0/1893297941 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc4880f5410 0x7fc488109400 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7fc47c013510 tx=0x7fc47c00b040).stop 2023-11-15T01:51:42.354 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:42.352+0000 7fc48d108700 1 -- 172.21.15.137:0/1893297941 shutdown_connections 2023-11-15T01:51:42.354 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:42.352+0000 7fc48d108700 1 --2- 172.21.15.137:0/1893297941 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fc47002ff60 0x7fc470032420 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:51:42.354 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:42.352+0000 7fc48d108700 1 --2- 172.21.15.137:0/1893297941 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc4880f5410 0x7fc488109400 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:51:42.354 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:42.352+0000 7fc48d108700 1 -- 172.21.15.137:0/1893297941 >> 172.21.15.137:0/1893297941 conn(0x7fc4880f0b70 msgr2=0x7fc4880f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:51:42.355 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:42.352+0000 7fc48d108700 1 -- 172.21.15.137:0/1893297941 shutdown_connections 2023-11-15T01:51:42.355 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:42.352+0000 7fc48d108700 1 -- 172.21.15.137:0/1893297941 wait complete. 2023-11-15T01:51:42.355 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:51:42.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:51:42 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:51:42.351804+0000 mon.smithi055 (mon.0) 150 : audit [DBG] from='client.? 172.21.15.137:0/1893297941' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:51:43.749 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:51:43.749 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:51:43.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:51:43 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:51:42.708981+0000 mgr.smithi055.szwngw (mgr.14164) 61 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:51:43.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:51:43 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:51:42.848889+0000 mon.smithi055 (mon.0) 151 : audit [INF] from='mgr.14164 172.21.15.55:0/59463371' entity='mgr.smithi055.szwngw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi055.szwngw/mirror_snapshot_schedule"}]: dispatch 2023-11-15T01:51:43.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:51:43 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:51:42.852119+0000 mon.smithi055 (mon.0) 152 : audit [INF] from='mgr.14164 172.21.15.55:0/59463371' entity='mgr.smithi055.szwngw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi055.szwngw/trash_purge_schedule"}]: dispatch 2023-11-15T01:51:44.914 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:44.912+0000 7f1378b3d700 1 -- 172.21.15.137:0/937441646 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f13740f4740 msgr2=0x7f13740f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:51:44.914 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:44.912+0000 7f1378b3d700 1 --2- 172.21.15.137:0/937441646 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f13740f4740 0x7f13740f4b60 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f1368009fb0 tx=0x7f1368009b30).stop 2023-11-15T01:51:44.915 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:44.912+0000 7f1378b3d700 1 -- 172.21.15.137:0/937441646 shutdown_connections 2023-11-15T01:51:44.915 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:44.912+0000 7f1378b3d700 1 --2- 172.21.15.137:0/937441646 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f13740f4740 0x7f13740f4b60 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:51:44.915 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:44.912+0000 7f1378b3d700 1 -- 172.21.15.137:0/937441646 >> 172.21.15.137:0/937441646 conn(0x7f13740f0260 msgr2=0x7f13740f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:51:44.915 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:44.912+0000 7f1378b3d700 1 -- 172.21.15.137:0/937441646 shutdown_connections 2023-11-15T01:51:44.915 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:44.912+0000 7f1378b3d700 1 -- 172.21.15.137:0/937441646 wait complete. 2023-11-15T01:51:44.915 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:44.913+0000 7f1378b3d700 1 Processor -- start 2023-11-15T01:51:44.916 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:44.913+0000 7f1378b3d700 1 -- start start 2023-11-15T01:51:44.916 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:44.913+0000 7f1378b3d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f13740f4740 0x7f13741066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:51:44.916 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:44.913+0000 7f1378b3d700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1374106bc0 con 0x7f13740f4740 2023-11-15T01:51:44.916 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:44.913+0000 7f137259c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f13740f4740 0x7f13741066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:51:44.916 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:44.914+0000 7f137259c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f13740f4740 0x7f13741066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:34758/0 (socket says 172.21.15.137:34758) 2023-11-15T01:51:44.917 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:44.914+0000 7f137259c700 1 -- 172.21.15.137:0/304949323 learned_addr learned my addr 172.21.15.137:0/304949323 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:51:44.918 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:44.914+0000 7f137259c700 1 -- 172.21.15.137:0/304949323 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1368005040 con 0x7f13740f4740 2023-11-15T01:51:44.918 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:44.914+0000 7f137259c700 1 --2- 172.21.15.137:0/304949323 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f13740f4740 0x7f13741066d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f1368013510 tx=0x7f136800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:51:44.919 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:44.914+0000 7f1362ffd700 1 -- 172.21.15.137:0/304949323 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1368009320 con 0x7f13740f4740 2023-11-15T01:51:44.919 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:44.914+0000 7f1378b3d700 1 -- 172.21.15.137:0/304949323 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1374106dc0 con 0x7f13740f4740 2023-11-15T01:51:44.919 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:44.914+0000 7f1362ffd700 1 -- 172.21.15.137:0/304949323 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1368013f00 con 0x7f13740f4740 2023-11-15T01:51:44.919 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:44.914+0000 7f1362ffd700 1 -- 172.21.15.137:0/304949323 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1368014d90 con 0x7f13740f4740 2023-11-15T01:51:44.919 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:44.914+0000 7f1378b3d700 1 -- 172.21.15.137:0/304949323 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1374107250 con 0x7f13740f4740 2023-11-15T01:51:44.920 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:44.915+0000 7f1362ffd700 1 -- 172.21.15.137:0/304949323 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f13680095f0 con 0x7f13740f4740 2023-11-15T01:51:44.920 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:44.915+0000 7f1378b3d700 1 -- 172.21.15.137:0/304949323 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1374045ad0 con 0x7f13740f4740 2023-11-15T01:51:44.920 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:44.915+0000 7f1362ffd700 1 --2- 172.21.15.137:0/304949323 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f135c02ff60 0x7f135c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:51:44.920 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:44.915+0000 7f1362ffd700 1 -- 172.21.15.137:0/304949323 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1368041070 con 0x7f13740f4740 2023-11-15T01:51:44.920 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:44.915+0000 7f1371d9b700 1 --2- 172.21.15.137:0/304949323 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f135c02ff60 0x7f135c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:51:44.921 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:44.916+0000 7f1371d9b700 1 --2- 172.21.15.137:0/304949323 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f135c02ff60 0x7f135c032420 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f13640096a0 tx=0x7f1364006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:51:44.921 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:44.920+0000 7f1362ffd700 1 -- 172.21.15.137:0/304949323 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1368014420 con 0x7f13740f4740 2023-11-15T01:51:45.125 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:45.124+0000 7f1378b3d700 1 -- 172.21.15.137:0/304949323 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f13741080a0 con 0x7f13740f4740 2023-11-15T01:51:45.125 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:45.125+0000 7f1362ffd700 1 -- 172.21.15.137:0/304949323 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1368010230 con 0x7f13740f4740 2023-11-15T01:51:45.126 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:51:45.126 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:51:45.127 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:45.126+0000 7f1378b3d700 1 -- 172.21.15.137:0/304949323 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f135c02ff60 msgr2=0x7f135c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:51:45.127 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:45.126+0000 7f1378b3d700 1 --2- 172.21.15.137:0/304949323 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f135c02ff60 0x7f135c032420 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f13640096a0 tx=0x7f1364006b40).stop 2023-11-15T01:51:45.127 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:45.126+0000 7f1378b3d700 1 -- 172.21.15.137:0/304949323 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f13740f4740 msgr2=0x7f13741066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:51:45.128 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:45.126+0000 7f1378b3d700 1 --2- 172.21.15.137:0/304949323 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f13740f4740 0x7f13741066d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f1368013510 tx=0x7f136800b040).stop 2023-11-15T01:51:45.128 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:45.126+0000 7f1378b3d700 1 -- 172.21.15.137:0/304949323 shutdown_connections 2023-11-15T01:51:45.128 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:45.126+0000 7f1378b3d700 1 --2- 172.21.15.137:0/304949323 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f135c02ff60 0x7f135c032420 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:51:45.128 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:45.126+0000 7f1378b3d700 1 --2- 172.21.15.137:0/304949323 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f13740f4740 0x7f13741066d0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:51:45.128 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:45.126+0000 7f1378b3d700 1 -- 172.21.15.137:0/304949323 >> 172.21.15.137:0/304949323 conn(0x7f13740f0260 msgr2=0x7f13740f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:51:45.129 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:45.126+0000 7f1378b3d700 1 -- 172.21.15.137:0/304949323 shutdown_connections 2023-11-15T01:51:45.129 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:45.126+0000 7f1378b3d700 1 -- 172.21.15.137:0/304949323 wait complete. 2023-11-15T01:51:45.129 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:51:46.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:51:45 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:51:44.709311+0000 mgr.smithi055.szwngw (mgr.14164) 62 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:51:46.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:51:45 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:51:45.125822+0000 mon.smithi055 (mon.0) 153 : audit [DBG] from='client.? 172.21.15.137:0/304949323' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:51:46.530 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:51:46.531 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:51:47.663 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:47.661+0000 7f64fb1f9700 1 -- 172.21.15.137:0/1819554677 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f64f4060200 msgr2=0x7f64f40f9170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:51:47.664 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:47.661+0000 7f64fb1f9700 1 --2- 172.21.15.137:0/1819554677 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f64f4060200 0x7f64f40f9170 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f64e8009fb0 tx=0x7f64e8009b30).stop 2023-11-15T01:51:47.664 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:47.662+0000 7f64fb1f9700 1 -- 172.21.15.137:0/1819554677 shutdown_connections 2023-11-15T01:51:47.664 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:47.662+0000 7f64fb1f9700 1 --2- 172.21.15.137:0/1819554677 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f64f4060200 0x7f64f40f9170 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:51:47.664 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:47.662+0000 7f64fb1f9700 1 -- 172.21.15.137:0/1819554677 >> 172.21.15.137:0/1819554677 conn(0x7f64f40f0b70 msgr2=0x7f64f40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:51:47.664 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:47.662+0000 7f64fb1f9700 1 -- 172.21.15.137:0/1819554677 shutdown_connections 2023-11-15T01:51:47.665 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:47.662+0000 7f64fb1f9700 1 -- 172.21.15.137:0/1819554677 wait complete. 2023-11-15T01:51:47.665 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:47.662+0000 7f64fb1f9700 1 Processor -- start 2023-11-15T01:51:47.665 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:47.662+0000 7f64fb1f9700 1 -- start start 2023-11-15T01:51:47.665 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:47.662+0000 7f64fb1f9700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f64f4060200 0x7f64f41066a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:51:47.665 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:47.662+0000 7f64fb1f9700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64f4106b90 con 0x7f64f4060200 2023-11-15T01:51:47.666 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:47.663+0000 7f64f8f95700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f64f4060200 0x7f64f41066a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:51:47.666 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:47.663+0000 7f64f8f95700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f64f4060200 0x7f64f41066a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:34776/0 (socket says 172.21.15.137:34776) 2023-11-15T01:51:47.666 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:47.663+0000 7f64f8f95700 1 -- 172.21.15.137:0/3649108687 learned_addr learned my addr 172.21.15.137:0/3649108687 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:51:47.666 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:47.663+0000 7f64f8f95700 1 -- 172.21.15.137:0/3649108687 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64e8005040 con 0x7f64f4060200 2023-11-15T01:51:47.668 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:47.663+0000 7f64f8f95700 1 --2- 172.21.15.137:0/3649108687 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f64f4060200 0x7f64f41066a0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f64e8013510 tx=0x7f64e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:51:47.668 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:47.663+0000 7f64f17fa700 1 -- 172.21.15.137:0/3649108687 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f64e8009320 con 0x7f64f4060200 2023-11-15T01:51:47.668 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:47.663+0000 7f64fb1f9700 1 -- 172.21.15.137:0/3649108687 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64f4106d90 con 0x7f64f4060200 2023-11-15T01:51:47.668 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:47.663+0000 7f64f17fa700 1 -- 172.21.15.137:0/3649108687 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f64e8013f00 con 0x7f64f4060200 2023-11-15T01:51:47.668 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:47.663+0000 7f64f17fa700 1 -- 172.21.15.137:0/3649108687 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f64e8014dc0 con 0x7f64f4060200 2023-11-15T01:51:47.669 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:47.664+0000 7f64fb1f9700 1 -- 172.21.15.137:0/3649108687 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64f4107220 con 0x7f64f4060200 2023-11-15T01:51:47.669 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:47.664+0000 7f64f17fa700 1 -- 172.21.15.137:0/3649108687 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f64e80095f0 con 0x7f64f4060200 2023-11-15T01:51:47.669 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:47.664+0000 7f64fb1f9700 1 -- 172.21.15.137:0/3649108687 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64f4045ad0 con 0x7f64f4060200 2023-11-15T01:51:47.669 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:47.664+0000 7f64f17fa700 1 --2- 172.21.15.137:0/3649108687 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f64e402ff60 0x7f64e4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:51:47.669 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:47.664+0000 7f64f17fa700 1 -- 172.21.15.137:0/3649108687 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f64e80410c0 con 0x7f64f4060200 2023-11-15T01:51:47.670 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:47.665+0000 7f64f3fff700 1 --2- 172.21.15.137:0/3649108687 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f64e402ff60 0x7f64e4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:51:47.670 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:47.665+0000 7f64f3fff700 1 --2- 172.21.15.137:0/3649108687 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f64e402ff60 0x7f64e4032420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f64e00096a0 tx=0x7f64e0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:51:47.670 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:47.669+0000 7f64f17fa700 1 -- 172.21.15.137:0/3649108687 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f64e801c790 con 0x7f64f4060200 2023-11-15T01:51:47.877 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:47.876+0000 7f64fb1f9700 1 -- 172.21.15.137:0/3649108687 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f64f41080d0 con 0x7f64f4060200 2023-11-15T01:51:47.878 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:47.877+0000 7f64f17fa700 1 -- 172.21.15.137:0/3649108687 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f64e8010230 con 0x7f64f4060200 2023-11-15T01:51:47.878 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:51:47.878 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:51:47.879 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:47.878+0000 7f64fb1f9700 1 -- 172.21.15.137:0/3649108687 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f64e402ff60 msgr2=0x7f64e4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:51:47.879 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:47.878+0000 7f64fb1f9700 1 --2- 172.21.15.137:0/3649108687 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f64e402ff60 0x7f64e4032420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f64e00096a0 tx=0x7f64e0006b40).stop 2023-11-15T01:51:47.880 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:47.878+0000 7f64fb1f9700 1 -- 172.21.15.137:0/3649108687 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f64f4060200 msgr2=0x7f64f41066a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:51:47.880 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:47.878+0000 7f64fb1f9700 1 --2- 172.21.15.137:0/3649108687 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f64f4060200 0x7f64f41066a0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f64e8013510 tx=0x7f64e800b040).stop 2023-11-15T01:51:47.880 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:47.879+0000 7f64fb1f9700 1 -- 172.21.15.137:0/3649108687 shutdown_connections 2023-11-15T01:51:47.880 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:47.879+0000 7f64fb1f9700 1 --2- 172.21.15.137:0/3649108687 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f64e402ff60 0x7f64e4032420 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:51:47.880 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:47.879+0000 7f64fb1f9700 1 --2- 172.21.15.137:0/3649108687 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f64f4060200 0x7f64f41066a0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:51:47.881 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:47.879+0000 7f64fb1f9700 1 -- 172.21.15.137:0/3649108687 >> 172.21.15.137:0/3649108687 conn(0x7f64f40f0b70 msgr2=0x7f64f40f1650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:51:47.881 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:47.879+0000 7f64fb1f9700 1 -- 172.21.15.137:0/3649108687 shutdown_connections 2023-11-15T01:51:47.881 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:47.879+0000 7f64fb1f9700 1 -- 172.21.15.137:0/3649108687 wait complete. 2023-11-15T01:51:47.881 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:51:48.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:51:47 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:51:46.709746+0000 mgr.smithi055.szwngw (mgr.14164) 63 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:51:49.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:51:48 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:51:47.878059+0000 mon.smithi055 (mon.0) 154 : audit [DBG] from='client.? 172.21.15.137:0/3649108687' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:51:49.279 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:51:49.280 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:51:50.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:51:49 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:51:48.710087+0000 mgr.smithi055.szwngw (mgr.14164) 64 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:51:50.456 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:50.452+0000 7fb987c74700 1 -- 172.21.15.137:0/1487315460 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb9800f5410 msgr2=0x7fb9800f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:51:50.457 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:50.452+0000 7fb987c74700 1 --2- 172.21.15.137:0/1487315460 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb9800f5410 0x7fb9800f5830 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7fb970009fb0 tx=0x7fb970009b30).stop 2023-11-15T01:51:50.457 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:50.452+0000 7fb987c74700 1 -- 172.21.15.137:0/1487315460 shutdown_connections 2023-11-15T01:51:50.458 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:50.452+0000 7fb987c74700 1 --2- 172.21.15.137:0/1487315460 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb9800f5410 0x7fb9800f5830 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:51:50.458 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:50.452+0000 7fb987c74700 1 -- 172.21.15.137:0/1487315460 >> 172.21.15.137:0/1487315460 conn(0x7fb9800f0b70 msgr2=0x7fb9800f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:51:50.459 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:50.452+0000 7fb987c74700 1 -- 172.21.15.137:0/1487315460 shutdown_connections 2023-11-15T01:51:50.459 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:50.452+0000 7fb987c74700 1 -- 172.21.15.137:0/1487315460 wait complete. 2023-11-15T01:51:50.460 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:50.452+0000 7fb987c74700 1 Processor -- start 2023-11-15T01:51:50.460 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:50.452+0000 7fb987c74700 1 -- start start 2023-11-15T01:51:50.460 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:50.452+0000 7fb987c74700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb9800f5410 0x7fb980105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:51:50.460 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:50.452+0000 7fb987c74700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb980105b20 con 0x7fb9800f5410 2023-11-15T01:51:50.460 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:50.453+0000 7fb985a10700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb9800f5410 0x7fb980105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:51:50.461 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:50.453+0000 7fb985a10700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb9800f5410 0x7fb980105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:45292/0 (socket says 172.21.15.137:45292) 2023-11-15T01:51:50.461 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:50.453+0000 7fb985a10700 1 -- 172.21.15.137:0/3128749042 learned_addr learned my addr 172.21.15.137:0/3128749042 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:51:50.461 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:50.453+0000 7fb985a10700 1 -- 172.21.15.137:0/3128749042 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb970005040 con 0x7fb9800f5410 2023-11-15T01:51:50.461 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:50.453+0000 7fb985a10700 1 --2- 172.21.15.137:0/3128749042 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb9800f5410 0x7fb980105630 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fb970013510 tx=0x7fb97000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:51:50.461 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:50.453+0000 7fb9767fc700 1 -- 172.21.15.137:0/3128749042 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb970009320 con 0x7fb9800f5410 2023-11-15T01:51:50.461 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:50.453+0000 7fb987c74700 1 -- 172.21.15.137:0/3128749042 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb980105d20 con 0x7fb9800f5410 2023-11-15T01:51:50.462 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:50.453+0000 7fb9767fc700 1 -- 172.21.15.137:0/3128749042 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb970013f00 con 0x7fb9800f5410 2023-11-15T01:51:50.462 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:50.453+0000 7fb9767fc700 1 -- 172.21.15.137:0/3128749042 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb970014d90 con 0x7fb9800f5410 2023-11-15T01:51:50.462 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:50.453+0000 7fb987c74700 1 -- 172.21.15.137:0/3128749042 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9801061b0 con 0x7fb9800f5410 2023-11-15T01:51:50.462 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:50.454+0000 7fb9767fc700 1 -- 172.21.15.137:0/3128749042 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb970014420 con 0x7fb9800f5410 2023-11-15T01:51:50.462 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:50.454+0000 7fb9767fc700 1 --2- 172.21.15.137:0/3128749042 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fb96c02ffb0 0x7fb96c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:51:50.463 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:50.454+0000 7fb987c74700 1 -- 172.21.15.137:0/3128749042 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb980045ad0 con 0x7fb9800f5410 2023-11-15T01:51:50.463 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:50.454+0000 7fb9767fc700 1 -- 172.21.15.137:0/3128749042 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb9700410c0 con 0x7fb9800f5410 2023-11-15T01:51:50.463 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:50.455+0000 7fb98520f700 1 --2- 172.21.15.137:0/3128749042 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fb96c02ffb0 0x7fb96c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:51:50.463 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:50.455+0000 7fb98520f700 1 --2- 172.21.15.137:0/3128749042 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fb96c02ffb0 0x7fb96c032470 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fb97c0096a0 tx=0x7fb97c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:51:50.464 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:50.459+0000 7fb9767fc700 1 -- 172.21.15.137:0/3128749042 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb9700095f0 con 0x7fb9800f5410 2023-11-15T01:51:50.668 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:50.667+0000 7fb987c74700 1 -- 172.21.15.137:0/3128749042 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb9800faf50 con 0x7fb9800f5410 2023-11-15T01:51:50.669 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:50.668+0000 7fb9767fc700 1 -- 172.21.15.137:0/3128749042 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb97001db70 con 0x7fb9800f5410 2023-11-15T01:51:50.669 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:51:50.670 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:51:50.671 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:50.669+0000 7fb987c74700 1 -- 172.21.15.137:0/3128749042 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fb96c02ffb0 msgr2=0x7fb96c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:51:50.671 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:50.669+0000 7fb987c74700 1 --2- 172.21.15.137:0/3128749042 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fb96c02ffb0 0x7fb96c032470 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fb97c0096a0 tx=0x7fb97c006b40).stop 2023-11-15T01:51:50.671 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:50.669+0000 7fb987c74700 1 -- 172.21.15.137:0/3128749042 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb9800f5410 msgr2=0x7fb980105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:51:50.671 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:50.669+0000 7fb987c74700 1 --2- 172.21.15.137:0/3128749042 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb9800f5410 0x7fb980105630 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fb970013510 tx=0x7fb97000b040).stop 2023-11-15T01:51:50.672 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:50.669+0000 7fb987c74700 1 -- 172.21.15.137:0/3128749042 shutdown_connections 2023-11-15T01:51:50.672 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:50.669+0000 7fb987c74700 1 --2- 172.21.15.137:0/3128749042 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fb96c02ffb0 0x7fb96c032470 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:51:50.672 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:50.669+0000 7fb987c74700 1 --2- 172.21.15.137:0/3128749042 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb9800f5410 0x7fb980105630 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:51:50.672 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:50.669+0000 7fb987c74700 1 -- 172.21.15.137:0/3128749042 >> 172.21.15.137:0/3128749042 conn(0x7fb9800f0b70 msgr2=0x7fb9800f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:51:50.672 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:50.669+0000 7fb987c74700 1 -- 172.21.15.137:0/3128749042 shutdown_connections 2023-11-15T01:51:50.673 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:50.669+0000 7fb987c74700 1 -- 172.21.15.137:0/3128749042 wait complete. 2023-11-15T01:51:50.673 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:51:51.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:51:51 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:51:50.668910+0000 mon.smithi055 (mon.0) 155 : audit [DBG] from='client.? 172.21.15.137:0/3128749042' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:51:51.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:51:51 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:51:50.710373+0000 mgr.smithi055.szwngw (mgr.14164) 65 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:51:52.060 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:51:52.060 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:51:53.194 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:53.193+0000 7f7042def700 1 -- 172.21.15.137:0/3709676307 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f703c0f5410 msgr2=0x7f703c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:51:53.195 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:53.193+0000 7f7042def700 1 --2- 172.21.15.137:0/3709676307 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f703c0f5410 0x7f703c0f5830 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f7030009fb0 tx=0x7f7030009b30).stop 2023-11-15T01:51:53.195 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:53.193+0000 7f7042def700 1 -- 172.21.15.137:0/3709676307 shutdown_connections 2023-11-15T01:51:53.195 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:53.193+0000 7f7042def700 1 --2- 172.21.15.137:0/3709676307 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f703c0f5410 0x7f703c0f5830 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:51:53.195 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:53.193+0000 7f7042def700 1 -- 172.21.15.137:0/3709676307 >> 172.21.15.137:0/3709676307 conn(0x7f703c0f0b70 msgr2=0x7f703c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:51:53.196 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:53.193+0000 7f7042def700 1 -- 172.21.15.137:0/3709676307 shutdown_connections 2023-11-15T01:51:53.196 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:53.193+0000 7f7042def700 1 -- 172.21.15.137:0/3709676307 wait complete. 2023-11-15T01:51:53.196 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:53.194+0000 7f7042def700 1 Processor -- start 2023-11-15T01:51:53.196 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:53.194+0000 7f7042def700 1 -- start start 2023-11-15T01:51:53.196 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:53.194+0000 7f7042def700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f703c0f5410 0x7f703c105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:51:53.197 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:53.194+0000 7f7042def700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f703c105b20 con 0x7f703c0f5410 2023-11-15T01:51:53.198 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:53.194+0000 7f7040b8b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f703c0f5410 0x7f703c105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:51:53.199 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:53.194+0000 7f7040b8b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f703c0f5410 0x7f703c105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:45320/0 (socket says 172.21.15.137:45320) 2023-11-15T01:51:53.199 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:53.194+0000 7f7040b8b700 1 -- 172.21.15.137:0/1746924056 learned_addr learned my addr 172.21.15.137:0/1746924056 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:51:53.199 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:53.194+0000 7f7040b8b700 1 -- 172.21.15.137:0/1746924056 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7030005040 con 0x7f703c0f5410 2023-11-15T01:51:53.199 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:53.195+0000 7f7040b8b700 1 --2- 172.21.15.137:0/1746924056 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f703c0f5410 0x7f703c105630 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f7030013510 tx=0x7f703000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:51:53.199 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:53.195+0000 7f70397fa700 1 -- 172.21.15.137:0/1746924056 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7030009320 con 0x7f703c0f5410 2023-11-15T01:51:53.200 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:53.195+0000 7f7042def700 1 -- 172.21.15.137:0/1746924056 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f703c105d20 con 0x7f703c0f5410 2023-11-15T01:51:53.200 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:53.195+0000 7f70397fa700 1 -- 172.21.15.137:0/1746924056 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7030013f00 con 0x7f703c0f5410 2023-11-15T01:51:53.200 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:53.195+0000 7f70397fa700 1 -- 172.21.15.137:0/1746924056 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7030014d90 con 0x7f703c0f5410 2023-11-15T01:51:53.200 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:53.195+0000 7f7042def700 1 -- 172.21.15.137:0/1746924056 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f703c1061b0 con 0x7f703c0f5410 2023-11-15T01:51:53.200 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:53.196+0000 7f70397fa700 1 -- 172.21.15.137:0/1746924056 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f70300263a0 con 0x7f703c0f5410 2023-11-15T01:51:53.201 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:53.196+0000 7f70397fa700 1 --2- 172.21.15.137:0/1746924056 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f702c02ff60 0x7f702c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:51:53.201 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:53.196+0000 7f70397fa700 1 -- 172.21.15.137:0/1746924056 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f70300410a0 con 0x7f703c0f5410 2023-11-15T01:51:53.201 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:53.196+0000 7f7042def700 1 -- 172.21.15.137:0/1746924056 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f703c045ad0 con 0x7f703c0f5410 2023-11-15T01:51:53.201 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:53.196+0000 7f703bfff700 1 --2- 172.21.15.137:0/1746924056 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f702c02ff60 0x7f702c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:51:53.201 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:53.196+0000 7f703bfff700 1 --2- 172.21.15.137:0/1746924056 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f702c02ff60 0x7f702c032420 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f70280096a0 tx=0x7f7028006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:51:53.202 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:53.201+0000 7f70397fa700 1 -- 172.21.15.137:0/1746924056 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7030012920 con 0x7f703c0f5410 2023-11-15T01:51:53.407 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:53.406+0000 7f7042def700 1 -- 172.21.15.137:0/1746924056 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f703c0faf50 con 0x7f703c0f5410 2023-11-15T01:51:53.407 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:53.407+0000 7f70397fa700 1 -- 172.21.15.137:0/1746924056 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f703002b020 con 0x7f703c0f5410 2023-11-15T01:51:53.408 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:51:53.408 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:51:53.409 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:53.408+0000 7f7042def700 1 -- 172.21.15.137:0/1746924056 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f702c02ff60 msgr2=0x7f702c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:51:53.410 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:53.408+0000 7f7042def700 1 --2- 172.21.15.137:0/1746924056 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f702c02ff60 0x7f702c032420 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f70280096a0 tx=0x7f7028006b40).stop 2023-11-15T01:51:53.410 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:53.408+0000 7f7042def700 1 -- 172.21.15.137:0/1746924056 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f703c0f5410 msgr2=0x7f703c105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:51:53.410 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:53.408+0000 7f7042def700 1 --2- 172.21.15.137:0/1746924056 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f703c0f5410 0x7f703c105630 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f7030013510 tx=0x7f703000b040).stop 2023-11-15T01:51:53.410 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:53.408+0000 7f7042def700 1 -- 172.21.15.137:0/1746924056 shutdown_connections 2023-11-15T01:51:53.411 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:53.408+0000 7f7042def700 1 --2- 172.21.15.137:0/1746924056 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f702c02ff60 0x7f702c032420 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:51:53.411 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:53.409+0000 7f7042def700 1 --2- 172.21.15.137:0/1746924056 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f703c0f5410 0x7f703c105630 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:51:53.411 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:53.409+0000 7f7042def700 1 -- 172.21.15.137:0/1746924056 >> 172.21.15.137:0/1746924056 conn(0x7f703c0f0b70 msgr2=0x7f703c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:51:53.411 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:53.409+0000 7f7042def700 1 -- 172.21.15.137:0/1746924056 shutdown_connections 2023-11-15T01:51:53.411 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:53.409+0000 7f7042def700 1 -- 172.21.15.137:0/1746924056 wait complete. 2023-11-15T01:51:53.412 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:51:54.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:51:53 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:51:52.710735+0000 mgr.smithi055.szwngw (mgr.14164) 66 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:51:54.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:51:53 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:51:53.407903+0000 mon.smithi055 (mon.0) 156 : audit [DBG] from='client.? 172.21.15.137:0/1746924056' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:51:54.792 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:51:54.793 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:51:55.936 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:55.935+0000 7f341133a700 1 -- 172.21.15.137:0/3484303099 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f340c0f5410 msgr2=0x7f340c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:51:55.937 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:55.935+0000 7f341133a700 1 --2- 172.21.15.137:0/3484303099 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f340c0f5410 0x7f340c0f5830 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f3400009fb0 tx=0x7f3400009b30).stop 2023-11-15T01:51:55.937 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:55.935+0000 7f341133a700 1 -- 172.21.15.137:0/3484303099 shutdown_connections 2023-11-15T01:51:55.937 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:55.935+0000 7f341133a700 1 --2- 172.21.15.137:0/3484303099 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f340c0f5410 0x7f340c0f5830 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:51:55.937 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:55.935+0000 7f341133a700 1 -- 172.21.15.137:0/3484303099 >> 172.21.15.137:0/3484303099 conn(0x7f340c0f0b70 msgr2=0x7f340c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:51:55.937 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:55.935+0000 7f341133a700 1 -- 172.21.15.137:0/3484303099 shutdown_connections 2023-11-15T01:51:55.938 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:55.935+0000 7f341133a700 1 -- 172.21.15.137:0/3484303099 wait complete. 2023-11-15T01:51:55.939 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:55.936+0000 7f341133a700 1 Processor -- start 2023-11-15T01:51:55.940 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:55.936+0000 7f341133a700 1 -- start start 2023-11-15T01:51:55.940 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:55.936+0000 7f341133a700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f340c0f5410 0x7f340c0fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:51:55.940 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:55.936+0000 7f341133a700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f340c0fb820 con 0x7f340c0f5410 2023-11-15T01:51:55.940 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:55.936+0000 7f340affd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f340c0f5410 0x7f340c0fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:51:55.940 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:55.937+0000 7f340affd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f340c0f5410 0x7f340c0fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:45338/0 (socket says 172.21.15.137:45338) 2023-11-15T01:51:55.941 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:55.937+0000 7f340affd700 1 -- 172.21.15.137:0/454012023 learned_addr learned my addr 172.21.15.137:0/454012023 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:51:55.941 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:55.937+0000 7f340affd700 1 -- 172.21.15.137:0/454012023 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3400005040 con 0x7f340c0f5410 2023-11-15T01:51:55.941 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:55.937+0000 7f340affd700 1 --2- 172.21.15.137:0/454012023 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f340c0f5410 0x7f340c0fd3a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f3400013510 tx=0x7f340000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:51:55.941 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:55.937+0000 7f33f37fe700 1 -- 172.21.15.137:0/454012023 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3400009320 con 0x7f340c0f5410 2023-11-15T01:51:55.941 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:55.937+0000 7f341133a700 1 -- 172.21.15.137:0/454012023 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f340c0fb9c0 con 0x7f340c0f5410 2023-11-15T01:51:55.942 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:55.937+0000 7f33f37fe700 1 -- 172.21.15.137:0/454012023 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3400013f00 con 0x7f340c0f5410 2023-11-15T01:51:55.942 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:55.937+0000 7f33f37fe700 1 -- 172.21.15.137:0/454012023 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3400014d90 con 0x7f340c0f5410 2023-11-15T01:51:55.942 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:55.937+0000 7f341133a700 1 -- 172.21.15.137:0/454012023 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f340c0fbe50 con 0x7f340c0f5410 2023-11-15T01:51:55.942 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:55.938+0000 7f33f37fe700 1 -- 172.21.15.137:0/454012023 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3400014420 con 0x7f340c0f5410 2023-11-15T01:51:55.942 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:55.938+0000 7f341133a700 1 -- 172.21.15.137:0/454012023 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f340c059100 con 0x7f340c0f5410 2023-11-15T01:51:55.943 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:55.938+0000 7f33f37fe700 1 --2- 172.21.15.137:0/454012023 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f33f402ff60 0x7f33f4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:51:55.943 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:55.938+0000 7f33f37fe700 1 -- 172.21.15.137:0/454012023 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3400041020 con 0x7f340c0f5410 2023-11-15T01:51:55.943 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:55.938+0000 7f340a7fc700 1 --2- 172.21.15.137:0/454012023 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f33f402ff60 0x7f33f4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:51:55.943 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:55.939+0000 7f340a7fc700 1 --2- 172.21.15.137:0/454012023 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f33f402ff60 0x7f33f4032420 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f33fc0096a0 tx=0x7f33fc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:51:55.944 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:55.943+0000 7f33f37fe700 1 -- 172.21.15.137:0/454012023 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f34000095f0 con 0x7f340c0f5410 2023-11-15T01:51:56.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:51:55 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:51:54.711120+0000 mgr.smithi055.szwngw (mgr.14164) 67 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:51:56.150 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:56.149+0000 7f341133a700 1 -- 172.21.15.137:0/454012023 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f340c045ad0 con 0x7f340c0f5410 2023-11-15T01:51:56.150 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:56.149+0000 7f33f37fe700 1 -- 172.21.15.137:0/454012023 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f340001db70 con 0x7f340c0f5410 2023-11-15T01:51:56.151 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:51:56.151 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:51:56.152 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:56.151+0000 7f341133a700 1 -- 172.21.15.137:0/454012023 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f33f402ff60 msgr2=0x7f33f4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:51:56.152 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:56.151+0000 7f341133a700 1 --2- 172.21.15.137:0/454012023 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f33f402ff60 0x7f33f4032420 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f33fc0096a0 tx=0x7f33fc006b40).stop 2023-11-15T01:51:56.152 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:56.151+0000 7f341133a700 1 -- 172.21.15.137:0/454012023 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f340c0f5410 msgr2=0x7f340c0fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:51:56.152 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:56.151+0000 7f341133a700 1 --2- 172.21.15.137:0/454012023 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f340c0f5410 0x7f340c0fd3a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f3400013510 tx=0x7f340000b040).stop 2023-11-15T01:51:56.152 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:56.151+0000 7f341133a700 1 -- 172.21.15.137:0/454012023 shutdown_connections 2023-11-15T01:51:56.153 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:56.151+0000 7f341133a700 1 --2- 172.21.15.137:0/454012023 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f33f402ff60 0x7f33f4032420 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:51:56.153 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:56.151+0000 7f341133a700 1 --2- 172.21.15.137:0/454012023 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f340c0f5410 0x7f340c0fd3a0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:51:56.153 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:56.151+0000 7f341133a700 1 -- 172.21.15.137:0/454012023 >> 172.21.15.137:0/454012023 conn(0x7f340c0f0b70 msgr2=0x7f340c05fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:51:56.153 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:56.151+0000 7f341133a700 1 -- 172.21.15.137:0/454012023 shutdown_connections 2023-11-15T01:51:56.153 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:56.151+0000 7f341133a700 1 -- 172.21.15.137:0/454012023 wait complete. 2023-11-15T01:51:56.154 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:51:57.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:51:56 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:51:56.150637+0000 mon.smithi055 (mon.0) 157 : audit [DBG] from='client.? 172.21.15.137:0/454012023' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:51:57.534 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:51:57.534 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:51:58.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:51:57 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:51:56.711483+0000 mgr.smithi055.szwngw (mgr.14164) 68 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:51:58.730 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:58.728+0000 7f757359e700 1 -- 172.21.15.137:0/3039049517 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f75740f51a0 msgr2=0x7f75740f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:51:58.730 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:58.728+0000 7f757359e700 1 --2- 172.21.15.137:0/3039049517 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f75740f51a0 0x7f75740f55c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f7564009fb0 tx=0x7f7564009b30).stop 2023-11-15T01:51:58.731 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:58.728+0000 7f757359e700 1 -- 172.21.15.137:0/3039049517 shutdown_connections 2023-11-15T01:51:58.731 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:58.728+0000 7f757359e700 1 --2- 172.21.15.137:0/3039049517 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f75740f51a0 0x7f75740f55c0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:51:58.731 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:58.728+0000 7f757359e700 1 -- 172.21.15.137:0/3039049517 >> 172.21.15.137:0/3039049517 conn(0x7f75740f09a0 msgr2=0x7f75740f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:51:58.731 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:58.728+0000 7f757359e700 1 -- 172.21.15.137:0/3039049517 shutdown_connections 2023-11-15T01:51:58.732 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:58.728+0000 7f757359e700 1 -- 172.21.15.137:0/3039049517 wait complete. 2023-11-15T01:51:58.732 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:58.728+0000 7f757359e700 1 Processor -- start 2023-11-15T01:51:58.732 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:58.728+0000 7f757359e700 1 -- start start 2023-11-15T01:51:58.732 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:58.728+0000 7f757359e700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f75740f51a0 0x7f75741064a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:51:58.732 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:58.728+0000 7f757359e700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7574106990 con 0x7f75740f51a0 2023-11-15T01:51:58.733 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:58.729+0000 7f757259c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f75740f51a0 0x7f75741064a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:51:58.733 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:58.729+0000 7f757259c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f75740f51a0 0x7f75741064a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:59828/0 (socket says 172.21.15.137:59828) 2023-11-15T01:51:58.733 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:58.729+0000 7f757259c700 1 -- 172.21.15.137:0/365518587 learned_addr learned my addr 172.21.15.137:0/365518587 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:51:58.733 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:58.729+0000 7f757259c700 1 -- 172.21.15.137:0/365518587 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7564005040 con 0x7f75740f51a0 2023-11-15T01:51:58.734 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:58.729+0000 7f757259c700 1 --2- 172.21.15.137:0/365518587 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f75740f51a0 0x7f75741064a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f7564013510 tx=0x7f756400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:51:58.734 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:58.729+0000 7f7562ffd700 1 -- 172.21.15.137:0/365518587 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7564009320 con 0x7f75740f51a0 2023-11-15T01:51:58.734 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:58.729+0000 7f757359e700 1 -- 172.21.15.137:0/365518587 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7574106b90 con 0x7f75740f51a0 2023-11-15T01:51:58.734 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:58.729+0000 7f7562ffd700 1 -- 172.21.15.137:0/365518587 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7564013f00 con 0x7f75740f51a0 2023-11-15T01:51:58.734 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:58.729+0000 7f7562ffd700 1 -- 172.21.15.137:0/365518587 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7564014dc0 con 0x7f75740f51a0 2023-11-15T01:51:58.735 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:58.729+0000 7f757359e700 1 -- 172.21.15.137:0/365518587 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7574107020 con 0x7f75740f51a0 2023-11-15T01:51:58.736 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:58.730+0000 7f7562ffd700 1 -- 172.21.15.137:0/365518587 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f75640095f0 con 0x7f75740f51a0 2023-11-15T01:51:58.736 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:58.730+0000 7f757359e700 1 -- 172.21.15.137:0/365518587 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7574045ad0 con 0x7f75740f51a0 2023-11-15T01:51:58.736 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:58.730+0000 7f7562ffd700 1 --2- 172.21.15.137:0/365518587 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f755c02ff60 0x7f755c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:51:58.736 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:58.730+0000 7f7562ffd700 1 -- 172.21.15.137:0/365518587 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7564041000 con 0x7f75740f51a0 2023-11-15T01:51:58.737 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:58.731+0000 7f7571d9b700 1 --2- 172.21.15.137:0/365518587 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f755c02ff60 0x7f755c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:51:58.737 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:58.731+0000 7f7571d9b700 1 --2- 172.21.15.137:0/365518587 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f755c02ff60 0x7f755c032420 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f75680096a0 tx=0x7f7568006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:51:58.737 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:58.735+0000 7f7562ffd700 1 -- 172.21.15.137:0/365518587 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7564014420 con 0x7f75740f51a0 2023-11-15T01:51:58.942 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:58.941+0000 7f757359e700 1 -- 172.21.15.137:0/365518587 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7574107ed0 con 0x7f75740f51a0 2023-11-15T01:51:58.943 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:58.942+0000 7f7562ffd700 1 -- 172.21.15.137:0/365518587 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7564010230 con 0x7f75740f51a0 2023-11-15T01:51:58.943 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:51:58.943 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:51:58.944 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:58.943+0000 7f757359e700 1 -- 172.21.15.137:0/365518587 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f755c02ff60 msgr2=0x7f755c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:51:58.945 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:58.943+0000 7f757359e700 1 --2- 172.21.15.137:0/365518587 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f755c02ff60 0x7f755c032420 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f75680096a0 tx=0x7f7568006b40).stop 2023-11-15T01:51:58.945 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:58.943+0000 7f757359e700 1 -- 172.21.15.137:0/365518587 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f75740f51a0 msgr2=0x7f75741064a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:51:58.945 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:58.943+0000 7f757359e700 1 --2- 172.21.15.137:0/365518587 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f75740f51a0 0x7f75741064a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f7564013510 tx=0x7f756400b040).stop 2023-11-15T01:51:58.945 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:58.943+0000 7f757359e700 1 -- 172.21.15.137:0/365518587 shutdown_connections 2023-11-15T01:51:58.945 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:58.943+0000 7f757359e700 1 --2- 172.21.15.137:0/365518587 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f755c02ff60 0x7f755c032420 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:51:58.946 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:58.943+0000 7f757359e700 1 --2- 172.21.15.137:0/365518587 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f75740f51a0 0x7f75741064a0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:51:58.946 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:58.943+0000 7f757359e700 1 -- 172.21.15.137:0/365518587 >> 172.21.15.137:0/365518587 conn(0x7f75740f09a0 msgr2=0x7f757405fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:51:58.946 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:58.943+0000 7f757359e700 1 -- 172.21.15.137:0/365518587 shutdown_connections 2023-11-15T01:51:58.946 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:51:58.943+0000 7f757359e700 1 -- 172.21.15.137:0/365518587 wait complete. 2023-11-15T01:51:58.946 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:52:00.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:51:59 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:51:58.711850+0000 mgr.smithi055.szwngw (mgr.14164) 69 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:52:00.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:51:59 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:51:58.942971+0000 mon.smithi055 (mon.0) 158 : audit [DBG] from='client.? 172.21.15.137:0/365518587' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:52:00.334 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:52:00.335 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:52:01.484 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:01.483+0000 7f6f44856700 1 -- 172.21.15.137:0/3380466457 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6f400f51a0 msgr2=0x7f6f400f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:52:01.485 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:01.483+0000 7f6f44856700 1 --2- 172.21.15.137:0/3380466457 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6f400f51a0 0x7f6f400f55c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f6f34009fb0 tx=0x7f6f34009b30).stop 2023-11-15T01:52:01.485 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:01.483+0000 7f6f44856700 1 -- 172.21.15.137:0/3380466457 shutdown_connections 2023-11-15T01:52:01.485 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:01.483+0000 7f6f44856700 1 --2- 172.21.15.137:0/3380466457 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6f400f51a0 0x7f6f400f55c0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:52:01.485 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:01.483+0000 7f6f44856700 1 -- 172.21.15.137:0/3380466457 >> 172.21.15.137:0/3380466457 conn(0x7f6f400f09a0 msgr2=0x7f6f400f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:52:01.486 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:01.484+0000 7f6f44856700 1 -- 172.21.15.137:0/3380466457 shutdown_connections 2023-11-15T01:52:01.486 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:01.484+0000 7f6f44856700 1 -- 172.21.15.137:0/3380466457 wait complete. 2023-11-15T01:52:01.488 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:01.484+0000 7f6f44856700 1 Processor -- start 2023-11-15T01:52:01.488 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:01.484+0000 7f6f44856700 1 -- start start 2023-11-15T01:52:01.488 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:01.484+0000 7f6f44856700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6f400f51a0 0x7f6f401053e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:52:01.488 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:01.484+0000 7f6f44856700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f401058d0 con 0x7f6f400f51a0 2023-11-15T01:52:01.488 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:01.485+0000 7f6f3ed9d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6f400f51a0 0x7f6f401053e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:52:01.489 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:01.485+0000 7f6f3ed9d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6f400f51a0 0x7f6f401053e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:59840/0 (socket says 172.21.15.137:59840) 2023-11-15T01:52:01.489 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:01.485+0000 7f6f3ed9d700 1 -- 172.21.15.137:0/261456533 learned_addr learned my addr 172.21.15.137:0/261456533 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:52:01.489 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:01.485+0000 7f6f3ed9d700 1 -- 172.21.15.137:0/261456533 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f34005040 con 0x7f6f400f51a0 2023-11-15T01:52:01.489 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:01.485+0000 7f6f3ed9d700 1 --2- 172.21.15.137:0/261456533 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6f400f51a0 0x7f6f401053e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f6f34013510 tx=0x7f6f3400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:52:01.490 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:01.485+0000 7f6f277fe700 1 -- 172.21.15.137:0/261456533 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f34009320 con 0x7f6f400f51a0 2023-11-15T01:52:01.490 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:01.485+0000 7f6f44856700 1 -- 172.21.15.137:0/261456533 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f40105ad0 con 0x7f6f400f51a0 2023-11-15T01:52:01.490 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:01.485+0000 7f6f277fe700 1 -- 172.21.15.137:0/261456533 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6f34013f00 con 0x7f6f400f51a0 2023-11-15T01:52:01.490 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:01.485+0000 7f6f277fe700 1 -- 172.21.15.137:0/261456533 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f34014dc0 con 0x7f6f400f51a0 2023-11-15T01:52:01.490 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:01.485+0000 7f6f44856700 1 -- 172.21.15.137:0/261456533 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f40105f60 con 0x7f6f400f51a0 2023-11-15T01:52:01.491 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:01.486+0000 7f6f277fe700 1 -- 172.21.15.137:0/261456533 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6f340095f0 con 0x7f6f400f51a0 2023-11-15T01:52:01.491 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:01.486+0000 7f6f44856700 1 -- 172.21.15.137:0/261456533 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f40045ad0 con 0x7f6f400f51a0 2023-11-15T01:52:01.491 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:01.486+0000 7f6f277fe700 1 --2- 172.21.15.137:0/261456533 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f6f2802ff60 0x7f6f28032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:52:01.491 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:01.486+0000 7f6f277fe700 1 -- 172.21.15.137:0/261456533 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6f34041080 con 0x7f6f400f51a0 2023-11-15T01:52:01.491 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:01.486+0000 7f6f3e59c700 1 --2- 172.21.15.137:0/261456533 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f6f2802ff60 0x7f6f28032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:52:01.492 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:01.487+0000 7f6f3e59c700 1 --2- 172.21.15.137:0/261456533 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f6f2802ff60 0x7f6f28032420 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f6f300096a0 tx=0x7f6f30006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:52:01.492 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:01.491+0000 7f6f277fe700 1 -- 172.21.15.137:0/261456533 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6f34014420 con 0x7f6f400f51a0 2023-11-15T01:52:01.699 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:01.698+0000 7f6f44856700 1 -- 172.21.15.137:0/261456533 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6f40106e10 con 0x7f6f400f51a0 2023-11-15T01:52:01.700 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:01.699+0000 7f6f277fe700 1 -- 172.21.15.137:0/261456533 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6f34010230 con 0x7f6f400f51a0 2023-11-15T01:52:01.700 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:52:01.700 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:52:01.701 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:01.701+0000 7f6f44856700 1 -- 172.21.15.137:0/261456533 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f6f2802ff60 msgr2=0x7f6f28032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:52:01.702 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:01.701+0000 7f6f44856700 1 --2- 172.21.15.137:0/261456533 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f6f2802ff60 0x7f6f28032420 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f6f300096a0 tx=0x7f6f30006b40).stop 2023-11-15T01:52:01.702 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:01.701+0000 7f6f44856700 1 -- 172.21.15.137:0/261456533 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6f400f51a0 msgr2=0x7f6f401053e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:52:01.702 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:01.701+0000 7f6f44856700 1 --2- 172.21.15.137:0/261456533 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6f400f51a0 0x7f6f401053e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f6f34013510 tx=0x7f6f3400b040).stop 2023-11-15T01:52:01.702 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:01.701+0000 7f6f44856700 1 -- 172.21.15.137:0/261456533 shutdown_connections 2023-11-15T01:52:01.702 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:01.701+0000 7f6f44856700 1 --2- 172.21.15.137:0/261456533 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f6f2802ff60 0x7f6f28032420 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:52:01.703 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:01.701+0000 7f6f44856700 1 --2- 172.21.15.137:0/261456533 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6f400f51a0 0x7f6f401053e0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:52:01.703 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:01.701+0000 7f6f44856700 1 -- 172.21.15.137:0/261456533 >> 172.21.15.137:0/261456533 conn(0x7f6f400f09a0 msgr2=0x7f6f400f13e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:52:01.703 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:01.701+0000 7f6f44856700 1 -- 172.21.15.137:0/261456533 shutdown_connections 2023-11-15T01:52:01.703 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:01.701+0000 7f6f44856700 1 -- 172.21.15.137:0/261456533 wait complete. 2023-11-15T01:52:01.704 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:52:01.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:52:01 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:52:00.712176+0000 mgr.smithi055.szwngw (mgr.14164) 70 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:52:02.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:52:02 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:52:01.699978+0000 mon.smithi055 (mon.0) 159 : audit [DBG] from='client.? 172.21.15.137:0/261456533' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:52:03.099 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:52:03.100 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:52:03.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:52:03 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:52:02.712501+0000 mgr.smithi055.szwngw (mgr.14164) 71 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:52:04.219 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:04.215+0000 7f85e7fb2700 1 -- 172.21.15.137:0/1240967901 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f85e00f5410 msgr2=0x7f85e00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:52:04.219 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:04.215+0000 7f85e7fb2700 1 --2- 172.21.15.137:0/1240967901 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f85e00f5410 0x7f85e00f5830 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f85d0009fb0 tx=0x7f85d0009b30).stop 2023-11-15T01:52:04.220 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:04.215+0000 7f85e7fb2700 1 -- 172.21.15.137:0/1240967901 shutdown_connections 2023-11-15T01:52:04.220 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:04.215+0000 7f85e7fb2700 1 --2- 172.21.15.137:0/1240967901 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f85e00f5410 0x7f85e00f5830 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:52:04.220 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:04.215+0000 7f85e7fb2700 1 -- 172.21.15.137:0/1240967901 >> 172.21.15.137:0/1240967901 conn(0x7f85e00f0b70 msgr2=0x7f85e00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:52:04.220 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:04.215+0000 7f85e7fb2700 1 -- 172.21.15.137:0/1240967901 shutdown_connections 2023-11-15T01:52:04.220 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:04.215+0000 7f85e7fb2700 1 -- 172.21.15.137:0/1240967901 wait complete. 2023-11-15T01:52:04.221 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:04.216+0000 7f85e7fb2700 1 Processor -- start 2023-11-15T01:52:04.221 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:04.216+0000 7f85e7fb2700 1 -- start start 2023-11-15T01:52:04.221 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:04.216+0000 7f85e7fb2700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f85e00f5410 0x7f85e00fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:52:04.221 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:04.216+0000 7f85e7fb2700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85e00fb820 con 0x7f85e00f5410 2023-11-15T01:52:04.221 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:04.216+0000 7f85e5d4e700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f85e00f5410 0x7f85e00fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:52:04.222 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:04.216+0000 7f85e5d4e700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f85e00f5410 0x7f85e00fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:59868/0 (socket says 172.21.15.137:59868) 2023-11-15T01:52:04.222 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:04.216+0000 7f85e5d4e700 1 -- 172.21.15.137:0/2914437149 learned_addr learned my addr 172.21.15.137:0/2914437149 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:52:04.222 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:04.216+0000 7f85e5d4e700 1 -- 172.21.15.137:0/2914437149 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85d0005040 con 0x7f85e00f5410 2023-11-15T01:52:04.222 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:04.216+0000 7f85e5d4e700 1 --2- 172.21.15.137:0/2914437149 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f85e00f5410 0x7f85e00fd3a0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f85d0013510 tx=0x7f85d000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:52:04.223 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:04.217+0000 7f85d67fc700 1 -- 172.21.15.137:0/2914437149 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f85d0009320 con 0x7f85e00f5410 2023-11-15T01:52:04.223 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:04.217+0000 7f85e7fb2700 1 -- 172.21.15.137:0/2914437149 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85e00fb9c0 con 0x7f85e00f5410 2023-11-15T01:52:04.223 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:04.217+0000 7f85d67fc700 1 -- 172.21.15.137:0/2914437149 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f85d0013f00 con 0x7f85e00f5410 2023-11-15T01:52:04.223 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:04.217+0000 7f85d67fc700 1 -- 172.21.15.137:0/2914437149 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f85d0014d90 con 0x7f85e00f5410 2023-11-15T01:52:04.223 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:04.217+0000 7f85e7fb2700 1 -- 172.21.15.137:0/2914437149 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85e00fbe50 con 0x7f85e00f5410 2023-11-15T01:52:04.224 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:04.217+0000 7f85e7fb2700 1 -- 172.21.15.137:0/2914437149 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85e0059100 con 0x7f85e00f5410 2023-11-15T01:52:04.224 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:04.218+0000 7f85d67fc700 1 -- 172.21.15.137:0/2914437149 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f85d00095f0 con 0x7f85e00f5410 2023-11-15T01:52:04.224 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:04.218+0000 7f85d67fc700 1 --2- 172.21.15.137:0/2914437149 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f85cc02ff60 0x7f85cc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:52:04.224 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:04.218+0000 7f85d67fc700 1 -- 172.21.15.137:0/2914437149 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f85d0041000 con 0x7f85e00f5410 2023-11-15T01:52:04.224 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:04.218+0000 7f85e554d700 1 --2- 172.21.15.137:0/2914437149 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f85cc02ff60 0x7f85cc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:52:04.225 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:04.218+0000 7f85e554d700 1 --2- 172.21.15.137:0/2914437149 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f85cc02ff60 0x7f85cc032420 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f85dc0096a0 tx=0x7f85dc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:52:04.225 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:04.223+0000 7f85d67fc700 1 -- 172.21.15.137:0/2914437149 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f85d0014420 con 0x7f85e00f5410 2023-11-15T01:52:04.429 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:04.429+0000 7f85e7fb2700 1 -- 172.21.15.137:0/2914437149 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f85e00fae60 con 0x7f85e00f5410 2023-11-15T01:52:04.430 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:04.429+0000 7f85d67fc700 1 -- 172.21.15.137:0/2914437149 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f85d0010230 con 0x7f85e00f5410 2023-11-15T01:52:04.431 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:52:04.431 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:52:04.432 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:04.431+0000 7f85e7fb2700 1 -- 172.21.15.137:0/2914437149 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f85cc02ff60 msgr2=0x7f85cc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:52:04.432 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:04.431+0000 7f85e7fb2700 1 --2- 172.21.15.137:0/2914437149 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f85cc02ff60 0x7f85cc032420 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f85dc0096a0 tx=0x7f85dc006b40).stop 2023-11-15T01:52:04.432 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:04.431+0000 7f85e7fb2700 1 -- 172.21.15.137:0/2914437149 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f85e00f5410 msgr2=0x7f85e00fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:52:04.432 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:04.431+0000 7f85e7fb2700 1 --2- 172.21.15.137:0/2914437149 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f85e00f5410 0x7f85e00fd3a0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f85d0013510 tx=0x7f85d000b040).stop 2023-11-15T01:52:04.433 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:04.431+0000 7f85e7fb2700 1 -- 172.21.15.137:0/2914437149 shutdown_connections 2023-11-15T01:52:04.433 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:04.431+0000 7f85e7fb2700 1 --2- 172.21.15.137:0/2914437149 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f85cc02ff60 0x7f85cc032420 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:52:04.433 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:04.431+0000 7f85e7fb2700 1 --2- 172.21.15.137:0/2914437149 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f85e00f5410 0x7f85e00fd3a0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:52:04.433 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:04.431+0000 7f85e7fb2700 1 -- 172.21.15.137:0/2914437149 >> 172.21.15.137:0/2914437149 conn(0x7f85e00f0b70 msgr2=0x7f85e005fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:52:04.433 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:04.431+0000 7f85e7fb2700 1 -- 172.21.15.137:0/2914437149 shutdown_connections 2023-11-15T01:52:04.434 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:04.431+0000 7f85e7fb2700 1 -- 172.21.15.137:0/2914437149 wait complete. 2023-11-15T01:52:04.434 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:52:04.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:52:04 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:52:04.430587+0000 mon.smithi055 (mon.0) 160 : audit [DBG] from='client.? 172.21.15.137:0/2914437149' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:52:05.818 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:52:05.819 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:52:05.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:52:05 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:52:04.712836+0000 mgr.smithi055.szwngw (mgr.14164) 72 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:52:06.956 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:06.955+0000 7faba39b3700 1 -- 172.21.15.137:0/1451827546 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fab9c0f6990 msgr2=0x7fab9c0f6db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:52:06.956 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:06.955+0000 7faba39b3700 1 --2- 172.21.15.137:0/1451827546 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fab9c0f6990 0x7fab9c0f6db0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7fab98009fb0 tx=0x7fab98009b30).stop 2023-11-15T01:52:06.956 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:06.955+0000 7faba39b3700 1 -- 172.21.15.137:0/1451827546 shutdown_connections 2023-11-15T01:52:06.957 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:06.955+0000 7faba39b3700 1 --2- 172.21.15.137:0/1451827546 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fab9c0f6990 0x7fab9c0f6db0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:52:06.957 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:06.955+0000 7faba39b3700 1 -- 172.21.15.137:0/1451827546 >> 172.21.15.137:0/1451827546 conn(0x7fab9c0f0260 msgr2=0x7fab9c0f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:52:06.957 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:06.955+0000 7faba39b3700 1 -- 172.21.15.137:0/1451827546 shutdown_connections 2023-11-15T01:52:06.957 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:06.955+0000 7faba39b3700 1 -- 172.21.15.137:0/1451827546 wait complete. 2023-11-15T01:52:06.959 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:06.955+0000 7faba39b3700 1 Processor -- start 2023-11-15T01:52:06.960 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:06.956+0000 7faba39b3700 1 -- start start 2023-11-15T01:52:06.960 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:06.956+0000 7faba39b3700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fab9c0f6990 0x7fab9c109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:52:06.960 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:06.956+0000 7faba39b3700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab9c0f52d0 con 0x7fab9c0f6990 2023-11-15T01:52:06.961 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:06.956+0000 7faba174f700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fab9c0f6990 0x7fab9c109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:52:06.961 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:06.956+0000 7faba174f700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fab9c0f6990 0x7fab9c109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:59896/0 (socket says 172.21.15.137:59896) 2023-11-15T01:52:06.961 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:06.956+0000 7faba174f700 1 -- 172.21.15.137:0/2385642174 learned_addr learned my addr 172.21.15.137:0/2385642174 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:52:06.961 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:06.956+0000 7faba174f700 1 -- 172.21.15.137:0/2385642174 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab98005040 con 0x7fab9c0f6990 2023-11-15T01:52:06.961 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:06.957+0000 7faba174f700 1 --2- 172.21.15.137:0/2385642174 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fab9c0f6990 0x7fab9c109430 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fab98014000 tx=0x7fab9800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:52:06.962 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:06.957+0000 7fab91ffb700 1 -- 172.21.15.137:0/2385642174 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab980095f0 con 0x7fab9c0f6990 2023-11-15T01:52:06.962 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:06.957+0000 7faba39b3700 1 -- 172.21.15.137:0/2385642174 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab9c0f54d0 con 0x7fab9c0f6990 2023-11-15T01:52:06.962 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:06.957+0000 7fab91ffb700 1 -- 172.21.15.137:0/2385642174 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fab98007e70 con 0x7fab9c0f6990 2023-11-15T01:52:06.962 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:06.957+0000 7fab91ffb700 1 -- 172.21.15.137:0/2385642174 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab98014ea0 con 0x7fab9c0f6990 2023-11-15T01:52:06.963 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:06.957+0000 7faba39b3700 1 -- 172.21.15.137:0/2385642174 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab9c109ae0 con 0x7fab9c0f6990 2023-11-15T01:52:06.963 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:06.958+0000 7fab91ffb700 1 -- 172.21.15.137:0/2385642174 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fab98025070 con 0x7fab9c0f6990 2023-11-15T01:52:06.963 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:06.958+0000 7faba39b3700 1 -- 172.21.15.137:0/2385642174 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab9c045ad0 con 0x7fab9c0f6990 2023-11-15T01:52:06.963 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:06.958+0000 7fab91ffb700 1 --2- 172.21.15.137:0/2385642174 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fab8802ff60 0x7fab88032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:52:06.964 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:06.958+0000 7fab91ffb700 1 -- 172.21.15.137:0/2385642174 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fab98040970 con 0x7fab9c0f6990 2023-11-15T01:52:06.964 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:06.958+0000 7faba0f4e700 1 --2- 172.21.15.137:0/2385642174 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fab8802ff60 0x7fab88032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:52:06.964 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:06.959+0000 7faba0f4e700 1 --2- 172.21.15.137:0/2385642174 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fab8802ff60 0x7fab88032420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7fab8c0096a0 tx=0x7fab8c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:52:06.965 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:06.963+0000 7fab91ffb700 1 -- 172.21.15.137:0/2385642174 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fab980079c0 con 0x7fab9c0f6990 2023-11-15T01:52:07.173 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:07.172+0000 7faba39b3700 1 -- 172.21.15.137:0/2385642174 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fab9c0fae60 con 0x7fab9c0f6990 2023-11-15T01:52:07.174 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:07.173+0000 7fab91ffb700 1 -- 172.21.15.137:0/2385642174 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fab9800f300 con 0x7fab9c0f6990 2023-11-15T01:52:07.174 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:52:07.174 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:52:07.175 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:07.174+0000 7faba39b3700 1 -- 172.21.15.137:0/2385642174 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fab8802ff60 msgr2=0x7fab88032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:52:07.175 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:07.174+0000 7faba39b3700 1 --2- 172.21.15.137:0/2385642174 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fab8802ff60 0x7fab88032420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7fab8c0096a0 tx=0x7fab8c006b40).stop 2023-11-15T01:52:07.176 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:07.174+0000 7faba39b3700 1 -- 172.21.15.137:0/2385642174 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fab9c0f6990 msgr2=0x7fab9c109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:52:07.176 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:07.174+0000 7faba39b3700 1 --2- 172.21.15.137:0/2385642174 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fab9c0f6990 0x7fab9c109430 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fab98014000 tx=0x7fab9800b040).stop 2023-11-15T01:52:07.176 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:07.175+0000 7faba39b3700 1 -- 172.21.15.137:0/2385642174 shutdown_connections 2023-11-15T01:52:07.176 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:07.175+0000 7faba39b3700 1 --2- 172.21.15.137:0/2385642174 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fab8802ff60 0x7fab88032420 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:52:07.177 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:07.175+0000 7faba39b3700 1 --2- 172.21.15.137:0/2385642174 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fab9c0f6990 0x7fab9c109430 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:52:07.177 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:07.175+0000 7faba39b3700 1 -- 172.21.15.137:0/2385642174 >> 172.21.15.137:0/2385642174 conn(0x7fab9c0f0260 msgr2=0x7fab9c0f0d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:52:07.177 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:07.175+0000 7faba39b3700 1 -- 172.21.15.137:0/2385642174 shutdown_connections 2023-11-15T01:52:07.177 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:07.175+0000 7faba39b3700 1 -- 172.21.15.137:0/2385642174 wait complete. 2023-11-15T01:52:07.178 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:52:08.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:52:07 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:52:06.713173+0000 mgr.smithi055.szwngw (mgr.14164) 73 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:52:08.079 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:52:07 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:52:07.174111+0000 mon.smithi055 (mon.0) 161 : audit [DBG] from='client.? 172.21.15.137:0/2385642174' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:52:08.556 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:52:08.557 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:52:09.693 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:09.691+0000 7ff9ab471700 1 -- 172.21.15.137:0/3220536921 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff9a40f5410 msgr2=0x7ff9a40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:52:09.694 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:09.691+0000 7ff9ab471700 1 --2- 172.21.15.137:0/3220536921 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff9a40f5410 0x7ff9a40f5830 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7ff998009fb0 tx=0x7ff998009b30).stop 2023-11-15T01:52:09.694 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:09.691+0000 7ff9ab471700 1 -- 172.21.15.137:0/3220536921 shutdown_connections 2023-11-15T01:52:09.694 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:09.691+0000 7ff9ab471700 1 --2- 172.21.15.137:0/3220536921 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff9a40f5410 0x7ff9a40f5830 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:52:09.697 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:09.691+0000 7ff9ab471700 1 -- 172.21.15.137:0/3220536921 >> 172.21.15.137:0/3220536921 conn(0x7ff9a40f0b70 msgr2=0x7ff9a40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:52:09.698 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:09.691+0000 7ff9ab471700 1 -- 172.21.15.137:0/3220536921 shutdown_connections 2023-11-15T01:52:09.698 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:09.691+0000 7ff9ab471700 1 -- 172.21.15.137:0/3220536921 wait complete. 2023-11-15T01:52:09.698 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:09.691+0000 7ff9ab471700 1 Processor -- start 2023-11-15T01:52:09.698 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:09.691+0000 7ff9ab471700 1 -- start start 2023-11-15T01:52:09.698 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:09.691+0000 7ff9ab471700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff9a40f5410 0x7ff9a40fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:52:09.699 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:09.691+0000 7ff9ab471700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9a40fb820 con 0x7ff9a40f5410 2023-11-15T01:52:09.699 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:09.692+0000 7ff9a920d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff9a40f5410 0x7ff9a40fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:52:09.699 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:09.692+0000 7ff9a920d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff9a40f5410 0x7ff9a40fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:41114/0 (socket says 172.21.15.137:41114) 2023-11-15T01:52:09.699 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:09.692+0000 7ff9a920d700 1 -- 172.21.15.137:0/2790812507 learned_addr learned my addr 172.21.15.137:0/2790812507 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:52:09.699 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:09.692+0000 7ff9a920d700 1 -- 172.21.15.137:0/2790812507 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff998005040 con 0x7ff9a40f5410 2023-11-15T01:52:09.700 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:09.692+0000 7ff9a920d700 1 --2- 172.21.15.137:0/2790812507 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff9a40f5410 0x7ff9a40fd3a0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7ff998013510 tx=0x7ff99800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:52:09.701 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:09.692+0000 7ff995ffb700 1 -- 172.21.15.137:0/2790812507 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff998009320 con 0x7ff9a40f5410 2023-11-15T01:52:09.701 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:09.692+0000 7ff9ab471700 1 -- 172.21.15.137:0/2790812507 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9a40fb9c0 con 0x7ff9a40f5410 2023-11-15T01:52:09.702 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:09.692+0000 7ff995ffb700 1 -- 172.21.15.137:0/2790812507 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff998013f00 con 0x7ff9a40f5410 2023-11-15T01:52:09.702 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:09.692+0000 7ff995ffb700 1 -- 172.21.15.137:0/2790812507 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff998014d90 con 0x7ff9a40f5410 2023-11-15T01:52:09.702 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:09.693+0000 7ff9ab471700 1 -- 172.21.15.137:0/2790812507 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9a40fbe50 con 0x7ff9a40f5410 2023-11-15T01:52:09.702 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:09.693+0000 7ff995ffb700 1 -- 172.21.15.137:0/2790812507 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff998014420 con 0x7ff9a40f5410 2023-11-15T01:52:09.702 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:09.693+0000 7ff9ab471700 1 -- 172.21.15.137:0/2790812507 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9a4059100 con 0x7ff9a40f5410 2023-11-15T01:52:09.703 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:09.693+0000 7ff995ffb700 1 --2- 172.21.15.137:0/2790812507 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7ff99002ff60 0x7ff990032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:52:09.703 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:09.693+0000 7ff995ffb700 1 -- 172.21.15.137:0/2790812507 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff998041110 con 0x7ff9a40f5410 2023-11-15T01:52:09.703 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:09.694+0000 7ff9a8a0c700 1 --2- 172.21.15.137:0/2790812507 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7ff99002ff60 0x7ff990032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:52:09.703 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:09.694+0000 7ff9a8a0c700 1 --2- 172.21.15.137:0/2790812507 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7ff99002ff60 0x7ff990032420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7ff9a00096a0 tx=0x7ff9a0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:52:09.704 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:09.698+0000 7ff995ffb700 1 -- 172.21.15.137:0/2790812507 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff9980095f0 con 0x7ff9a40f5410 2023-11-15T01:52:09.906 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:09.904+0000 7ff9ab471700 1 -- 172.21.15.137:0/2790812507 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff9a4045ad0 con 0x7ff9a40f5410 2023-11-15T01:52:09.907 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:09.905+0000 7ff995ffb700 1 -- 172.21.15.137:0/2790812507 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff99801db70 con 0x7ff9a40f5410 2023-11-15T01:52:09.907 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:52:09.908 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:52:09.908 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:09.907+0000 7ff9ab471700 1 -- 172.21.15.137:0/2790812507 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7ff99002ff60 msgr2=0x7ff990032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:52:09.909 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:09.907+0000 7ff9ab471700 1 --2- 172.21.15.137:0/2790812507 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7ff99002ff60 0x7ff990032420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7ff9a00096a0 tx=0x7ff9a0006b40).stop 2023-11-15T01:52:09.909 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:09.907+0000 7ff9ab471700 1 -- 172.21.15.137:0/2790812507 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff9a40f5410 msgr2=0x7ff9a40fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:52:09.909 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:09.907+0000 7ff9ab471700 1 --2- 172.21.15.137:0/2790812507 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff9a40f5410 0x7ff9a40fd3a0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7ff998013510 tx=0x7ff99800b040).stop 2023-11-15T01:52:09.909 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:09.907+0000 7ff9ab471700 1 -- 172.21.15.137:0/2790812507 shutdown_connections 2023-11-15T01:52:09.909 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:09.907+0000 7ff9ab471700 1 --2- 172.21.15.137:0/2790812507 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7ff99002ff60 0x7ff990032420 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:52:09.910 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:09.907+0000 7ff9ab471700 1 --2- 172.21.15.137:0/2790812507 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff9a40f5410 0x7ff9a40fd3a0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:52:09.910 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:09.907+0000 7ff9ab471700 1 -- 172.21.15.137:0/2790812507 >> 172.21.15.137:0/2790812507 conn(0x7ff9a40f0b70 msgr2=0x7ff9a405fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:52:09.910 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:09.907+0000 7ff9ab471700 1 -- 172.21.15.137:0/2790812507 shutdown_connections 2023-11-15T01:52:09.910 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:09.907+0000 7ff9ab471700 1 -- 172.21.15.137:0/2790812507 wait complete. 2023-11-15T01:52:09.911 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:52:10.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:52:09 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:52:08.713505+0000 mgr.smithi055.szwngw (mgr.14164) 74 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:52:11.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:52:10 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:52:09.907129+0000 mon.smithi055 (mon.0) 162 : audit [DBG] from='client.? 172.21.15.137:0/2790812507' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:52:11.296 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:52:11.297 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:52:12.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:52:11 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:52:10.713752+0000 mgr.smithi055.szwngw (mgr.14164) 75 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:52:12.499 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:12.496+0000 7fcafd1ef700 1 -- 172.21.15.137:0/2901484910 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcaf80f5410 msgr2=0x7fcaf80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:52:12.499 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:12.496+0000 7fcafd1ef700 1 --2- 172.21.15.137:0/2901484910 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcaf80f5410 0x7fcaf80f5830 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7fcaec009fb0 tx=0x7fcaec009b30).stop 2023-11-15T01:52:12.500 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:12.496+0000 7fcafd1ef700 1 -- 172.21.15.137:0/2901484910 shutdown_connections 2023-11-15T01:52:12.500 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:12.496+0000 7fcafd1ef700 1 --2- 172.21.15.137:0/2901484910 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcaf80f5410 0x7fcaf80f5830 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:52:12.500 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:12.496+0000 7fcafd1ef700 1 -- 172.21.15.137:0/2901484910 >> 172.21.15.137:0/2901484910 conn(0x7fcaf80f0b70 msgr2=0x7fcaf80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:52:12.500 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:12.496+0000 7fcafd1ef700 1 -- 172.21.15.137:0/2901484910 shutdown_connections 2023-11-15T01:52:12.501 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:12.496+0000 7fcafd1ef700 1 -- 172.21.15.137:0/2901484910 wait complete. 2023-11-15T01:52:12.501 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:12.496+0000 7fcafd1ef700 1 Processor -- start 2023-11-15T01:52:12.501 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:12.496+0000 7fcafd1ef700 1 -- start start 2023-11-15T01:52:12.501 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:12.496+0000 7fcafd1ef700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcaf80f5410 0x7fcaf8105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:52:12.501 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:12.496+0000 7fcafd1ef700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf8105b20 con 0x7fcaf80f5410 2023-11-15T01:52:12.502 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:12.497+0000 7fcaf6d9d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcaf80f5410 0x7fcaf8105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:52:12.502 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:12.497+0000 7fcaf6d9d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcaf80f5410 0x7fcaf8105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:41124/0 (socket says 172.21.15.137:41124) 2023-11-15T01:52:12.502 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:12.497+0000 7fcaf6d9d700 1 -- 172.21.15.137:0/2094526131 learned_addr learned my addr 172.21.15.137:0/2094526131 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:52:12.502 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:12.497+0000 7fcaf6d9d700 1 -- 172.21.15.137:0/2094526131 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaec005040 con 0x7fcaf80f5410 2023-11-15T01:52:12.502 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:12.497+0000 7fcaf6d9d700 1 --2- 172.21.15.137:0/2094526131 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcaf80f5410 0x7fcaf8105630 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7fcaec013510 tx=0x7fcaec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:52:12.503 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:12.497+0000 7fcadf7fe700 1 -- 172.21.15.137:0/2094526131 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcaec009320 con 0x7fcaf80f5410 2023-11-15T01:52:12.503 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:12.497+0000 7fcafd1ef700 1 -- 172.21.15.137:0/2094526131 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcaf8105d20 con 0x7fcaf80f5410 2023-11-15T01:52:12.503 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:12.497+0000 7fcadf7fe700 1 -- 172.21.15.137:0/2094526131 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcaec013f00 con 0x7fcaf80f5410 2023-11-15T01:52:12.503 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:12.497+0000 7fcadf7fe700 1 -- 172.21.15.137:0/2094526131 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcaec014d90 con 0x7fcaf80f5410 2023-11-15T01:52:12.503 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:12.498+0000 7fcafd1ef700 1 -- 172.21.15.137:0/2094526131 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcaf81061b0 con 0x7fcaf80f5410 2023-11-15T01:52:12.504 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:12.498+0000 7fcadf7fe700 1 -- 172.21.15.137:0/2094526131 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcaec014420 con 0x7fcaf80f5410 2023-11-15T01:52:12.505 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:12.498+0000 7fcafd1ef700 1 -- 172.21.15.137:0/2094526131 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcaf8045ad0 con 0x7fcaf80f5410 2023-11-15T01:52:12.505 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:12.498+0000 7fcadf7fe700 1 --2- 172.21.15.137:0/2094526131 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fcae002ffb0 0x7fcae0032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:52:12.505 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:12.498+0000 7fcadf7fe700 1 -- 172.21.15.137:0/2094526131 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcaec0410b0 con 0x7fcaf80f5410 2023-11-15T01:52:12.505 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:12.499+0000 7fcaf659c700 1 --2- 172.21.15.137:0/2094526131 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fcae002ffb0 0x7fcae0032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:52:12.505 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:12.499+0000 7fcaf659c700 1 --2- 172.21.15.137:0/2094526131 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fcae002ffb0 0x7fcae0032470 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7fcae80096a0 tx=0x7fcae8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:52:12.506 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:12.504+0000 7fcadf7fe700 1 -- 172.21.15.137:0/2094526131 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcaec06d2b0 con 0x7fcaf80f5410 2023-11-15T01:52:12.708 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:12.707+0000 7fcafd1ef700 1 -- 172.21.15.137:0/2094526131 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcaf80fae60 con 0x7fcaf80f5410 2023-11-15T01:52:12.709 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:12.707+0000 7fcadf7fe700 1 -- 172.21.15.137:0/2094526131 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcaec0095f0 con 0x7fcaf80f5410 2023-11-15T01:52:12.709 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:52:12.710 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:52:12.711 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:12.709+0000 7fcafd1ef700 1 -- 172.21.15.137:0/2094526131 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fcae002ffb0 msgr2=0x7fcae0032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:52:12.711 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:12.709+0000 7fcafd1ef700 1 --2- 172.21.15.137:0/2094526131 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fcae002ffb0 0x7fcae0032470 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7fcae80096a0 tx=0x7fcae8006b40).stop 2023-11-15T01:52:12.711 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:12.709+0000 7fcafd1ef700 1 -- 172.21.15.137:0/2094526131 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcaf80f5410 msgr2=0x7fcaf8105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:52:12.711 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:12.709+0000 7fcafd1ef700 1 --2- 172.21.15.137:0/2094526131 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcaf80f5410 0x7fcaf8105630 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7fcaec013510 tx=0x7fcaec00b040).stop 2023-11-15T01:52:12.712 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:12.709+0000 7fcafd1ef700 1 -- 172.21.15.137:0/2094526131 shutdown_connections 2023-11-15T01:52:12.712 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:12.709+0000 7fcafd1ef700 1 --2- 172.21.15.137:0/2094526131 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fcae002ffb0 0x7fcae0032470 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:52:12.712 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:12.709+0000 7fcafd1ef700 1 --2- 172.21.15.137:0/2094526131 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcaf80f5410 0x7fcaf8105630 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:52:12.712 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:12.709+0000 7fcafd1ef700 1 -- 172.21.15.137:0/2094526131 >> 172.21.15.137:0/2094526131 conn(0x7fcaf80f0b70 msgr2=0x7fcaf80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:52:12.712 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:12.709+0000 7fcafd1ef700 1 -- 172.21.15.137:0/2094526131 shutdown_connections 2023-11-15T01:52:12.713 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:12.709+0000 7fcafd1ef700 1 -- 172.21.15.137:0/2094526131 wait complete. 2023-11-15T01:52:12.713 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:52:13.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:52:12 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:52:12.709635+0000 mon.smithi055 (mon.0) 163 : audit [DBG] from='client.? 172.21.15.137:0/2094526131' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:52:14.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:52:13 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:52:12.714106+0000 mgr.smithi055.szwngw (mgr.14164) 76 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:52:14.108 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:52:14.109 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:52:15.265 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:15.263+0000 7f7139bb1700 1 -- 172.21.15.137:0/167593861 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f71340f6510 msgr2=0x7f71340f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:52:15.266 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:15.263+0000 7f7139bb1700 1 --2- 172.21.15.137:0/167593861 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f71340f6510 0x7f71340f6930 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f711c009fb0 tx=0x7f711c009b30).stop 2023-11-15T01:52:15.266 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:15.263+0000 7f7139bb1700 1 -- 172.21.15.137:0/167593861 shutdown_connections 2023-11-15T01:52:15.266 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:15.263+0000 7f7139bb1700 1 --2- 172.21.15.137:0/167593861 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f71340f6510 0x7f71340f6930 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:52:15.266 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:15.263+0000 7f7139bb1700 1 -- 172.21.15.137:0/167593861 >> 172.21.15.137:0/167593861 conn(0x7f71340f1c70 msgr2=0x7f71340f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:52:15.267 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:15.263+0000 7f7139bb1700 1 -- 172.21.15.137:0/167593861 shutdown_connections 2023-11-15T01:52:15.267 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:15.263+0000 7f7139bb1700 1 -- 172.21.15.137:0/167593861 wait complete. 2023-11-15T01:52:15.269 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:15.264+0000 7f7139bb1700 1 Processor -- start 2023-11-15T01:52:15.270 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:15.264+0000 7f7139bb1700 1 -- start start 2023-11-15T01:52:15.270 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:15.264+0000 7f7139bb1700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f71340f6510 0x7f7134106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:52:15.270 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:15.264+0000 7f7139bb1700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7134106c20 con 0x7f71340f6510 2023-11-15T01:52:15.271 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:15.265+0000 7f71337fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f71340f6510 0x7f7134106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:52:15.271 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:15.265+0000 7f71337fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f71340f6510 0x7f7134106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:41136/0 (socket says 172.21.15.137:41136) 2023-11-15T01:52:15.271 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:15.265+0000 7f71337fe700 1 -- 172.21.15.137:0/3780301587 learned_addr learned my addr 172.21.15.137:0/3780301587 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:52:15.271 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:15.265+0000 7f71337fe700 1 -- 172.21.15.137:0/3780301587 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f711c005040 con 0x7f71340f6510 2023-11-15T01:52:15.272 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:15.265+0000 7f71337fe700 1 --2- 172.21.15.137:0/3780301587 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f71340f6510 0x7f7134106730 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f711c014000 tx=0x7f711c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:52:15.272 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:15.265+0000 7f712bfff700 1 -- 172.21.15.137:0/3780301587 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f711c0095f0 con 0x7f71340f6510 2023-11-15T01:52:15.272 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:15.265+0000 7f7139bb1700 1 -- 172.21.15.137:0/3780301587 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7134106e20 con 0x7f71340f6510 2023-11-15T01:52:15.272 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:15.265+0000 7f712bfff700 1 -- 172.21.15.137:0/3780301587 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f711c007e70 con 0x7f71340f6510 2023-11-15T01:52:15.273 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:15.265+0000 7f712bfff700 1 -- 172.21.15.137:0/3780301587 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f711c014ea0 con 0x7f71340f6510 2023-11-15T01:52:15.273 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:15.265+0000 7f7139bb1700 1 -- 172.21.15.137:0/3780301587 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71341072b0 con 0x7f71340f6510 2023-11-15T01:52:15.273 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:15.266+0000 7f712bfff700 1 -- 172.21.15.137:0/3780301587 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f711c025070 con 0x7f71340f6510 2023-11-15T01:52:15.273 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:15.266+0000 7f7139bb1700 1 -- 172.21.15.137:0/3780301587 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7134045ad0 con 0x7f71340f6510 2023-11-15T01:52:15.274 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:15.266+0000 7f712bfff700 1 --2- 172.21.15.137:0/3780301587 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f712002ff60 0x7f7120032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:52:15.274 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:15.266+0000 7f712bfff700 1 -- 172.21.15.137:0/3780301587 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f711c040a30 con 0x7f71340f6510 2023-11-15T01:52:15.274 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:15.267+0000 7f7132ffd700 1 --2- 172.21.15.137:0/3780301587 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f712002ff60 0x7f7120032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:52:15.275 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:15.267+0000 7f7132ffd700 1 --2- 172.21.15.137:0/3780301587 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f712002ff60 0x7f7120032420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f71240096a0 tx=0x7f7124006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:52:15.275 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:15.272+0000 7f712bfff700 1 -- 172.21.15.137:0/3780301587 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f711c0079c0 con 0x7f71340f6510 2023-11-15T01:52:15.479 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:15.477+0000 7f7139bb1700 1 -- 172.21.15.137:0/3780301587 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f71340fbf60 con 0x7f71340f6510 2023-11-15T01:52:15.480 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:15.478+0000 7f712bfff700 1 -- 172.21.15.137:0/3780301587 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f711c00f300 con 0x7f71340f6510 2023-11-15T01:52:15.480 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:52:15.480 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:52:15.481 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:15.479+0000 7f7139bb1700 1 -- 172.21.15.137:0/3780301587 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f712002ff60 msgr2=0x7f7120032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:52:15.481 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:15.479+0000 7f7139bb1700 1 --2- 172.21.15.137:0/3780301587 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f712002ff60 0x7f7120032420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f71240096a0 tx=0x7f7124006b40).stop 2023-11-15T01:52:15.482 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:15.479+0000 7f7139bb1700 1 -- 172.21.15.137:0/3780301587 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f71340f6510 msgr2=0x7f7134106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:52:15.482 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:15.479+0000 7f7139bb1700 1 --2- 172.21.15.137:0/3780301587 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f71340f6510 0x7f7134106730 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f711c014000 tx=0x7f711c00b040).stop 2023-11-15T01:52:15.482 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:15.479+0000 7f7139bb1700 1 -- 172.21.15.137:0/3780301587 shutdown_connections 2023-11-15T01:52:15.482 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:15.479+0000 7f7139bb1700 1 --2- 172.21.15.137:0/3780301587 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f712002ff60 0x7f7120032420 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:52:15.482 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:15.479+0000 7f7139bb1700 1 --2- 172.21.15.137:0/3780301587 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f71340f6510 0x7f7134106730 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:52:15.483 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:15.479+0000 7f7139bb1700 1 -- 172.21.15.137:0/3780301587 >> 172.21.15.137:0/3780301587 conn(0x7f71340f1c70 msgr2=0x7f71340f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:52:15.483 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:15.480+0000 7f7139bb1700 1 -- 172.21.15.137:0/3780301587 shutdown_connections 2023-11-15T01:52:15.483 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:15.480+0000 7f7139bb1700 1 -- 172.21.15.137:0/3780301587 wait complete. 2023-11-15T01:52:15.484 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:52:16.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:52:15 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:52:14.714493+0000 mgr.smithi055.szwngw (mgr.14164) 77 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:52:16.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:52:15 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:52:15.480039+0000 mon.smithi055 (mon.0) 164 : audit [DBG] from='client.? 172.21.15.137:0/3780301587' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:52:16.878 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:52:16.878 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:52:18.033 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:18.030+0000 7fb93c9aa700 1 -- 172.21.15.137:0/1708536798 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb9380f51a0 msgr2=0x7fb9380f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:52:18.033 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:18.030+0000 7fb93c9aa700 1 --2- 172.21.15.137:0/1708536798 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb9380f51a0 0x7fb9380f55c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7fb928009fb0 tx=0x7fb928009b30).stop 2023-11-15T01:52:18.034 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:18.031+0000 7fb93c9aa700 1 -- 172.21.15.137:0/1708536798 shutdown_connections 2023-11-15T01:52:18.034 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:18.031+0000 7fb93c9aa700 1 --2- 172.21.15.137:0/1708536798 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb9380f51a0 0x7fb9380f55c0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:52:18.034 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:18.031+0000 7fb93c9aa700 1 -- 172.21.15.137:0/1708536798 >> 172.21.15.137:0/1708536798 conn(0x7fb9380f09a0 msgr2=0x7fb9380f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:52:18.034 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:18.031+0000 7fb93c9aa700 1 -- 172.21.15.137:0/1708536798 shutdown_connections 2023-11-15T01:52:18.034 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:18.031+0000 7fb93c9aa700 1 -- 172.21.15.137:0/1708536798 wait complete. 2023-11-15T01:52:18.035 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:18.031+0000 7fb93c9aa700 1 Processor -- start 2023-11-15T01:52:18.035 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:18.031+0000 7fb93c9aa700 1 -- start start 2023-11-15T01:52:18.035 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:18.031+0000 7fb93c9aa700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb9380f51a0 0x7fb938109200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:52:18.035 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:18.031+0000 7fb93c9aa700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb938060220 con 0x7fb9380f51a0 2023-11-15T01:52:18.035 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:18.032+0000 7fb9377fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb9380f51a0 0x7fb938109200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:52:18.036 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:18.032+0000 7fb9377fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb9380f51a0 0x7fb938109200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:41158/0 (socket says 172.21.15.137:41158) 2023-11-15T01:52:18.036 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:18.032+0000 7fb9377fe700 1 -- 172.21.15.137:0/4105969870 learned_addr learned my addr 172.21.15.137:0/4105969870 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:52:18.037 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:18.032+0000 7fb9377fe700 1 -- 172.21.15.137:0/4105969870 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb928005040 con 0x7fb9380f51a0 2023-11-15T01:52:18.038 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:18.032+0000 7fb9377fe700 1 --2- 172.21.15.137:0/4105969870 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb9380f51a0 0x7fb938109200 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7fb928009830 tx=0x7fb92800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:52:18.038 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:18.032+0000 7fb91ffff700 1 -- 172.21.15.137:0/4105969870 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb928009320 con 0x7fb9380f51a0 2023-11-15T01:52:18.038 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:18.032+0000 7fb93c9aa700 1 -- 172.21.15.137:0/4105969870 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb938060420 con 0x7fb9380f51a0 2023-11-15T01:52:18.038 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:18.032+0000 7fb91ffff700 1 -- 172.21.15.137:0/4105969870 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb9280136c0 con 0x7fb9380f51a0 2023-11-15T01:52:18.038 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:18.032+0000 7fb91ffff700 1 -- 172.21.15.137:0/4105969870 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb92801ee90 con 0x7fb9380f51a0 2023-11-15T01:52:18.039 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:18.032+0000 7fb93c9aa700 1 -- 172.21.15.137:0/4105969870 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9381098b0 con 0x7fb9380f51a0 2023-11-15T01:52:18.039 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:18.033+0000 7fb91ffff700 1 -- 172.21.15.137:0/4105969870 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb92801e420 con 0x7fb9380f51a0 2023-11-15T01:52:18.039 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:18.033+0000 7fb93c9aa700 1 -- 172.21.15.137:0/4105969870 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb938059100 con 0x7fb9380f51a0 2023-11-15T01:52:18.039 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:18.033+0000 7fb91ffff700 1 --2- 172.21.15.137:0/4105969870 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fb92002ff10 0x7fb9200323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:52:18.039 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:18.033+0000 7fb91ffff700 1 -- 172.21.15.137:0/4105969870 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb928040fe0 con 0x7fb9380f51a0 2023-11-15T01:52:18.040 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:18.033+0000 7fb936ffd700 1 --2- 172.21.15.137:0/4105969870 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fb92002ff10 0x7fb9200323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:52:18.040 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:18.034+0000 7fb936ffd700 1 --2- 172.21.15.137:0/4105969870 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fb92002ff10 0x7fb9200323d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7fb92c0096a0 tx=0x7fb92c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:52:18.040 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:18.039+0000 7fb91ffff700 1 -- 172.21.15.137:0/4105969870 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb9280095f0 con 0x7fb9380f51a0 2023-11-15T01:52:18.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:52:17 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:52:16.714833+0000 mgr.smithi055.szwngw (mgr.14164) 78 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:52:18.243 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:18.242+0000 7fb93c9aa700 1 -- 172.21.15.137:0/4105969870 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb938045ad0 con 0x7fb9380f51a0 2023-11-15T01:52:18.244 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:18.242+0000 7fb91ffff700 1 -- 172.21.15.137:0/4105969870 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb92802c2d0 con 0x7fb9380f51a0 2023-11-15T01:52:18.244 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:52:18.245 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:52:18.245 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:18.244+0000 7fb93c9aa700 1 -- 172.21.15.137:0/4105969870 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fb92002ff10 msgr2=0x7fb9200323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:52:18.246 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:18.244+0000 7fb93c9aa700 1 --2- 172.21.15.137:0/4105969870 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fb92002ff10 0x7fb9200323d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7fb92c0096a0 tx=0x7fb92c006b40).stop 2023-11-15T01:52:18.246 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:18.244+0000 7fb93c9aa700 1 -- 172.21.15.137:0/4105969870 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb9380f51a0 msgr2=0x7fb938109200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:52:18.246 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:18.244+0000 7fb93c9aa700 1 --2- 172.21.15.137:0/4105969870 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb9380f51a0 0x7fb938109200 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7fb928009830 tx=0x7fb92800b040).stop 2023-11-15T01:52:18.246 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:18.244+0000 7fb93c9aa700 1 -- 172.21.15.137:0/4105969870 shutdown_connections 2023-11-15T01:52:18.247 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:18.244+0000 7fb93c9aa700 1 --2- 172.21.15.137:0/4105969870 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fb92002ff10 0x7fb9200323d0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:52:18.247 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:18.244+0000 7fb93c9aa700 1 --2- 172.21.15.137:0/4105969870 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb9380f51a0 0x7fb938109200 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:52:18.247 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:18.244+0000 7fb93c9aa700 1 -- 172.21.15.137:0/4105969870 >> 172.21.15.137:0/4105969870 conn(0x7fb9380f09a0 msgr2=0x7fb9380f13e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:52:18.247 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:18.244+0000 7fb93c9aa700 1 -- 172.21.15.137:0/4105969870 shutdown_connections 2023-11-15T01:52:18.247 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:18.244+0000 7fb93c9aa700 1 -- 172.21.15.137:0/4105969870 wait complete. 2023-11-15T01:52:18.248 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:52:19.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:52:18 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:52:18.244309+0000 mon.smithi055 (mon.0) 165 : audit [DBG] from='client.? 172.21.15.137:0/4105969870' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:52:19.646 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:52:19.647 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:52:20.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:52:19 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:52:18.715108+0000 mgr.smithi055.szwngw (mgr.14164) 79 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:52:20.866 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:20.864+0000 7fa04f59e700 1 -- 172.21.15.137:0/2340661480 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa0500f51a0 msgr2=0x7fa0500f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:52:20.866 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:20.864+0000 7fa04f59e700 1 --2- 172.21.15.137:0/2340661480 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa0500f51a0 0x7fa0500f55c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7fa038009fb0 tx=0x7fa038009b30).stop 2023-11-15T01:52:20.866 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:20.865+0000 7fa04f59e700 1 -- 172.21.15.137:0/2340661480 shutdown_connections 2023-11-15T01:52:20.867 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:20.865+0000 7fa04f59e700 1 --2- 172.21.15.137:0/2340661480 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa0500f51a0 0x7fa0500f55c0 secure :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0x7fa038009fb0 tx=0x7fa038009b30).stop 2023-11-15T01:52:20.867 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:20.865+0000 7fa04f59e700 1 -- 172.21.15.137:0/2340661480 >> 172.21.15.137:0/2340661480 conn(0x7fa0500f09a0 msgr2=0x7fa0500f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:52:20.867 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:20.865+0000 7fa04f59e700 1 -- 172.21.15.137:0/2340661480 shutdown_connections 2023-11-15T01:52:20.867 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:20.865+0000 7fa04f59e700 1 -- 172.21.15.137:0/2340661480 wait complete. 2023-11-15T01:52:20.867 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:20.865+0000 7fa04f59e700 1 Processor -- start 2023-11-15T01:52:20.868 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:20.865+0000 7fa04f59e700 1 -- start start 2023-11-15T01:52:20.868 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:20.865+0000 7fa04f59e700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa050060200 0x7fa050109340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:52:20.868 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:20.865+0000 7fa04f59e700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa050109830 con 0x7fa050060200 2023-11-15T01:52:20.870 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:20.866+0000 7fa04e59c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa050060200 0x7fa050109340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:52:20.870 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:20.866+0000 7fa04e59c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa050060200 0x7fa050109340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:59450/0 (socket says 172.21.15.137:59450) 2023-11-15T01:52:20.871 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:20.866+0000 7fa04e59c700 1 -- 172.21.15.137:0/405552507 learned_addr learned my addr 172.21.15.137:0/405552507 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:52:20.871 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:20.866+0000 7fa04e59c700 1 -- 172.21.15.137:0/405552507 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa038005040 con 0x7fa050060200 2023-11-15T01:52:20.871 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:20.866+0000 7fa04e59c700 1 --2- 172.21.15.137:0/405552507 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa050060200 0x7fa050109340 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7fa038009f80 tx=0x7fa038009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:52:20.871 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:20.866+0000 7fa046ffd700 1 -- 172.21.15.137:0/405552507 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa0380136c0 con 0x7fa050060200 2023-11-15T01:52:20.871 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:20.866+0000 7fa046ffd700 1 -- 172.21.15.137:0/405552507 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa0380095f0 con 0x7fa050060200 2023-11-15T01:52:20.872 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:20.866+0000 7fa04f59e700 1 -- 172.21.15.137:0/405552507 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa050109970 con 0x7fa050060200 2023-11-15T01:52:20.872 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:20.866+0000 7fa046ffd700 1 -- 172.21.15.137:0/405552507 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa038014420 con 0x7fa050060200 2023-11-15T01:52:20.872 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:20.866+0000 7fa04f59e700 1 -- 172.21.15.137:0/405552507 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa050109da0 con 0x7fa050060200 2023-11-15T01:52:20.872 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:20.867+0000 7fa046ffd700 1 -- 172.21.15.137:0/405552507 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa038025060 con 0x7fa050060200 2023-11-15T01:52:20.873 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:20.867+0000 7fa04f59e700 1 -- 172.21.15.137:0/405552507 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa050045ad0 con 0x7fa050060200 2023-11-15T01:52:20.873 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:20.867+0000 7fa046ffd700 1 --2- 172.21.15.137:0/405552507 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fa03c02ff60 0x7fa03c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:52:20.873 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:20.867+0000 7fa046ffd700 1 -- 172.21.15.137:0/405552507 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa038040d80 con 0x7fa050060200 2023-11-15T01:52:20.873 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:20.867+0000 7fa04dd9b700 1 --2- 172.21.15.137:0/405552507 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fa03c02ff60 0x7fa03c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:52:20.873 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:20.868+0000 7fa04dd9b700 1 --2- 172.21.15.137:0/405552507 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fa03c02ff60 0x7fa03c032420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7fa0400096a0 tx=0x7fa040006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:52:20.874 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:20.872+0000 7fa046ffd700 1 -- 172.21.15.137:0/405552507 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa038004490 con 0x7fa050060200 2023-11-15T01:52:21.076 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:21.076+0000 7fa04f59e700 1 -- 172.21.15.137:0/405552507 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa0500fabe0 con 0x7fa050060200 2023-11-15T01:52:21.077 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:21.076+0000 7fa046ffd700 1 -- 172.21.15.137:0/405552507 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa038042090 con 0x7fa050060200 2023-11-15T01:52:21.078 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:52:21.078 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:52:21.080 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:21.078+0000 7fa04f59e700 1 -- 172.21.15.137:0/405552507 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fa03c02ff60 msgr2=0x7fa03c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:52:21.080 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:21.078+0000 7fa04f59e700 1 --2- 172.21.15.137:0/405552507 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fa03c02ff60 0x7fa03c032420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7fa0400096a0 tx=0x7fa040006b40).stop 2023-11-15T01:52:21.081 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:21.078+0000 7fa04f59e700 1 -- 172.21.15.137:0/405552507 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa050060200 msgr2=0x7fa050109340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:52:21.081 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:21.078+0000 7fa04f59e700 1 --2- 172.21.15.137:0/405552507 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa050060200 0x7fa050109340 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7fa038009f80 tx=0x7fa038009b30).stop 2023-11-15T01:52:21.081 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:21.078+0000 7fa04f59e700 1 -- 172.21.15.137:0/405552507 shutdown_connections 2023-11-15T01:52:21.082 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:21.078+0000 7fa04f59e700 1 --2- 172.21.15.137:0/405552507 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fa03c02ff60 0x7fa03c032420 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:52:21.082 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:21.078+0000 7fa04f59e700 1 --2- 172.21.15.137:0/405552507 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa050060200 0x7fa050109340 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:52:21.082 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:21.078+0000 7fa04f59e700 1 -- 172.21.15.137:0/405552507 >> 172.21.15.137:0/405552507 conn(0x7fa0500f09a0 msgr2=0x7fa0500f1500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:52:21.083 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:21.078+0000 7fa04f59e700 1 -- 172.21.15.137:0/405552507 shutdown_connections 2023-11-15T01:52:21.083 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:21.078+0000 7fa04f59e700 1 -- 172.21.15.137:0/405552507 wait complete. 2023-11-15T01:52:21.083 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:52:21.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:52:21 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:52:20.715350+0000 mgr.smithi055.szwngw (mgr.14164) 80 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:52:21.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:52:21 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:52:21.077499+0000 mon.smithi055 (mon.0) 166 : audit [DBG] from='client.? 172.21.15.137:0/405552507' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:52:22.465 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:52:22.466 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:52:23.642 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:23.640+0000 7f2522bdc700 1 -- 172.21.15.137:0/4138057333 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f251c060280 msgr2=0x7f251c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:52:23.642 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:23.640+0000 7f2522bdc700 1 --2- 172.21.15.137:0/4138057333 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f251c060280 0x7f251c0f90f0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f2510009fb0 tx=0x7f2510009b30).stop 2023-11-15T01:52:23.642 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:23.640+0000 7f2522bdc700 1 -- 172.21.15.137:0/4138057333 shutdown_connections 2023-11-15T01:52:23.643 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:23.640+0000 7f2522bdc700 1 --2- 172.21.15.137:0/4138057333 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f251c060280 0x7f251c0f90f0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:52:23.643 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:23.640+0000 7f2522bdc700 1 -- 172.21.15.137:0/4138057333 >> 172.21.15.137:0/4138057333 conn(0x7f251c0f0b70 msgr2=0x7f251c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:52:23.643 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:23.640+0000 7f2522bdc700 1 -- 172.21.15.137:0/4138057333 shutdown_connections 2023-11-15T01:52:23.643 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:23.640+0000 7f2522bdc700 1 -- 172.21.15.137:0/4138057333 wait complete. 2023-11-15T01:52:23.643 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:23.641+0000 7f2522bdc700 1 Processor -- start 2023-11-15T01:52:23.643 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:23.641+0000 7f2522bdc700 1 -- start start 2023-11-15T01:52:23.644 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:23.641+0000 7f2522bdc700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f251c060280 0x7f251c1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:52:23.644 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:23.641+0000 7f2522bdc700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f251c106bd0 con 0x7f251c060280 2023-11-15T01:52:23.644 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:23.641+0000 7f2520978700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f251c060280 0x7f251c1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:52:23.644 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:23.641+0000 7f2520978700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f251c060280 0x7f251c1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:59482/0 (socket says 172.21.15.137:59482) 2023-11-15T01:52:23.644 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:23.641+0000 7f2520978700 1 -- 172.21.15.137:0/2878838440 learned_addr learned my addr 172.21.15.137:0/2878838440 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:52:23.646 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:23.642+0000 7f2520978700 1 -- 172.21.15.137:0/2878838440 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2510005040 con 0x7f251c060280 2023-11-15T01:52:23.646 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:23.642+0000 7f2520978700 1 --2- 172.21.15.137:0/2878838440 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f251c060280 0x7f251c1066e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f2510013510 tx=0x7f251000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:52:23.646 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:23.642+0000 7f25197fa700 1 -- 172.21.15.137:0/2878838440 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2510009320 con 0x7f251c060280 2023-11-15T01:52:23.646 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:23.642+0000 7f2522bdc700 1 -- 172.21.15.137:0/2878838440 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f251c106dd0 con 0x7f251c060280 2023-11-15T01:52:23.647 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:23.642+0000 7f25197fa700 1 -- 172.21.15.137:0/2878838440 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2510013f00 con 0x7f251c060280 2023-11-15T01:52:23.647 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:23.642+0000 7f25197fa700 1 -- 172.21.15.137:0/2878838440 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2510014dc0 con 0x7f251c060280 2023-11-15T01:52:23.647 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:23.642+0000 7f2522bdc700 1 -- 172.21.15.137:0/2878838440 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f251c107260 con 0x7f251c060280 2023-11-15T01:52:23.647 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:23.643+0000 7f25197fa700 1 -- 172.21.15.137:0/2878838440 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f25100095f0 con 0x7f251c060280 2023-11-15T01:52:23.647 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:23.643+0000 7f2522bdc700 1 -- 172.21.15.137:0/2878838440 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f251c045ad0 con 0x7f251c060280 2023-11-15T01:52:23.648 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:23.643+0000 7f25197fa700 1 --2- 172.21.15.137:0/2878838440 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f250c02ff60 0x7f250c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:52:23.648 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:23.643+0000 7f25197fa700 1 -- 172.21.15.137:0/2878838440 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2510041100 con 0x7f251c060280 2023-11-15T01:52:23.648 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:23.643+0000 7f251bfff700 1 --2- 172.21.15.137:0/2878838440 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f250c02ff60 0x7f250c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:52:23.648 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:23.644+0000 7f251bfff700 1 --2- 172.21.15.137:0/2878838440 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f250c02ff60 0x7f250c032420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f25080096a0 tx=0x7f2508006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:52:23.649 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:23.648+0000 7f25197fa700 1 -- 172.21.15.137:0/2878838440 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2510014420 con 0x7f251c060280 2023-11-15T01:52:23.854 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:23.853+0000 7f2522bdc700 1 -- 172.21.15.137:0/2878838440 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f251c108110 con 0x7f251c060280 2023-11-15T01:52:23.854 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:23.854+0000 7f25197fa700 1 -- 172.21.15.137:0/2878838440 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2510010230 con 0x7f251c060280 2023-11-15T01:52:23.855 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:52:23.855 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:52:23.856 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:23.855+0000 7f2522bdc700 1 -- 172.21.15.137:0/2878838440 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f250c02ff60 msgr2=0x7f250c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:52:23.856 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:23.855+0000 7f2522bdc700 1 --2- 172.21.15.137:0/2878838440 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f250c02ff60 0x7f250c032420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f25080096a0 tx=0x7f2508006b40).stop 2023-11-15T01:52:23.856 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:23.855+0000 7f2522bdc700 1 -- 172.21.15.137:0/2878838440 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f251c060280 msgr2=0x7f251c1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:52:23.856 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:23.855+0000 7f2522bdc700 1 --2- 172.21.15.137:0/2878838440 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f251c060280 0x7f251c1066e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f2510013510 tx=0x7f251000b040).stop 2023-11-15T01:52:23.857 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:23.855+0000 7f2522bdc700 1 -- 172.21.15.137:0/2878838440 shutdown_connections 2023-11-15T01:52:23.857 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:23.855+0000 7f2522bdc700 1 --2- 172.21.15.137:0/2878838440 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f250c02ff60 0x7f250c032420 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:52:23.857 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:23.855+0000 7f2522bdc700 1 --2- 172.21.15.137:0/2878838440 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f251c060280 0x7f251c1066e0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:52:23.857 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:23.855+0000 7f2522bdc700 1 -- 172.21.15.137:0/2878838440 >> 172.21.15.137:0/2878838440 conn(0x7f251c0f0b70 msgr2=0x7f251c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:52:23.857 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:23.855+0000 7f2522bdc700 1 -- 172.21.15.137:0/2878838440 shutdown_connections 2023-11-15T01:52:23.858 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:23.855+0000 7f2522bdc700 1 -- 172.21.15.137:0/2878838440 wait complete. 2023-11-15T01:52:23.858 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:52:24.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:52:23 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:52:22.715667+0000 mgr.smithi055.szwngw (mgr.14164) 81 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:52:25.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:52:24 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:52:23.854664+0000 mon.smithi055 (mon.0) 167 : audit [DBG] from='client.? 172.21.15.137:0/2878838440' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:52:25.254 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:52:25.255 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:52:26.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:52:25 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:52:24.716042+0000 mgr.smithi055.szwngw (mgr.14164) 82 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:52:26.427 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:26.426+0000 7f49214b0700 1 -- 172.21.15.137:0/1747086343 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f491c0f5410 msgr2=0x7f491c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:52:26.428 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:26.426+0000 7f49214b0700 1 --2- 172.21.15.137:0/1747086343 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f491c0f5410 0x7f491c0f5830 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f4910009fb0 tx=0x7f4910009b30).stop 2023-11-15T01:52:26.428 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:26.426+0000 7f49214b0700 1 -- 172.21.15.137:0/1747086343 shutdown_connections 2023-11-15T01:52:26.428 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:26.426+0000 7f49214b0700 1 --2- 172.21.15.137:0/1747086343 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f491c0f5410 0x7f491c0f5830 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:52:26.428 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:26.426+0000 7f49214b0700 1 -- 172.21.15.137:0/1747086343 >> 172.21.15.137:0/1747086343 conn(0x7f491c0f0b70 msgr2=0x7f491c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:52:26.429 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:26.426+0000 7f49214b0700 1 -- 172.21.15.137:0/1747086343 shutdown_connections 2023-11-15T01:52:26.429 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:26.426+0000 7f49214b0700 1 -- 172.21.15.137:0/1747086343 wait complete. 2023-11-15T01:52:26.431 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:26.427+0000 7f49214b0700 1 Processor -- start 2023-11-15T01:52:26.431 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:26.427+0000 7f49214b0700 1 -- start start 2023-11-15T01:52:26.431 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:26.427+0000 7f49214b0700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f491c0f5410 0x7f491c106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:52:26.431 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:26.427+0000 7f49214b0700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f491c106c10 con 0x7f491c0f5410 2023-11-15T01:52:26.431 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:26.428+0000 7f491affd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f491c0f5410 0x7f491c106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:52:26.432 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:26.428+0000 7f491affd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f491c0f5410 0x7f491c106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:59502/0 (socket says 172.21.15.137:59502) 2023-11-15T01:52:26.432 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:26.428+0000 7f491affd700 1 -- 172.21.15.137:0/3422759713 learned_addr learned my addr 172.21.15.137:0/3422759713 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:52:26.432 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:26.428+0000 7f491affd700 1 -- 172.21.15.137:0/3422759713 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4910005040 con 0x7f491c0f5410 2023-11-15T01:52:26.432 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:26.428+0000 7f491affd700 1 --2- 172.21.15.137:0/3422759713 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f491c0f5410 0x7f491c106720 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f4910013510 tx=0x7f491000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:52:26.432 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:26.428+0000 7f49037fe700 1 -- 172.21.15.137:0/3422759713 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4910009320 con 0x7f491c0f5410 2023-11-15T01:52:26.432 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:26.428+0000 7f49214b0700 1 -- 172.21.15.137:0/3422759713 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f491c106e10 con 0x7f491c0f5410 2023-11-15T01:52:26.433 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:26.428+0000 7f49037fe700 1 -- 172.21.15.137:0/3422759713 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4910013f00 con 0x7f491c0f5410 2023-11-15T01:52:26.433 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:26.428+0000 7f49037fe700 1 -- 172.21.15.137:0/3422759713 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4910014d90 con 0x7f491c0f5410 2023-11-15T01:52:26.433 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:26.428+0000 7f49214b0700 1 -- 172.21.15.137:0/3422759713 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f491c1072a0 con 0x7f491c0f5410 2023-11-15T01:52:26.433 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:26.429+0000 7f49037fe700 1 -- 172.21.15.137:0/3422759713 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f49100263a0 con 0x7f491c0f5410 2023-11-15T01:52:26.433 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:26.429+0000 7f49214b0700 1 -- 172.21.15.137:0/3422759713 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f491c045ad0 con 0x7f491c0f5410 2023-11-15T01:52:26.434 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:26.429+0000 7f49037fe700 1 --2- 172.21.15.137:0/3422759713 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f490402ff60 0x7f4904032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:52:26.434 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:26.429+0000 7f49037fe700 1 -- 172.21.15.137:0/3422759713 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4910041130 con 0x7f491c0f5410 2023-11-15T01:52:26.434 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:26.429+0000 7f491a7fc700 1 --2- 172.21.15.137:0/3422759713 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f490402ff60 0x7f4904032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:52:26.434 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:26.430+0000 7f491a7fc700 1 --2- 172.21.15.137:0/3422759713 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f490402ff60 0x7f4904032420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f490c0096a0 tx=0x7f490c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:52:26.435 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:26.435+0000 7f49037fe700 1 -- 172.21.15.137:0/3422759713 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4910026c30 con 0x7f491c0f5410 2023-11-15T01:52:26.644 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:26.642+0000 7f49214b0700 1 -- 172.21.15.137:0/3422759713 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f491c108150 con 0x7f491c0f5410 2023-11-15T01:52:26.646 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:26.643+0000 7f49037fe700 1 -- 172.21.15.137:0/3422759713 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f491000f340 con 0x7f491c0f5410 2023-11-15T01:52:26.646 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:52:26.646 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:52:26.647 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:26.645+0000 7f49214b0700 1 -- 172.21.15.137:0/3422759713 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f490402ff60 msgr2=0x7f4904032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:52:26.648 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:26.645+0000 7f49214b0700 1 --2- 172.21.15.137:0/3422759713 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f490402ff60 0x7f4904032420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f490c0096a0 tx=0x7f490c006b40).stop 2023-11-15T01:52:26.648 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:26.645+0000 7f49214b0700 1 -- 172.21.15.137:0/3422759713 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f491c0f5410 msgr2=0x7f491c106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:52:26.648 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:26.645+0000 7f49214b0700 1 --2- 172.21.15.137:0/3422759713 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f491c0f5410 0x7f491c106720 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f4910013510 tx=0x7f491000b040).stop 2023-11-15T01:52:26.648 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:26.645+0000 7f49214b0700 1 -- 172.21.15.137:0/3422759713 shutdown_connections 2023-11-15T01:52:26.648 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:26.645+0000 7f49214b0700 1 --2- 172.21.15.137:0/3422759713 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f490402ff60 0x7f4904032420 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:52:26.649 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:26.645+0000 7f49214b0700 1 --2- 172.21.15.137:0/3422759713 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f491c0f5410 0x7f491c106720 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:52:26.649 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:26.645+0000 7f49214b0700 1 -- 172.21.15.137:0/3422759713 >> 172.21.15.137:0/3422759713 conn(0x7f491c0f0b70 msgr2=0x7f491c05fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:52:26.649 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:26.645+0000 7f49214b0700 1 -- 172.21.15.137:0/3422759713 shutdown_connections 2023-11-15T01:52:26.649 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:26.645+0000 7f49214b0700 1 -- 172.21.15.137:0/3422759713 wait complete. 2023-11-15T01:52:26.649 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:52:27.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:52:26 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:52:26.644105+0000 mon.smithi055 (mon.0) 168 : audit [DBG] from='client.? 172.21.15.137:0/3422759713' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:52:28.037 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:52:28.038 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:52:28.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:52:27 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:52:26.716302+0000 mgr.smithi055.szwngw (mgr.14164) 83 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:52:29.164 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:29.160+0000 7fda8cfe4700 1 -- 172.21.15.137:0/34480773 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fda88060280 msgr2=0x7fda880f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:52:29.165 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:29.160+0000 7fda8cfe4700 1 --2- 172.21.15.137:0/34480773 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fda88060280 0x7fda880f90f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7fda70009fb0 tx=0x7fda70009b30).stop 2023-11-15T01:52:29.165 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:29.160+0000 7fda8cfe4700 1 -- 172.21.15.137:0/34480773 shutdown_connections 2023-11-15T01:52:29.166 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:29.160+0000 7fda8cfe4700 1 --2- 172.21.15.137:0/34480773 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fda88060280 0x7fda880f90f0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:52:29.168 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:29.160+0000 7fda8cfe4700 1 -- 172.21.15.137:0/34480773 >> 172.21.15.137:0/34480773 conn(0x7fda880f0b70 msgr2=0x7fda880f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:52:29.168 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:29.160+0000 7fda8cfe4700 1 -- 172.21.15.137:0/34480773 shutdown_connections 2023-11-15T01:52:29.168 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:29.160+0000 7fda8cfe4700 1 -- 172.21.15.137:0/34480773 wait complete. 2023-11-15T01:52:29.168 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:29.161+0000 7fda8cfe4700 1 Processor -- start 2023-11-15T01:52:29.169 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:29.161+0000 7fda8cfe4700 1 -- start start 2023-11-15T01:52:29.169 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:29.161+0000 7fda8cfe4700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fda88060280 0x7fda881066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:52:29.169 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:29.161+0000 7fda8cfe4700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda88106bd0 con 0x7fda88060280 2023-11-15T01:52:29.169 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:29.161+0000 7fda8659c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fda88060280 0x7fda881066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:52:29.170 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:29.161+0000 7fda8659c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fda88060280 0x7fda881066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:37026/0 (socket says 172.21.15.137:37026) 2023-11-15T01:52:29.170 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:29.161+0000 7fda8659c700 1 -- 172.21.15.137:0/2682361761 learned_addr learned my addr 172.21.15.137:0/2682361761 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:52:29.170 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:29.161+0000 7fda8659c700 1 -- 172.21.15.137:0/2682361761 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda70005040 con 0x7fda88060280 2023-11-15T01:52:29.171 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:29.161+0000 7fda8659c700 1 --2- 172.21.15.137:0/2682361761 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fda88060280 0x7fda881066e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7fda70009830 tx=0x7fda7000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:52:29.171 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:29.162+0000 7fda7effd700 1 -- 172.21.15.137:0/2682361761 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda70009320 con 0x7fda88060280 2023-11-15T01:52:29.171 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:29.162+0000 7fda8cfe4700 1 -- 172.21.15.137:0/2682361761 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda88106dd0 con 0x7fda88060280 2023-11-15T01:52:29.172 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:29.162+0000 7fda7effd700 1 -- 172.21.15.137:0/2682361761 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fda700136c0 con 0x7fda88060280 2023-11-15T01:52:29.172 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:29.162+0000 7fda7effd700 1 -- 172.21.15.137:0/2682361761 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda7001ee90 con 0x7fda88060280 2023-11-15T01:52:29.172 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:29.162+0000 7fda8cfe4700 1 -- 172.21.15.137:0/2682361761 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda88107260 con 0x7fda88060280 2023-11-15T01:52:29.172 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:29.163+0000 7fda7effd700 1 -- 172.21.15.137:0/2682361761 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fda7001e420 con 0x7fda88060280 2023-11-15T01:52:29.172 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:29.163+0000 7fda8cfe4700 1 -- 172.21.15.137:0/2682361761 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda88045ad0 con 0x7fda88060280 2023-11-15T01:52:29.172 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:29.163+0000 7fda7effd700 1 --2- 172.21.15.137:0/2682361761 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fda7402ff10 0x7fda740323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:52:29.173 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:29.163+0000 7fda7effd700 1 -- 172.21.15.137:0/2682361761 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fda70040f40 con 0x7fda88060280 2023-11-15T01:52:29.173 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:29.163+0000 7fda85d9b700 1 --2- 172.21.15.137:0/2682361761 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fda7402ff10 0x7fda740323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:52:29.173 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:29.163+0000 7fda85d9b700 1 --2- 172.21.15.137:0/2682361761 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fda7402ff10 0x7fda740323d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7fda780096a0 tx=0x7fda78006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:52:29.174 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:29.168+0000 7fda7effd700 1 -- 172.21.15.137:0/2682361761 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fda7006d2b0 con 0x7fda88060280 2023-11-15T01:52:29.375 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:29.374+0000 7fda8cfe4700 1 -- 172.21.15.137:0/2682361761 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fda88108110 con 0x7fda88060280 2023-11-15T01:52:29.376 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:29.375+0000 7fda7effd700 1 -- 172.21.15.137:0/2682361761 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fda700095f0 con 0x7fda88060280 2023-11-15T01:52:29.377 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:52:29.378 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:52:29.381 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:29.376+0000 7fda8cfe4700 1 -- 172.21.15.137:0/2682361761 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fda7402ff10 msgr2=0x7fda740323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:52:29.381 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:29.376+0000 7fda8cfe4700 1 --2- 172.21.15.137:0/2682361761 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fda7402ff10 0x7fda740323d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7fda780096a0 tx=0x7fda78006b40).stop 2023-11-15T01:52:29.382 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:29.376+0000 7fda8cfe4700 1 -- 172.21.15.137:0/2682361761 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fda88060280 msgr2=0x7fda881066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:52:29.382 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:29.376+0000 7fda8cfe4700 1 --2- 172.21.15.137:0/2682361761 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fda88060280 0x7fda881066e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7fda70009830 tx=0x7fda7000b040).stop 2023-11-15T01:52:29.383 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:29.376+0000 7fda8cfe4700 1 -- 172.21.15.137:0/2682361761 shutdown_connections 2023-11-15T01:52:29.383 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:29.376+0000 7fda8cfe4700 1 --2- 172.21.15.137:0/2682361761 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fda7402ff10 0x7fda740323d0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:52:29.384 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:29.376+0000 7fda8cfe4700 1 --2- 172.21.15.137:0/2682361761 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fda88060280 0x7fda881066e0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:52:29.384 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:29.376+0000 7fda8cfe4700 1 -- 172.21.15.137:0/2682361761 >> 172.21.15.137:0/2682361761 conn(0x7fda880f0b70 msgr2=0x7fda880f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:52:29.385 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:29.376+0000 7fda8cfe4700 1 -- 172.21.15.137:0/2682361761 shutdown_connections 2023-11-15T01:52:29.385 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:29.376+0000 7fda8cfe4700 1 -- 172.21.15.137:0/2682361761 wait complete. 2023-11-15T01:52:29.386 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:52:30.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:52:29 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:52:28.716547+0000 mgr.smithi055.szwngw (mgr.14164) 84 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:52:30.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:52:29 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:52:29.375788+0000 mon.smithi055 (mon.0) 169 : audit [DBG] from='client.? 172.21.15.137:0/2682361761' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:52:30.762 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:52:30.763 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:52:31.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:52:31 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:52:30.716851+0000 mgr.smithi055.szwngw (mgr.14164) 85 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:52:31.905 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:31.903+0000 7f8448cbb700 1 -- 172.21.15.137:0/1590730004 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f84440f5330 msgr2=0x7f84440f5750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:52:31.905 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:31.903+0000 7f8448cbb700 1 --2- 172.21.15.137:0/1590730004 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f84440f5330 0x7f84440f5750 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f8438009fb0 tx=0x7f8438009b30).stop 2023-11-15T01:52:31.905 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:31.903+0000 7f8448cbb700 1 -- 172.21.15.137:0/1590730004 shutdown_connections 2023-11-15T01:52:31.905 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:31.903+0000 7f8448cbb700 1 --2- 172.21.15.137:0/1590730004 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f84440f5330 0x7f84440f5750 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:52:31.906 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:31.903+0000 7f8448cbb700 1 -- 172.21.15.137:0/1590730004 >> 172.21.15.137:0/1590730004 conn(0x7f84440f0a90 msgr2=0x7f84440f2eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:52:31.906 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:31.903+0000 7f8448cbb700 1 -- 172.21.15.137:0/1590730004 shutdown_connections 2023-11-15T01:52:31.906 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:31.903+0000 7f8448cbb700 1 -- 172.21.15.137:0/1590730004 wait complete. 2023-11-15T01:52:31.906 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:31.904+0000 7f8448cbb700 1 Processor -- start 2023-11-15T01:52:31.906 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:31.904+0000 7f8448cbb700 1 -- start start 2023-11-15T01:52:31.907 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:31.904+0000 7f8448cbb700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f84440f5330 0x7f84440fd330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:52:31.907 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:31.904+0000 7f8448cbb700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84440fb7b0 con 0x7f84440f5330 2023-11-15T01:52:31.907 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:31.904+0000 7f844259c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f84440f5330 0x7f84440fd330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:52:31.907 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:31.904+0000 7f844259c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f84440f5330 0x7f84440fd330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:37044/0 (socket says 172.21.15.137:37044) 2023-11-15T01:52:31.907 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:31.904+0000 7f844259c700 1 -- 172.21.15.137:0/517516876 learned_addr learned my addr 172.21.15.137:0/517516876 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:52:31.909 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:31.904+0000 7f844259c700 1 -- 172.21.15.137:0/517516876 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8438005040 con 0x7f84440f5330 2023-11-15T01:52:31.909 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:31.905+0000 7f844259c700 1 --2- 172.21.15.137:0/517516876 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f84440f5330 0x7f84440fd330 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f8438013510 tx=0x7f843800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:52:31.909 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:31.905+0000 7f8432ffd700 1 -- 172.21.15.137:0/517516876 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8438009320 con 0x7f84440f5330 2023-11-15T01:52:31.909 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:31.905+0000 7f8448cbb700 1 -- 172.21.15.137:0/517516876 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84440fb950 con 0x7f84440f5330 2023-11-15T01:52:31.910 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:31.905+0000 7f8432ffd700 1 -- 172.21.15.137:0/517516876 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8438013f00 con 0x7f84440f5330 2023-11-15T01:52:31.910 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:31.905+0000 7f8432ffd700 1 -- 172.21.15.137:0/517516876 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8438014d90 con 0x7f84440f5330 2023-11-15T01:52:31.910 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:31.905+0000 7f8448cbb700 1 -- 172.21.15.137:0/517516876 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84440fbde0 con 0x7f84440f5330 2023-11-15T01:52:31.910 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:31.906+0000 7f8448cbb700 1 -- 172.21.15.137:0/517516876 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8444045ad0 con 0x7f84440f5330 2023-11-15T01:52:31.910 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:31.906+0000 7f8432ffd700 1 -- 172.21.15.137:0/517516876 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f84380095f0 con 0x7f84440f5330 2023-11-15T01:52:31.911 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:31.906+0000 7f8432ffd700 1 --2- 172.21.15.137:0/517516876 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f842c02ff60 0x7f842c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:52:31.911 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:31.906+0000 7f8432ffd700 1 -- 172.21.15.137:0/517516876 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8438041080 con 0x7f84440f5330 2023-11-15T01:52:31.911 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:31.906+0000 7f8441d9b700 1 --2- 172.21.15.137:0/517516876 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f842c02ff60 0x7f842c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:52:31.911 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:31.906+0000 7f8441d9b700 1 --2- 172.21.15.137:0/517516876 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f842c02ff60 0x7f842c032420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f84340096a0 tx=0x7f8434006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:52:31.912 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:31.911+0000 7f8432ffd700 1 -- 172.21.15.137:0/517516876 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8438014420 con 0x7f84440f5330 2023-11-15T01:52:32.116 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:32.115+0000 7f8448cbb700 1 -- 172.21.15.137:0/517516876 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8444027a00 con 0x7f84440f5330 2023-11-15T01:52:32.116 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:32.116+0000 7f8432ffd700 1 -- 172.21.15.137:0/517516876 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8438010230 con 0x7f84440f5330 2023-11-15T01:52:32.117 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:52:32.117 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:52:32.118 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:32.117+0000 7f8448cbb700 1 -- 172.21.15.137:0/517516876 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f842c02ff60 msgr2=0x7f842c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:52:32.118 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:32.117+0000 7f8448cbb700 1 --2- 172.21.15.137:0/517516876 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f842c02ff60 0x7f842c032420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f84340096a0 tx=0x7f8434006b40).stop 2023-11-15T01:52:32.118 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:32.117+0000 7f8448cbb700 1 -- 172.21.15.137:0/517516876 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f84440f5330 msgr2=0x7f84440fd330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:52:32.119 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:32.117+0000 7f8448cbb700 1 --2- 172.21.15.137:0/517516876 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f84440f5330 0x7f84440fd330 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f8438013510 tx=0x7f843800b040).stop 2023-11-15T01:52:32.119 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:32.117+0000 7f8448cbb700 1 -- 172.21.15.137:0/517516876 shutdown_connections 2023-11-15T01:52:32.119 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:32.117+0000 7f8448cbb700 1 --2- 172.21.15.137:0/517516876 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f842c02ff60 0x7f842c032420 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:52:32.119 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:32.117+0000 7f8448cbb700 1 --2- 172.21.15.137:0/517516876 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f84440f5330 0x7f84440fd330 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:52:32.119 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:32.117+0000 7f8448cbb700 1 -- 172.21.15.137:0/517516876 >> 172.21.15.137:0/517516876 conn(0x7f84440f0a90 msgr2=0x7f84440f18d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:52:32.120 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:32.117+0000 7f8448cbb700 1 -- 172.21.15.137:0/517516876 shutdown_connections 2023-11-15T01:52:32.120 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:32.117+0000 7f8448cbb700 1 -- 172.21.15.137:0/517516876 wait complete. 2023-11-15T01:52:32.120 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:52:32.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:52:32 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:52:32.116832+0000 mon.smithi055 (mon.0) 170 : audit [DBG] from='client.? 172.21.15.137:0/517516876' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:52:33.501 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:52:33.502 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:52:33.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:52:33 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:52:32.717125+0000 mgr.smithi055.szwngw (mgr.14164) 86 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:52:34.672 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:34.670+0000 7f510ef38700 1 -- 172.21.15.137:0/2781309697 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f51080f47c0 msgr2=0x7f51080f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:52:34.673 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:34.670+0000 7f510ef38700 1 --2- 172.21.15.137:0/2781309697 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f51080f47c0 0x7f51080f4be0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f50fc009fb0 tx=0x7f50fc009b30).stop 2023-11-15T01:52:34.673 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:34.670+0000 7f510ef38700 1 -- 172.21.15.137:0/2781309697 shutdown_connections 2023-11-15T01:52:34.673 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:34.670+0000 7f510ef38700 1 --2- 172.21.15.137:0/2781309697 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f51080f47c0 0x7f51080f4be0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:52:34.673 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:34.670+0000 7f510ef38700 1 -- 172.21.15.137:0/2781309697 >> 172.21.15.137:0/2781309697 conn(0x7f51080f02e0 msgr2=0x7f51080f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:52:34.673 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:34.670+0000 7f510ef38700 1 -- 172.21.15.137:0/2781309697 shutdown_connections 2023-11-15T01:52:34.674 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:34.670+0000 7f510ef38700 1 -- 172.21.15.137:0/2781309697 wait complete. 2023-11-15T01:52:34.674 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:34.671+0000 7f510ef38700 1 Processor -- start 2023-11-15T01:52:34.674 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:34.671+0000 7f510ef38700 1 -- start start 2023-11-15T01:52:34.674 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:34.671+0000 7f510ef38700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f51080f47c0 0x7f51080f7370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:52:34.674 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:34.671+0000 7f510ef38700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51080f57f0 con 0x7f51080f47c0 2023-11-15T01:52:34.675 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:34.671+0000 7f510ccd4700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f51080f47c0 0x7f51080f7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:52:34.675 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:34.671+0000 7f510ccd4700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f51080f47c0 0x7f51080f7370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:37058/0 (socket says 172.21.15.137:37058) 2023-11-15T01:52:34.675 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:34.671+0000 7f510ccd4700 1 -- 172.21.15.137:0/991630685 learned_addr learned my addr 172.21.15.137:0/991630685 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:52:34.675 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:34.672+0000 7f510ccd4700 1 -- 172.21.15.137:0/991630685 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50fc005040 con 0x7f51080f47c0 2023-11-15T01:52:34.675 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:34.672+0000 7f510ccd4700 1 --2- 172.21.15.137:0/991630685 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f51080f47c0 0x7f51080f7370 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f50fc013510 tx=0x7f50fc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:52:34.675 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:34.672+0000 7f51057fa700 1 -- 172.21.15.137:0/991630685 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f50fc009320 con 0x7f51080f47c0 2023-11-15T01:52:34.676 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:34.672+0000 7f510ef38700 1 -- 172.21.15.137:0/991630685 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51080f5990 con 0x7f51080f47c0 2023-11-15T01:52:34.677 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:34.672+0000 7f51057fa700 1 -- 172.21.15.137:0/991630685 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f50fc013f00 con 0x7f51080f47c0 2023-11-15T01:52:34.677 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:34.672+0000 7f51057fa700 1 -- 172.21.15.137:0/991630685 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f50fc014d90 con 0x7f51080f47c0 2023-11-15T01:52:34.677 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:34.672+0000 7f510ef38700 1 -- 172.21.15.137:0/991630685 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51080f5e20 con 0x7f51080f47c0 2023-11-15T01:52:34.677 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:34.673+0000 7f510ef38700 1 -- 172.21.15.137:0/991630685 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5108059100 con 0x7f51080f47c0 2023-11-15T01:52:34.677 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:34.673+0000 7f51057fa700 1 -- 172.21.15.137:0/991630685 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f50fc0095f0 con 0x7f51080f47c0 2023-11-15T01:52:34.678 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:34.673+0000 7f51057fa700 1 --2- 172.21.15.137:0/991630685 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f50f802ff60 0x7f50f8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:52:34.678 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:34.673+0000 7f51057fa700 1 -- 172.21.15.137:0/991630685 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f50fc041000 con 0x7f51080f47c0 2023-11-15T01:52:34.678 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:34.673+0000 7f5107fff700 1 --2- 172.21.15.137:0/991630685 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f50f802ff60 0x7f50f8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:52:34.678 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:34.674+0000 7f5107fff700 1 --2- 172.21.15.137:0/991630685 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f50f802ff60 0x7f50f8032420 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f50f40096a0 tx=0x7f50f4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:52:34.679 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:34.678+0000 7f51057fa700 1 -- 172.21.15.137:0/991630685 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f50fc014420 con 0x7f51080f47c0 2023-11-15T01:52:34.886 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:34.885+0000 7f510ef38700 1 -- 172.21.15.137:0/991630685 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f51080fbf60 con 0x7f51080f47c0 2023-11-15T01:52:34.886 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:34.886+0000 7f51057fa700 1 -- 172.21.15.137:0/991630685 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f50fc010230 con 0x7f51080f47c0 2023-11-15T01:52:34.887 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:52:34.887 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:52:34.888 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:34.887+0000 7f510ef38700 1 -- 172.21.15.137:0/991630685 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f50f802ff60 msgr2=0x7f50f8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:52:34.888 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:34.887+0000 7f510ef38700 1 --2- 172.21.15.137:0/991630685 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f50f802ff60 0x7f50f8032420 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f50f40096a0 tx=0x7f50f4006b40).stop 2023-11-15T01:52:34.888 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:34.887+0000 7f510ef38700 1 -- 172.21.15.137:0/991630685 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f51080f47c0 msgr2=0x7f51080f7370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:52:34.889 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:34.887+0000 7f510ef38700 1 --2- 172.21.15.137:0/991630685 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f51080f47c0 0x7f51080f7370 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f50fc013510 tx=0x7f50fc00b040).stop 2023-11-15T01:52:34.889 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:34.887+0000 7f510ef38700 1 -- 172.21.15.137:0/991630685 shutdown_connections 2023-11-15T01:52:34.889 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:34.887+0000 7f510ef38700 1 --2- 172.21.15.137:0/991630685 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f50f802ff60 0x7f50f8032420 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:52:34.889 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:34.887+0000 7f510ef38700 1 --2- 172.21.15.137:0/991630685 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f51080f47c0 0x7f51080f7370 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:52:34.889 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:34.887+0000 7f510ef38700 1 -- 172.21.15.137:0/991630685 >> 172.21.15.137:0/991630685 conn(0x7f51080f02e0 msgr2=0x7f51080f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:52:34.890 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:34.887+0000 7f510ef38700 1 -- 172.21.15.137:0/991630685 shutdown_connections 2023-11-15T01:52:34.890 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:34.887+0000 7f510ef38700 1 -- 172.21.15.137:0/991630685 wait complete. 2023-11-15T01:52:34.890 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:52:36.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:52:35 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:52:34.717543+0000 mgr.smithi055.szwngw (mgr.14164) 87 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:52:36.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:52:35 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:52:34.886878+0000 mon.smithi055 (mon.0) 171 : audit [DBG] from='client.? 172.21.15.137:0/991630685' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:52:36.268 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:52:36.269 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:52:37.438 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:37.437+0000 7f772d84a700 1 -- 172.21.15.137:0/3322532778 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f77280f6510 msgr2=0x7f77280f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:52:37.439 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:37.437+0000 7f772d84a700 1 --2- 172.21.15.137:0/3322532778 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f77280f6510 0x7f77280f6930 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f7718009fb0 tx=0x7f7718009b30).stop 2023-11-15T01:52:37.439 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:37.437+0000 7f772d84a700 1 -- 172.21.15.137:0/3322532778 shutdown_connections 2023-11-15T01:52:37.439 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:37.437+0000 7f772d84a700 1 --2- 172.21.15.137:0/3322532778 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f77280f6510 0x7f77280f6930 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:52:37.440 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:37.437+0000 7f772d84a700 1 -- 172.21.15.137:0/3322532778 >> 172.21.15.137:0/3322532778 conn(0x7f77280f1c70 msgr2=0x7f77280f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:52:37.440 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:37.437+0000 7f772d84a700 1 -- 172.21.15.137:0/3322532778 shutdown_connections 2023-11-15T01:52:37.440 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:37.437+0000 7f772d84a700 1 -- 172.21.15.137:0/3322532778 wait complete. 2023-11-15T01:52:37.440 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:37.438+0000 7f772d84a700 1 Processor -- start 2023-11-15T01:52:37.440 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:37.438+0000 7f772d84a700 1 -- start start 2023-11-15T01:52:37.441 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:37.438+0000 7f772d84a700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f77280f6510 0x7f77281110f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:52:37.441 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:37.438+0000 7f772d84a700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77281115e0 con 0x7f77280f6510 2023-11-15T01:52:37.442 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:37.438+0000 7f7726ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f77280f6510 0x7f77281110f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:52:37.443 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:37.438+0000 7f7726ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f77280f6510 0x7f77281110f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:37078/0 (socket says 172.21.15.137:37078) 2023-11-15T01:52:37.443 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:37.438+0000 7f7726ffd700 1 -- 172.21.15.137:0/3678721447 learned_addr learned my addr 172.21.15.137:0/3678721447 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:52:37.443 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:37.439+0000 7f7726ffd700 1 -- 172.21.15.137:0/3678721447 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7718005040 con 0x7f77280f6510 2023-11-15T01:52:37.444 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:37.439+0000 7f7726ffd700 1 --2- 172.21.15.137:0/3678721447 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f77280f6510 0x7f77281110f0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f7718013510 tx=0x7f771800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:52:37.444 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:37.439+0000 7f770f7fe700 1 -- 172.21.15.137:0/3678721447 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7718009320 con 0x7f77280f6510 2023-11-15T01:52:37.444 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:37.439+0000 7f772d84a700 1 -- 172.21.15.137:0/3678721447 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77281117e0 con 0x7f77280f6510 2023-11-15T01:52:37.444 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:37.439+0000 7f770f7fe700 1 -- 172.21.15.137:0/3678721447 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7718013f00 con 0x7f77280f6510 2023-11-15T01:52:37.444 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:37.439+0000 7f770f7fe700 1 -- 172.21.15.137:0/3678721447 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7718014d90 con 0x7f77280f6510 2023-11-15T01:52:37.445 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:37.439+0000 7f772d84a700 1 -- 172.21.15.137:0/3678721447 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7728111c70 con 0x7f77280f6510 2023-11-15T01:52:37.445 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:37.440+0000 7f770f7fe700 1 -- 172.21.15.137:0/3678721447 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f77180095f0 con 0x7f77280f6510 2023-11-15T01:52:37.445 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:37.440+0000 7f772d84a700 1 -- 172.21.15.137:0/3678721447 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7728045ad0 con 0x7f77280f6510 2023-11-15T01:52:37.445 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:37.440+0000 7f770f7fe700 1 --2- 172.21.15.137:0/3678721447 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f771002ff60 0x7f7710032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:52:37.445 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:37.440+0000 7f770f7fe700 1 -- 172.21.15.137:0/3678721447 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7718040fc0 con 0x7f77280f6510 2023-11-15T01:52:37.446 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:37.440+0000 7f77267fc700 1 --2- 172.21.15.137:0/3678721447 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f771002ff60 0x7f7710032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:52:37.446 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:37.440+0000 7f77267fc700 1 --2- 172.21.15.137:0/3678721447 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f771002ff60 0x7f7710032420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f771c0096a0 tx=0x7f771c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:52:37.446 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:37.445+0000 7f770f7fe700 1 -- 172.21.15.137:0/3678721447 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7718014420 con 0x7f77280f6510 2023-11-15T01:52:37.650 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:37.650+0000 7f772d84a700 1 -- 172.21.15.137:0/3678721447 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7728113290 con 0x7f77280f6510 2023-11-15T01:52:37.651 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:37.651+0000 7f770f7fe700 1 -- 172.21.15.137:0/3678721447 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7718010230 con 0x7f77280f6510 2023-11-15T01:52:37.652 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:52:37.652 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:52:37.653 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:37.652+0000 7f772d84a700 1 -- 172.21.15.137:0/3678721447 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f771002ff60 msgr2=0x7f7710032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:52:37.653 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:37.652+0000 7f772d84a700 1 --2- 172.21.15.137:0/3678721447 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f771002ff60 0x7f7710032420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f771c0096a0 tx=0x7f771c006b40).stop 2023-11-15T01:52:37.654 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:37.652+0000 7f772d84a700 1 -- 172.21.15.137:0/3678721447 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f77280f6510 msgr2=0x7f77281110f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:52:37.654 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:37.652+0000 7f772d84a700 1 --2- 172.21.15.137:0/3678721447 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f77280f6510 0x7f77281110f0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f7718013510 tx=0x7f771800b040).stop 2023-11-15T01:52:37.654 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:37.652+0000 7f772d84a700 1 -- 172.21.15.137:0/3678721447 shutdown_connections 2023-11-15T01:52:37.654 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:37.652+0000 7f772d84a700 1 --2- 172.21.15.137:0/3678721447 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f771002ff60 0x7f7710032420 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:52:37.655 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:37.652+0000 7f772d84a700 1 --2- 172.21.15.137:0/3678721447 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f77280f6510 0x7f77281110f0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:52:37.655 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:37.652+0000 7f772d84a700 1 -- 172.21.15.137:0/3678721447 >> 172.21.15.137:0/3678721447 conn(0x7f77280f1c70 msgr2=0x7f77280f4030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:52:37.655 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:37.653+0000 7f772d84a700 1 -- 172.21.15.137:0/3678721447 shutdown_connections 2023-11-15T01:52:37.655 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:37.653+0000 7f772d84a700 1 -- 172.21.15.137:0/3678721447 wait complete. 2023-11-15T01:52:37.656 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:52:38.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:52:37 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:52:36.717939+0000 mgr.smithi055.szwngw (mgr.14164) 88 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:52:38.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:52:37 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:52:37.651768+0000 mon.smithi055 (mon.0) 172 : audit [DBG] from='client.? 172.21.15.137:0/3678721447' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:52:39.049 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:52:39.050 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:52:40.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:52:39 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:52:38.718278+0000 mgr.smithi055.szwngw (mgr.14164) 89 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:52:40.222 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:40.220+0000 7f80bfe27700 1 -- 172.21.15.137:0/2280323972 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f80b80f5410 msgr2=0x7f80b80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:52:40.223 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:40.220+0000 7f80bfe27700 1 --2- 172.21.15.137:0/2280323972 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f80b80f5410 0x7f80b80f5830 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f80a8009fb0 tx=0x7f80a8009b30).stop 2023-11-15T01:52:40.223 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:40.220+0000 7f80bfe27700 1 -- 172.21.15.137:0/2280323972 shutdown_connections 2023-11-15T01:52:40.223 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:40.220+0000 7f80bfe27700 1 --2- 172.21.15.137:0/2280323972 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f80b80f5410 0x7f80b80f5830 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:52:40.224 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:40.220+0000 7f80bfe27700 1 -- 172.21.15.137:0/2280323972 >> 172.21.15.137:0/2280323972 conn(0x7f80b80f0b70 msgr2=0x7f80b80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:52:40.224 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:40.220+0000 7f80bfe27700 1 -- 172.21.15.137:0/2280323972 shutdown_connections 2023-11-15T01:52:40.224 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:40.220+0000 7f80bfe27700 1 -- 172.21.15.137:0/2280323972 wait complete. 2023-11-15T01:52:40.224 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:40.221+0000 7f80bfe27700 1 Processor -- start 2023-11-15T01:52:40.224 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:40.221+0000 7f80bfe27700 1 -- start start 2023-11-15T01:52:40.225 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:40.221+0000 7f80bfe27700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f80b80f5410 0x7f80b8105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:52:40.225 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:40.221+0000 7f80bfe27700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80b8105b20 con 0x7f80b80f5410 2023-11-15T01:52:40.225 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:40.221+0000 7f80bdbc3700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f80b80f5410 0x7f80b8105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:52:40.225 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:40.221+0000 7f80bdbc3700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f80b80f5410 0x7f80b8105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:48410/0 (socket says 172.21.15.137:48410) 2023-11-15T01:52:40.225 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:40.221+0000 7f80bdbc3700 1 -- 172.21.15.137:0/2133832991 learned_addr learned my addr 172.21.15.137:0/2133832991 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:52:40.226 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:40.221+0000 7f80bdbc3700 1 -- 172.21.15.137:0/2133832991 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80a8005040 con 0x7f80b80f5410 2023-11-15T01:52:40.226 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:40.222+0000 7f80bdbc3700 1 --2- 172.21.15.137:0/2133832991 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f80b80f5410 0x7f80b8105630 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f80a8013510 tx=0x7f80a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:52:40.226 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:40.222+0000 7f80ae7fc700 1 -- 172.21.15.137:0/2133832991 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f80a8009320 con 0x7f80b80f5410 2023-11-15T01:52:40.226 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:40.222+0000 7f80ae7fc700 1 -- 172.21.15.137:0/2133832991 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f80a8013f00 con 0x7f80b80f5410 2023-11-15T01:52:40.227 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:40.222+0000 7f80bfe27700 1 -- 172.21.15.137:0/2133832991 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80b8105d20 con 0x7f80b80f5410 2023-11-15T01:52:40.227 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:40.222+0000 7f80ae7fc700 1 -- 172.21.15.137:0/2133832991 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f80a8014d90 con 0x7f80b80f5410 2023-11-15T01:52:40.227 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:40.222+0000 7f80bfe27700 1 -- 172.21.15.137:0/2133832991 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80b81061b0 con 0x7f80b80f5410 2023-11-15T01:52:40.228 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:40.223+0000 7f80bfe27700 1 -- 172.21.15.137:0/2133832991 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80b8045ad0 con 0x7f80b80f5410 2023-11-15T01:52:40.228 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:40.223+0000 7f80ae7fc700 1 -- 172.21.15.137:0/2133832991 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f80a80095f0 con 0x7f80b80f5410 2023-11-15T01:52:40.228 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:40.223+0000 7f80ae7fc700 1 --2- 172.21.15.137:0/2133832991 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f80a402ff60 0x7f80a4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:52:40.229 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:40.223+0000 7f80ae7fc700 1 -- 172.21.15.137:0/2133832991 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f80a80410c0 con 0x7f80b80f5410 2023-11-15T01:52:40.229 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:40.223+0000 7f80bd3c2700 1 --2- 172.21.15.137:0/2133832991 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f80a402ff60 0x7f80a4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:52:40.229 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:40.224+0000 7f80bd3c2700 1 --2- 172.21.15.137:0/2133832991 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f80a402ff60 0x7f80a4032420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f80b40096a0 tx=0x7f80b4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:52:40.230 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:40.228+0000 7f80ae7fc700 1 -- 172.21.15.137:0/2133832991 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f80a8014420 con 0x7f80b80f5410 2023-11-15T01:52:40.436 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:40.434+0000 7f80bfe27700 1 -- 172.21.15.137:0/2133832991 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f80b80fae60 con 0x7f80b80f5410 2023-11-15T01:52:40.436 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:40.435+0000 7f80ae7fc700 1 -- 172.21.15.137:0/2133832991 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f80a801d7f0 con 0x7f80b80f5410 2023-11-15T01:52:40.436 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:52:40.437 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:52:40.438 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:40.437+0000 7f80bfe27700 1 -- 172.21.15.137:0/2133832991 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f80a402ff60 msgr2=0x7f80a4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:52:40.438 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:40.437+0000 7f80bfe27700 1 --2- 172.21.15.137:0/2133832991 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f80a402ff60 0x7f80a4032420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f80b40096a0 tx=0x7f80b4006b40).stop 2023-11-15T01:52:40.438 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:40.437+0000 7f80bfe27700 1 -- 172.21.15.137:0/2133832991 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f80b80f5410 msgr2=0x7f80b8105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:52:40.438 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:40.437+0000 7f80bfe27700 1 --2- 172.21.15.137:0/2133832991 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f80b80f5410 0x7f80b8105630 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f80a8013510 tx=0x7f80a800b040).stop 2023-11-15T01:52:40.439 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:40.437+0000 7f80bfe27700 1 -- 172.21.15.137:0/2133832991 shutdown_connections 2023-11-15T01:52:40.439 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:40.437+0000 7f80bfe27700 1 --2- 172.21.15.137:0/2133832991 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f80a402ff60 0x7f80a4032420 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:52:40.439 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:40.437+0000 7f80bfe27700 1 --2- 172.21.15.137:0/2133832991 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f80b80f5410 0x7f80b8105630 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:52:40.439 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:40.437+0000 7f80bfe27700 1 -- 172.21.15.137:0/2133832991 >> 172.21.15.137:0/2133832991 conn(0x7f80b80f0b70 msgr2=0x7f80b80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:52:40.439 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:40.437+0000 7f80bfe27700 1 -- 172.21.15.137:0/2133832991 shutdown_connections 2023-11-15T01:52:40.440 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:40.437+0000 7f80bfe27700 1 -- 172.21.15.137:0/2133832991 wait complete. 2023-11-15T01:52:40.440 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:52:41.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:52:40 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:52:40.436233+0000 mon.smithi055 (mon.0) 173 : audit [DBG] from='client.? 172.21.15.137:0/2133832991' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:52:41.836 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:52:41.837 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:52:42.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:52:41 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:52:40.718567+0000 mgr.smithi055.szwngw (mgr.14164) 90 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:52:42.953 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:42.952+0000 7feb7f9f7700 1 -- 172.21.15.137:0/636229349 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feb780f6510 msgr2=0x7feb780f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:52:42.954 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:42.952+0000 7feb7f9f7700 1 --2- 172.21.15.137:0/636229349 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feb780f6510 0x7feb780f6930 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7feb68009fb0 tx=0x7feb68009b30).stop 2023-11-15T01:52:42.954 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:42.952+0000 7feb7f9f7700 1 -- 172.21.15.137:0/636229349 shutdown_connections 2023-11-15T01:52:42.954 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:42.952+0000 7feb7f9f7700 1 --2- 172.21.15.137:0/636229349 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feb780f6510 0x7feb780f6930 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:52:42.954 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:42.952+0000 7feb7f9f7700 1 -- 172.21.15.137:0/636229349 >> 172.21.15.137:0/636229349 conn(0x7feb780f1c70 msgr2=0x7feb780f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:52:42.955 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:42.953+0000 7feb7f9f7700 1 -- 172.21.15.137:0/636229349 shutdown_connections 2023-11-15T01:52:42.955 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:42.953+0000 7feb7f9f7700 1 -- 172.21.15.137:0/636229349 wait complete. 2023-11-15T01:52:42.956 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:42.953+0000 7feb7f9f7700 1 Processor -- start 2023-11-15T01:52:42.957 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:42.953+0000 7feb7f9f7700 1 -- start start 2023-11-15T01:52:42.957 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:42.954+0000 7feb7f9f7700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feb780f6510 0x7feb78106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:52:42.957 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:42.954+0000 7feb7f9f7700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb78106c20 con 0x7feb780f6510 2023-11-15T01:52:42.957 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:42.954+0000 7feb7d793700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feb780f6510 0x7feb78106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:52:42.957 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:42.954+0000 7feb7d793700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feb780f6510 0x7feb78106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:48432/0 (socket says 172.21.15.137:48432) 2023-11-15T01:52:42.958 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:42.954+0000 7feb7d793700 1 -- 172.21.15.137:0/202949298 learned_addr learned my addr 172.21.15.137:0/202949298 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:52:42.958 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:42.954+0000 7feb7d793700 1 -- 172.21.15.137:0/202949298 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb68005040 con 0x7feb780f6510 2023-11-15T01:52:42.958 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:42.954+0000 7feb7d793700 1 --2- 172.21.15.137:0/202949298 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feb780f6510 0x7feb78106730 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7feb68013510 tx=0x7feb6800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:52:42.958 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:42.954+0000 7feb6dffb700 1 -- 172.21.15.137:0/202949298 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb68009320 con 0x7feb780f6510 2023-11-15T01:52:42.958 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:42.954+0000 7feb7f9f7700 1 -- 172.21.15.137:0/202949298 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb78106e20 con 0x7feb780f6510 2023-11-15T01:52:42.959 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:42.954+0000 7feb6dffb700 1 -- 172.21.15.137:0/202949298 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7feb68013f00 con 0x7feb780f6510 2023-11-15T01:52:42.959 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:42.954+0000 7feb6dffb700 1 -- 172.21.15.137:0/202949298 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb68014d90 con 0x7feb780f6510 2023-11-15T01:52:42.959 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:42.955+0000 7feb7f9f7700 1 -- 172.21.15.137:0/202949298 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb781072b0 con 0x7feb780f6510 2023-11-15T01:52:42.959 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:42.955+0000 7feb7f9f7700 1 -- 172.21.15.137:0/202949298 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb78045ad0 con 0x7feb780f6510 2023-11-15T01:52:42.959 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:42.955+0000 7feb6dffb700 1 -- 172.21.15.137:0/202949298 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7feb680095f0 con 0x7feb780f6510 2023-11-15T01:52:42.960 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:42.956+0000 7feb6dffb700 1 --2- 172.21.15.137:0/202949298 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7feb6402ff60 0x7feb64032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:52:42.960 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:42.956+0000 7feb6dffb700 1 -- 172.21.15.137:0/202949298 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7feb68041080 con 0x7feb780f6510 2023-11-15T01:52:42.960 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:42.956+0000 7feb7cf92700 1 --2- 172.21.15.137:0/202949298 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7feb6402ff60 0x7feb64032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:52:42.960 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:42.956+0000 7feb7cf92700 1 --2- 172.21.15.137:0/202949298 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7feb6402ff60 0x7feb64032420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7feb740096a0 tx=0x7feb74006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:52:42.961 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:42.961+0000 7feb6dffb700 1 -- 172.21.15.137:0/202949298 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7feb68014420 con 0x7feb780f6510 2023-11-15T01:52:43.168 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:43.168+0000 7feb7f9f7700 1 -- 172.21.15.137:0/202949298 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7feb780fbf60 con 0x7feb780f6510 2023-11-15T01:52:43.169 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:43.168+0000 7feb6dffb700 1 -- 172.21.15.137:0/202949298 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7feb68010230 con 0x7feb780f6510 2023-11-15T01:52:43.169 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:52:43.170 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:52:43.171 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:43.170+0000 7feb7f9f7700 1 -- 172.21.15.137:0/202949298 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7feb6402ff60 msgr2=0x7feb64032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:52:43.171 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:43.170+0000 7feb7f9f7700 1 --2- 172.21.15.137:0/202949298 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7feb6402ff60 0x7feb64032420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7feb740096a0 tx=0x7feb74006b40).stop 2023-11-15T01:52:43.171 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:43.170+0000 7feb7f9f7700 1 -- 172.21.15.137:0/202949298 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feb780f6510 msgr2=0x7feb78106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:52:43.172 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:43.170+0000 7feb7f9f7700 1 --2- 172.21.15.137:0/202949298 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feb780f6510 0x7feb78106730 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7feb68013510 tx=0x7feb6800b040).stop 2023-11-15T01:52:43.172 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:43.170+0000 7feb7f9f7700 1 -- 172.21.15.137:0/202949298 shutdown_connections 2023-11-15T01:52:43.172 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:43.170+0000 7feb7f9f7700 1 --2- 172.21.15.137:0/202949298 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7feb6402ff60 0x7feb64032420 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:52:43.172 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:43.170+0000 7feb7f9f7700 1 --2- 172.21.15.137:0/202949298 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feb780f6510 0x7feb78106730 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:52:43.172 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:43.170+0000 7feb7f9f7700 1 -- 172.21.15.137:0/202949298 >> 172.21.15.137:0/202949298 conn(0x7feb780f1c70 msgr2=0x7feb780f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:52:43.173 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:43.170+0000 7feb7f9f7700 1 -- 172.21.15.137:0/202949298 shutdown_connections 2023-11-15T01:52:43.173 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:43.170+0000 7feb7f9f7700 1 -- 172.21.15.137:0/202949298 wait complete. 2023-11-15T01:52:43.173 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:52:44.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:52:43 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:52:42.718862+0000 mgr.smithi055.szwngw (mgr.14164) 91 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:52:44.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:52:43 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:52:42.850206+0000 mon.smithi055 (mon.0) 174 : audit [INF] from='mgr.14164 172.21.15.55:0/59463371' entity='mgr.smithi055.szwngw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi055.szwngw/mirror_snapshot_schedule"}]: dispatch 2023-11-15T01:52:44.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:52:43 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:52:42.853023+0000 mon.smithi055 (mon.0) 175 : audit [INF] from='mgr.14164 172.21.15.55:0/59463371' entity='mgr.smithi055.szwngw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi055.szwngw/trash_purge_schedule"}]: dispatch 2023-11-15T01:52:44.079 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:52:43 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:52:43.169416+0000 mon.smithi055 (mon.0) 176 : audit [DBG] from='client.? 172.21.15.137:0/202949298' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:52:44.558 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:52:44.559 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:52:45.659 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:45.657+0000 7faa86bd7700 1 -- 172.21.15.137:0/946210137 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faa800f51a0 msgr2=0x7faa800f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:52:45.660 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:45.657+0000 7faa86bd7700 1 --2- 172.21.15.137:0/946210137 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faa800f51a0 0x7faa800f55c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7faa7c009fb0 tx=0x7faa7c009b30).stop 2023-11-15T01:52:45.660 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:45.658+0000 7faa86bd7700 1 -- 172.21.15.137:0/946210137 shutdown_connections 2023-11-15T01:52:45.660 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:45.658+0000 7faa86bd7700 1 --2- 172.21.15.137:0/946210137 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faa800f51a0 0x7faa800f55c0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:52:45.661 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:45.658+0000 7faa86bd7700 1 -- 172.21.15.137:0/946210137 >> 172.21.15.137:0/946210137 conn(0x7faa800f09a0 msgr2=0x7faa800f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:52:45.661 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:45.658+0000 7faa86bd7700 1 -- 172.21.15.137:0/946210137 shutdown_connections 2023-11-15T01:52:45.661 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:45.658+0000 7faa86bd7700 1 -- 172.21.15.137:0/946210137 wait complete. 2023-11-15T01:52:45.661 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:45.658+0000 7faa86bd7700 1 Processor -- start 2023-11-15T01:52:45.661 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:45.658+0000 7faa86bd7700 1 -- start start 2023-11-15T01:52:45.661 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:45.658+0000 7faa86bd7700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faa800f51a0 0x7faa801064c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:52:45.662 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:45.658+0000 7faa86bd7700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa801069b0 con 0x7faa800f51a0 2023-11-15T01:52:45.662 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:45.659+0000 7faa85bd5700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faa800f51a0 0x7faa801064c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:52:45.662 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:45.659+0000 7faa85bd5700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faa800f51a0 0x7faa801064c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:48450/0 (socket says 172.21.15.137:48450) 2023-11-15T01:52:45.662 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:45.659+0000 7faa85bd5700 1 -- 172.21.15.137:0/4028761600 learned_addr learned my addr 172.21.15.137:0/4028761600 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:52:45.662 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:45.659+0000 7faa85bd5700 1 -- 172.21.15.137:0/4028761600 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa7c005040 con 0x7faa800f51a0 2023-11-15T01:52:45.663 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:45.659+0000 7faa85bd5700 1 --2- 172.21.15.137:0/4028761600 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faa800f51a0 0x7faa801064c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7faa7c002040 tx=0x7faa7c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:52:45.664 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:45.659+0000 7faa767fc700 1 -- 172.21.15.137:0/4028761600 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faa7c0079c0 con 0x7faa800f51a0 2023-11-15T01:52:45.664 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:45.659+0000 7faa86bd7700 1 -- 172.21.15.137:0/4028761600 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa80106bb0 con 0x7faa800f51a0 2023-11-15T01:52:45.664 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:45.659+0000 7faa767fc700 1 -- 172.21.15.137:0/4028761600 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faa7c014420 con 0x7faa800f51a0 2023-11-15T01:52:45.665 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:45.659+0000 7faa767fc700 1 -- 172.21.15.137:0/4028761600 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faa7c01d5a0 con 0x7faa800f51a0 2023-11-15T01:52:45.665 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:45.659+0000 7faa86bd7700 1 -- 172.21.15.137:0/4028761600 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa80107040 con 0x7faa800f51a0 2023-11-15T01:52:45.665 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:45.660+0000 7faa767fc700 1 -- 172.21.15.137:0/4028761600 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7faa7c025070 con 0x7faa800f51a0 2023-11-15T01:52:45.665 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:45.660+0000 7faa86bd7700 1 -- 172.21.15.137:0/4028761600 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa80045ad0 con 0x7faa800f51a0 2023-11-15T01:52:45.665 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:45.660+0000 7faa767fc700 1 --2- 172.21.15.137:0/4028761600 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7faa6c02ff60 0x7faa6c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:52:45.665 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:45.660+0000 7faa767fc700 1 -- 172.21.15.137:0/4028761600 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7faa7c0408f0 con 0x7faa800f51a0 2023-11-15T01:52:45.666 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:45.661+0000 7faa853d4700 1 --2- 172.21.15.137:0/4028761600 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7faa6c02ff60 0x7faa6c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:52:45.666 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:45.661+0000 7faa853d4700 1 --2- 172.21.15.137:0/4028761600 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7faa6c02ff60 0x7faa6c032420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7faa700096a0 tx=0x7faa70006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:52:45.666 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:45.666+0000 7faa767fc700 1 -- 172.21.15.137:0/4028761600 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7faa7c00fb60 con 0x7faa800f51a0 2023-11-15T01:52:45.871 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:45.870+0000 7faa86bd7700 1 -- 172.21.15.137:0/4028761600 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7faa80107ef0 con 0x7faa800f51a0 2023-11-15T01:52:45.872 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:45.871+0000 7faa767fc700 1 -- 172.21.15.137:0/4028761600 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7faa7c010230 con 0x7faa800f51a0 2023-11-15T01:52:45.872 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:52:45.873 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:52:45.874 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:45.873+0000 7faa86bd7700 1 -- 172.21.15.137:0/4028761600 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7faa6c02ff60 msgr2=0x7faa6c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:52:45.874 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:45.873+0000 7faa86bd7700 1 --2- 172.21.15.137:0/4028761600 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7faa6c02ff60 0x7faa6c032420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7faa700096a0 tx=0x7faa70006b40).stop 2023-11-15T01:52:45.874 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:45.873+0000 7faa86bd7700 1 -- 172.21.15.137:0/4028761600 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faa800f51a0 msgr2=0x7faa801064c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:52:45.874 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:45.873+0000 7faa86bd7700 1 --2- 172.21.15.137:0/4028761600 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faa800f51a0 0x7faa801064c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7faa7c002040 tx=0x7faa7c00b040).stop 2023-11-15T01:52:45.875 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:45.873+0000 7faa86bd7700 1 -- 172.21.15.137:0/4028761600 shutdown_connections 2023-11-15T01:52:45.875 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:45.873+0000 7faa86bd7700 1 --2- 172.21.15.137:0/4028761600 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7faa6c02ff60 0x7faa6c032420 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:52:45.875 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:45.873+0000 7faa86bd7700 1 --2- 172.21.15.137:0/4028761600 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faa800f51a0 0x7faa801064c0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:52:45.875 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:45.873+0000 7faa86bd7700 1 -- 172.21.15.137:0/4028761600 >> 172.21.15.137:0/4028761600 conn(0x7faa800f09a0 msgr2=0x7faa800f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:52:45.875 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:45.873+0000 7faa86bd7700 1 -- 172.21.15.137:0/4028761600 shutdown_connections 2023-11-15T01:52:45.876 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:45.873+0000 7faa86bd7700 1 -- 172.21.15.137:0/4028761600 wait complete. 2023-11-15T01:52:45.876 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:52:46.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:52:45 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:52:44.719121+0000 mgr.smithi055.szwngw (mgr.14164) 92 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:52:47.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:52:46 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:52:45.872321+0000 mon.smithi055 (mon.0) 177 : audit [DBG] from='client.? 172.21.15.137:0/4028761600' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:52:47.262 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:52:47.262 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:52:48.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:52:47 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:52:46.719459+0000 mgr.smithi055.szwngw (mgr.14164) 93 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:52:48.432 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:48.431+0000 7fc7abf3b700 1 -- 172.21.15.137:0/4171297122 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc7a40f5410 msgr2=0x7fc7a40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:52:48.433 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:48.431+0000 7fc7abf3b700 1 --2- 172.21.15.137:0/4171297122 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc7a40f5410 0x7fc7a40f5830 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7fc794009fb0 tx=0x7fc794009b30).stop 2023-11-15T01:52:48.433 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:48.431+0000 7fc7abf3b700 1 -- 172.21.15.137:0/4171297122 shutdown_connections 2023-11-15T01:52:48.433 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:48.431+0000 7fc7abf3b700 1 --2- 172.21.15.137:0/4171297122 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc7a40f5410 0x7fc7a40f5830 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:52:48.433 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:48.431+0000 7fc7abf3b700 1 -- 172.21.15.137:0/4171297122 >> 172.21.15.137:0/4171297122 conn(0x7fc7a40f0b70 msgr2=0x7fc7a40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:52:48.433 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:48.431+0000 7fc7abf3b700 1 -- 172.21.15.137:0/4171297122 shutdown_connections 2023-11-15T01:52:48.434 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:48.431+0000 7fc7abf3b700 1 -- 172.21.15.137:0/4171297122 wait complete. 2023-11-15T01:52:48.436 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:48.432+0000 7fc7abf3b700 1 Processor -- start 2023-11-15T01:52:48.436 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:48.432+0000 7fc7abf3b700 1 -- start start 2023-11-15T01:52:48.436 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:48.432+0000 7fc7abf3b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc7a40f5410 0x7fc7a4105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:52:48.436 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:48.432+0000 7fc7abf3b700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a4105b20 con 0x7fc7a40f5410 2023-11-15T01:52:48.436 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:48.433+0000 7fc7a9cd7700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc7a40f5410 0x7fc7a4105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:52:48.437 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:48.433+0000 7fc7a9cd7700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc7a40f5410 0x7fc7a4105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:42958/0 (socket says 172.21.15.137:42958) 2023-11-15T01:52:48.437 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:48.433+0000 7fc7a9cd7700 1 -- 172.21.15.137:0/1086508602 learned_addr learned my addr 172.21.15.137:0/1086508602 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:52:48.437 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:48.433+0000 7fc7a9cd7700 1 -- 172.21.15.137:0/1086508602 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc794005040 con 0x7fc7a40f5410 2023-11-15T01:52:48.437 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:48.433+0000 7fc7a9cd7700 1 --2- 172.21.15.137:0/1086508602 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc7a40f5410 0x7fc7a4105630 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7fc794013510 tx=0x7fc79400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:52:48.437 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:48.433+0000 7fc79a7fc700 1 -- 172.21.15.137:0/1086508602 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc794009320 con 0x7fc7a40f5410 2023-11-15T01:52:48.438 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:48.433+0000 7fc7abf3b700 1 -- 172.21.15.137:0/1086508602 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7a4105d20 con 0x7fc7a40f5410 2023-11-15T01:52:48.438 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:48.433+0000 7fc79a7fc700 1 -- 172.21.15.137:0/1086508602 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc794013f00 con 0x7fc7a40f5410 2023-11-15T01:52:48.438 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:48.433+0000 7fc79a7fc700 1 -- 172.21.15.137:0/1086508602 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc794014d90 con 0x7fc7a40f5410 2023-11-15T01:52:48.438 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:48.433+0000 7fc7abf3b700 1 -- 172.21.15.137:0/1086508602 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7a41061b0 con 0x7fc7a40f5410 2023-11-15T01:52:48.438 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:48.434+0000 7fc79a7fc700 1 -- 172.21.15.137:0/1086508602 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc7940095f0 con 0x7fc7a40f5410 2023-11-15T01:52:48.439 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:48.434+0000 7fc7abf3b700 1 -- 172.21.15.137:0/1086508602 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7a4045ad0 con 0x7fc7a40f5410 2023-11-15T01:52:48.439 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:48.434+0000 7fc79a7fc700 1 --2- 172.21.15.137:0/1086508602 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fc79002ff60 0x7fc790032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:52:48.439 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:48.434+0000 7fc79a7fc700 1 -- 172.21.15.137:0/1086508602 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc794040fc0 con 0x7fc7a40f5410 2023-11-15T01:52:48.439 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:48.434+0000 7fc7a94d6700 1 --2- 172.21.15.137:0/1086508602 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fc79002ff60 0x7fc790032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:52:48.439 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:48.435+0000 7fc7a94d6700 1 --2- 172.21.15.137:0/1086508602 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fc79002ff60 0x7fc790032420 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fc7a00096a0 tx=0x7fc7a0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:52:48.440 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:48.439+0000 7fc79a7fc700 1 -- 172.21.15.137:0/1086508602 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc794014420 con 0x7fc7a40f5410 2023-11-15T01:52:48.647 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:48.646+0000 7fc7abf3b700 1 -- 172.21.15.137:0/1086508602 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc7a40fae60 con 0x7fc7a40f5410 2023-11-15T01:52:48.648 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:48.647+0000 7fc79a7fc700 1 -- 172.21.15.137:0/1086508602 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc794010230 con 0x7fc7a40f5410 2023-11-15T01:52:48.648 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:52:48.648 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:52:48.649 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:48.648+0000 7fc7abf3b700 1 -- 172.21.15.137:0/1086508602 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fc79002ff60 msgr2=0x7fc790032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:52:48.649 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:48.648+0000 7fc7abf3b700 1 --2- 172.21.15.137:0/1086508602 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fc79002ff60 0x7fc790032420 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fc7a00096a0 tx=0x7fc7a0006b40).stop 2023-11-15T01:52:48.650 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:48.648+0000 7fc7abf3b700 1 -- 172.21.15.137:0/1086508602 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc7a40f5410 msgr2=0x7fc7a4105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:52:48.650 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:48.648+0000 7fc7abf3b700 1 --2- 172.21.15.137:0/1086508602 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc7a40f5410 0x7fc7a4105630 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7fc794013510 tx=0x7fc79400b040).stop 2023-11-15T01:52:48.650 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:48.648+0000 7fc7abf3b700 1 -- 172.21.15.137:0/1086508602 shutdown_connections 2023-11-15T01:52:48.650 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:48.648+0000 7fc7abf3b700 1 --2- 172.21.15.137:0/1086508602 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fc79002ff60 0x7fc790032420 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:52:48.650 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:48.648+0000 7fc7abf3b700 1 --2- 172.21.15.137:0/1086508602 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc7a40f5410 0x7fc7a4105630 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:52:48.651 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:48.648+0000 7fc7abf3b700 1 -- 172.21.15.137:0/1086508602 >> 172.21.15.137:0/1086508602 conn(0x7fc7a40f0b70 msgr2=0x7fc7a40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:52:48.651 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:48.648+0000 7fc7abf3b700 1 -- 172.21.15.137:0/1086508602 shutdown_connections 2023-11-15T01:52:48.651 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:48.648+0000 7fc7abf3b700 1 -- 172.21.15.137:0/1086508602 wait complete. 2023-11-15T01:52:48.651 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:52:49.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:52:48 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:52:48.647877+0000 mon.smithi055 (mon.0) 178 : audit [DBG] from='client.? 172.21.15.137:0/1086508602' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:52:50.036 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:52:50.037 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:52:50.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:52:49 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:52:48.719865+0000 mgr.smithi055.szwngw (mgr.14164) 94 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:52:51.167 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:51.165+0000 7f096353f700 1 -- 172.21.15.137:0/3411549228 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f095c0f51a0 msgr2=0x7f095c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:52:51.167 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:51.165+0000 7f096353f700 1 --2- 172.21.15.137:0/3411549228 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f095c0f51a0 0x7f095c0f55c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f0958009fb0 tx=0x7f0958009b30).stop 2023-11-15T01:52:51.167 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:51.165+0000 7f096353f700 1 -- 172.21.15.137:0/3411549228 shutdown_connections 2023-11-15T01:52:51.168 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:51.165+0000 7f096353f700 1 --2- 172.21.15.137:0/3411549228 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f095c0f51a0 0x7f095c0f55c0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:52:51.168 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:51.165+0000 7f096353f700 1 -- 172.21.15.137:0/3411549228 >> 172.21.15.137:0/3411549228 conn(0x7f095c0f09a0 msgr2=0x7f095c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:52:51.168 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:51.165+0000 7f096353f700 1 -- 172.21.15.137:0/3411549228 shutdown_connections 2023-11-15T01:52:51.168 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:51.165+0000 7f096353f700 1 -- 172.21.15.137:0/3411549228 wait complete. 2023-11-15T01:52:51.169 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:51.165+0000 7f096353f700 1 Processor -- start 2023-11-15T01:52:51.169 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:51.165+0000 7f096353f700 1 -- start start 2023-11-15T01:52:51.169 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:51.165+0000 7f096353f700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f095c0f51a0 0x7f095c0fd120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:52:51.170 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:51.165+0000 7f096353f700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f095c0fb5a0 con 0x7f095c0f51a0 2023-11-15T01:52:51.170 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:51.166+0000 7f096253d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f095c0f51a0 0x7f095c0fd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:52:51.170 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:51.166+0000 7f096253d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f095c0f51a0 0x7f095c0fd120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:42976/0 (socket says 172.21.15.137:42976) 2023-11-15T01:52:51.170 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:51.166+0000 7f096253d700 1 -- 172.21.15.137:0/2878360190 learned_addr learned my addr 172.21.15.137:0/2878360190 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:52:51.171 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:51.166+0000 7f096253d700 1 -- 172.21.15.137:0/2878360190 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0958005040 con 0x7f095c0f51a0 2023-11-15T01:52:51.171 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:51.166+0000 7f096253d700 1 --2- 172.21.15.137:0/2878360190 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f095c0f51a0 0x7f095c0fd120 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f0958013510 tx=0x7f095800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:52:51.171 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:51.166+0000 7f0952ffd700 1 -- 172.21.15.137:0/2878360190 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0958009320 con 0x7f095c0f51a0 2023-11-15T01:52:51.171 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:51.166+0000 7f096353f700 1 -- 172.21.15.137:0/2878360190 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f095c0fb740 con 0x7f095c0f51a0 2023-11-15T01:52:51.171 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:51.166+0000 7f0952ffd700 1 -- 172.21.15.137:0/2878360190 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0958013f00 con 0x7f095c0f51a0 2023-11-15T01:52:51.172 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:51.166+0000 7f0952ffd700 1 -- 172.21.15.137:0/2878360190 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0958014dc0 con 0x7f095c0f51a0 2023-11-15T01:52:51.172 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:51.166+0000 7f096353f700 1 -- 172.21.15.137:0/2878360190 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f095c0fbbd0 con 0x7f095c0f51a0 2023-11-15T01:52:51.173 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:51.167+0000 7f0952ffd700 1 -- 172.21.15.137:0/2878360190 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f095801d3d0 con 0x7f095c0f51a0 2023-11-15T01:52:51.174 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:51.167+0000 7f0952ffd700 1 --2- 172.21.15.137:0/2878360190 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f094802ff60 0x7f0948032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:52:51.174 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:51.167+0000 7f0952ffd700 1 -- 172.21.15.137:0/2878360190 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0958040f50 con 0x7f095c0f51a0 2023-11-15T01:52:51.174 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:51.167+0000 7f096353f700 1 -- 172.21.15.137:0/2878360190 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0944004fa0 con 0x7f095c0f51a0 2023-11-15T01:52:51.174 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:51.167+0000 7f0961d3c700 1 --2- 172.21.15.137:0/2878360190 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f094802ff60 0x7f0948032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:52:51.174 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:51.168+0000 7f0961d3c700 1 --2- 172.21.15.137:0/2878360190 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f094802ff60 0x7f0948032420 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f094c0096a0 tx=0x7f094c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:52:51.175 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:51.172+0000 7f0952ffd700 1 -- 172.21.15.137:0/2878360190 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0958026490 con 0x7f095c0f51a0 2023-11-15T01:52:51.378 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:51.377+0000 7f096353f700 1 -- 172.21.15.137:0/2878360190 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0944005670 con 0x7f095c0f51a0 2023-11-15T01:52:51.379 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:51.378+0000 7f0952ffd700 1 -- 172.21.15.137:0/2878360190 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f09580126f0 con 0x7f095c0f51a0 2023-11-15T01:52:51.379 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:52:51.379 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:52:51.381 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:51.380+0000 7f096353f700 1 -- 172.21.15.137:0/2878360190 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f094802ff60 msgr2=0x7f0948032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:52:51.381 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:51.380+0000 7f096353f700 1 --2- 172.21.15.137:0/2878360190 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f094802ff60 0x7f0948032420 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f094c0096a0 tx=0x7f094c006b40).stop 2023-11-15T01:52:51.381 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:51.380+0000 7f096353f700 1 -- 172.21.15.137:0/2878360190 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f095c0f51a0 msgr2=0x7f095c0fd120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:52:51.381 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:51.380+0000 7f096353f700 1 --2- 172.21.15.137:0/2878360190 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f095c0f51a0 0x7f095c0fd120 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f0958013510 tx=0x7f095800b040).stop 2023-11-15T01:52:51.382 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:51.380+0000 7f096353f700 1 -- 172.21.15.137:0/2878360190 shutdown_connections 2023-11-15T01:52:51.382 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:51.380+0000 7f096353f700 1 --2- 172.21.15.137:0/2878360190 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f094802ff60 0x7f0948032420 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:52:51.382 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:51.380+0000 7f096353f700 1 --2- 172.21.15.137:0/2878360190 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f095c0f51a0 0x7f095c0fd120 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:52:51.382 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:51.380+0000 7f096353f700 1 -- 172.21.15.137:0/2878360190 >> 172.21.15.137:0/2878360190 conn(0x7f095c0f09a0 msgr2=0x7f095c05fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:52:51.382 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:51.380+0000 7f096353f700 1 -- 172.21.15.137:0/2878360190 shutdown_connections 2023-11-15T01:52:51.383 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:51.380+0000 7f096353f700 1 -- 172.21.15.137:0/2878360190 wait complete. 2023-11-15T01:52:51.383 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:52:51.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:52:51 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:52:50.720260+0000 mgr.smithi055.szwngw (mgr.14164) 95 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:52:51.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:52:51 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:52:51.379246+0000 mon.smithi055 (mon.0) 179 : audit [DBG] from='client.? 172.21.15.137:0/2878360190' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:52:52.766 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:52:52.766 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:52:53.916 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:53.914+0000 7f147d037700 1 -- 172.21.15.137:0/24476911 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f14780f59d0 msgr2=0x7f14780f5df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:52:53.917 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:53.914+0000 7f147d037700 1 --2- 172.21.15.137:0/24476911 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f14780f59d0 0x7f14780f5df0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7f1460009fb0 tx=0x7f1460009b30).stop 2023-11-15T01:52:53.917 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:53.915+0000 7f147d037700 1 -- 172.21.15.137:0/24476911 shutdown_connections 2023-11-15T01:52:53.917 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:53.915+0000 7f147d037700 1 --2- 172.21.15.137:0/24476911 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f14780f59d0 0x7f14780f5df0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:52:53.917 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:53.915+0000 7f147d037700 1 -- 172.21.15.137:0/24476911 >> 172.21.15.137:0/24476911 conn(0x7f14780f02e0 msgr2=0x7f14780f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:52:53.918 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:53.915+0000 7f147d037700 1 -- 172.21.15.137:0/24476911 shutdown_connections 2023-11-15T01:52:53.918 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:53.915+0000 7f147d037700 1 -- 172.21.15.137:0/24476911 wait complete. 2023-11-15T01:52:53.918 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:53.915+0000 7f147d037700 1 Processor -- start 2023-11-15T01:52:53.918 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:53.915+0000 7f147d037700 1 -- start start 2023-11-15T01:52:53.919 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:53.915+0000 7f147d037700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f14780f59d0 0x7f14781055c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:52:53.919 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:53.915+0000 7f147d037700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1478105ab0 con 0x7f14780f59d0 2023-11-15T01:52:53.919 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:53.916+0000 7f1476d9d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f14780f59d0 0x7f14781055c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:52:53.919 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:53.916+0000 7f1476d9d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f14780f59d0 0x7f14781055c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:42992/0 (socket says 172.21.15.137:42992) 2023-11-15T01:52:53.919 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:53.916+0000 7f1476d9d700 1 -- 172.21.15.137:0/3674953362 learned_addr learned my addr 172.21.15.137:0/3674953362 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:52:53.920 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:53.916+0000 7f1476d9d700 1 -- 172.21.15.137:0/3674953362 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1460005040 con 0x7f14780f59d0 2023-11-15T01:52:53.920 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:53.916+0000 7f1476d9d700 1 --2- 172.21.15.137:0/3674953362 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f14780f59d0 0x7f14781055c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f1460013510 tx=0x7f146000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:52:53.921 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:53.916+0000 7f146f7fe700 1 -- 172.21.15.137:0/3674953362 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1460009320 con 0x7f14780f59d0 2023-11-15T01:52:53.921 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:53.916+0000 7f147d037700 1 -- 172.21.15.137:0/3674953362 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1478105cb0 con 0x7f14780f59d0 2023-11-15T01:52:53.922 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:53.916+0000 7f146f7fe700 1 -- 172.21.15.137:0/3674953362 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1460013f00 con 0x7f14780f59d0 2023-11-15T01:52:53.922 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:53.916+0000 7f146f7fe700 1 -- 172.21.15.137:0/3674953362 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1460014d90 con 0x7f14780f59d0 2023-11-15T01:52:53.922 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:53.916+0000 7f147d037700 1 -- 172.21.15.137:0/3674953362 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1478106140 con 0x7f14780f59d0 2023-11-15T01:52:53.922 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:53.917+0000 7f146f7fe700 1 -- 172.21.15.137:0/3674953362 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f14600095f0 con 0x7f14780f59d0 2023-11-15T01:52:53.923 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:53.917+0000 7f147d037700 1 -- 172.21.15.137:0/3674953362 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1478045ad0 con 0x7f14780f59d0 2023-11-15T01:52:53.923 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:53.917+0000 7f146f7fe700 1 --2- 172.21.15.137:0/3674953362 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f146402ff60 0x7f1464032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:52:53.923 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:53.917+0000 7f146f7fe700 1 -- 172.21.15.137:0/3674953362 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1460040fe0 con 0x7f14780f59d0 2023-11-15T01:52:53.923 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:53.917+0000 7f147659c700 1 --2- 172.21.15.137:0/3674953362 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f146402ff60 0x7f1464032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:52:53.923 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:53.918+0000 7f147659c700 1 --2- 172.21.15.137:0/3674953362 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f146402ff60 0x7f1464032420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f14680096a0 tx=0x7f1468006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:52:53.924 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:53.922+0000 7f146f7fe700 1 -- 172.21.15.137:0/3674953362 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f146001d7f0 con 0x7f14780f59d0 2023-11-15T01:52:54.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:52:53 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:52:52.720646+0000 mgr.smithi055.szwngw (mgr.14164) 96 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:52:54.127 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:54.126+0000 7f147d037700 1 -- 172.21.15.137:0/3674953362 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1478106ff0 con 0x7f14780f59d0 2023-11-15T01:52:54.128 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:54.127+0000 7f146f7fe700 1 -- 172.21.15.137:0/3674953362 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1460010230 con 0x7f14780f59d0 2023-11-15T01:52:54.128 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:52:54.128 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:52:54.129 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:54.128+0000 7f147d037700 1 -- 172.21.15.137:0/3674953362 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f146402ff60 msgr2=0x7f1464032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:52:54.130 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:54.128+0000 7f147d037700 1 --2- 172.21.15.137:0/3674953362 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f146402ff60 0x7f1464032420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f14680096a0 tx=0x7f1468006b40).stop 2023-11-15T01:52:54.130 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:54.128+0000 7f147d037700 1 -- 172.21.15.137:0/3674953362 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f14780f59d0 msgr2=0x7f14781055c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:52:54.130 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:54.128+0000 7f147d037700 1 --2- 172.21.15.137:0/3674953362 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f14780f59d0 0x7f14781055c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f1460013510 tx=0x7f146000b040).stop 2023-11-15T01:52:54.130 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:54.128+0000 7f147d037700 1 -- 172.21.15.137:0/3674953362 shutdown_connections 2023-11-15T01:52:54.131 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:54.128+0000 7f147d037700 1 --2- 172.21.15.137:0/3674953362 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f146402ff60 0x7f1464032420 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:52:54.131 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:54.128+0000 7f147d037700 1 --2- 172.21.15.137:0/3674953362 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f14780f59d0 0x7f14781055c0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:52:54.131 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:54.128+0000 7f147d037700 1 -- 172.21.15.137:0/3674953362 >> 172.21.15.137:0/3674953362 conn(0x7f14780f02e0 msgr2=0x7f14780f0dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:52:54.131 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:54.128+0000 7f147d037700 1 -- 172.21.15.137:0/3674953362 shutdown_connections 2023-11-15T01:52:54.131 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:54.128+0000 7f147d037700 1 -- 172.21.15.137:0/3674953362 wait complete. 2023-11-15T01:52:54.131 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:52:55.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:52:54 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:52:54.128034+0000 mon.smithi055 (mon.0) 180 : audit [DBG] from='client.? 172.21.15.137:0/3674953362' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:52:55.510 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:52:55.511 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:52:56.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:52:55 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:52:54.721016+0000 mgr.smithi055.szwngw (mgr.14164) 97 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:52:56.674 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:56.671+0000 7f724d959700 1 -- 172.21.15.137:0/4136367636 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f72480f5410 msgr2=0x7f72480f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:52:56.675 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:56.671+0000 7f724d959700 1 --2- 172.21.15.137:0/4136367636 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f72480f5410 0x7f72480f5830 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f7240009960 tx=0x7f7240004d10).stop 2023-11-15T01:52:56.675 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:56.671+0000 7f724d959700 1 -- 172.21.15.137:0/4136367636 shutdown_connections 2023-11-15T01:52:56.675 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:56.671+0000 7f724d959700 1 --2- 172.21.15.137:0/4136367636 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f72480f5410 0x7f72480f5830 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:52:56.675 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:56.671+0000 7f724d959700 1 -- 172.21.15.137:0/4136367636 >> 172.21.15.137:0/4136367636 conn(0x7f72480f0b70 msgr2=0x7f72480f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:52:56.676 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:56.671+0000 7f724d959700 1 -- 172.21.15.137:0/4136367636 shutdown_connections 2023-11-15T01:52:56.676 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:56.671+0000 7f724d959700 1 -- 172.21.15.137:0/4136367636 wait complete. 2023-11-15T01:52:56.676 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:56.672+0000 7f724d959700 1 Processor -- start 2023-11-15T01:52:56.676 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:56.672+0000 7f724d959700 1 -- start start 2023-11-15T01:52:56.676 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:56.672+0000 7f724d959700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f72480f5410 0x7f7248109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:52:56.677 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:56.672+0000 7f724d959700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7248060220 con 0x7f72480f5410 2023-11-15T01:52:56.677 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:56.672+0000 7f7246ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f72480f5410 0x7f7248109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:52:56.677 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:56.672+0000 7f7246ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f72480f5410 0x7f7248109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:43000/0 (socket says 172.21.15.137:43000) 2023-11-15T01:52:56.677 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:56.672+0000 7f7246ffd700 1 -- 172.21.15.137:0/2635397877 learned_addr learned my addr 172.21.15.137:0/2635397877 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:52:56.678 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:56.672+0000 7f7246ffd700 1 -- 172.21.15.137:0/2635397877 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7240009640 con 0x7f72480f5410 2023-11-15T01:52:56.678 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:56.672+0000 7f7246ffd700 1 --2- 172.21.15.137:0/2635397877 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f72480f5410 0x7f7248109450 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f7240004b70 tx=0x7f7240004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:52:56.678 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:56.673+0000 7f722f7fe700 1 -- 172.21.15.137:0/2635397877 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7240018070 con 0x7f72480f5410 2023-11-15T01:52:56.678 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:56.673+0000 7f724d959700 1 -- 172.21.15.137:0/2635397877 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7248060420 con 0x7f72480f5410 2023-11-15T01:52:56.678 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:56.673+0000 7f722f7fe700 1 -- 172.21.15.137:0/2635397877 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f724000b870 con 0x7f72480f5410 2023-11-15T01:52:56.679 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:56.673+0000 7f722f7fe700 1 -- 172.21.15.137:0/2635397877 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f724001c560 con 0x7f72480f5410 2023-11-15T01:52:56.679 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:56.673+0000 7f724d959700 1 -- 172.21.15.137:0/2635397877 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7248109b00 con 0x7f72480f5410 2023-11-15T01:52:56.679 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:56.673+0000 7f722f7fe700 1 -- 172.21.15.137:0/2635397877 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f724001c6c0 con 0x7f72480f5410 2023-11-15T01:52:56.680 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:56.674+0000 7f722f7fe700 1 --2- 172.21.15.137:0/2635397877 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f723002ff60 0x7f7230032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:52:56.680 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:56.674+0000 7f722f7fe700 1 -- 172.21.15.137:0/2635397877 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7240041200 con 0x7f72480f5410 2023-11-15T01:52:56.680 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:56.674+0000 7f724d959700 1 -- 172.21.15.137:0/2635397877 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7248059100 con 0x7f72480f5410 2023-11-15T01:52:56.681 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:56.674+0000 7f72467fc700 1 --2- 172.21.15.137:0/2635397877 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f723002ff60 0x7f7230032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:52:56.682 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:56.674+0000 7f72467fc700 1 --2- 172.21.15.137:0/2635397877 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f723002ff60 0x7f7230032420 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f723c0096a0 tx=0x7f723c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:52:56.682 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:56.678+0000 7f722f7fe700 1 -- 172.21.15.137:0/2635397877 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7240012be0 con 0x7f72480f5410 2023-11-15T01:52:56.885 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:56.885+0000 7f724d959700 1 -- 172.21.15.137:0/2635397877 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f724810a1d0 con 0x7f72480f5410 2023-11-15T01:52:56.886 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:56.885+0000 7f722f7fe700 1 -- 172.21.15.137:0/2635397877 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f724002a070 con 0x7f72480f5410 2023-11-15T01:52:56.887 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:52:56.887 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:52:56.888 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:56.887+0000 7f724d959700 1 -- 172.21.15.137:0/2635397877 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f723002ff60 msgr2=0x7f7230032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:52:56.888 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:56.887+0000 7f724d959700 1 --2- 172.21.15.137:0/2635397877 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f723002ff60 0x7f7230032420 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f723c0096a0 tx=0x7f723c006b40).stop 2023-11-15T01:52:56.888 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:56.887+0000 7f724d959700 1 -- 172.21.15.137:0/2635397877 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f72480f5410 msgr2=0x7f7248109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:52:56.888 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:56.887+0000 7f724d959700 1 --2- 172.21.15.137:0/2635397877 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f72480f5410 0x7f7248109450 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f7240004b70 tx=0x7f7240004d10).stop 2023-11-15T01:52:56.889 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:56.887+0000 7f724d959700 1 -- 172.21.15.137:0/2635397877 shutdown_connections 2023-11-15T01:52:56.889 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:56.887+0000 7f724d959700 1 --2- 172.21.15.137:0/2635397877 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f723002ff60 0x7f7230032420 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:52:56.889 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:56.887+0000 7f724d959700 1 --2- 172.21.15.137:0/2635397877 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f72480f5410 0x7f7248109450 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:52:56.889 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:56.887+0000 7f724d959700 1 -- 172.21.15.137:0/2635397877 >> 172.21.15.137:0/2635397877 conn(0x7f72480f0b70 msgr2=0x7f72480f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:52:56.889 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:56.887+0000 7f724d959700 1 -- 172.21.15.137:0/2635397877 shutdown_connections 2023-11-15T01:52:56.890 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:56.887+0000 7f724d959700 1 -- 172.21.15.137:0/2635397877 wait complete. 2023-11-15T01:52:56.890 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:52:58.081 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:52:57 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:52:56.721249+0000 mgr.smithi055.szwngw (mgr.14164) 98 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:52:58.082 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:52:57 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:52:56.886669+0000 mon.smithi055 (mon.0) 181 : audit [DBG] from='client.? 172.21.15.137:0/2635397877' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:52:58.276 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:52:58.276 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:52:59.431 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:59.429+0000 7f253021b700 1 -- 172.21.15.137:0/878562830 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f25280f5410 msgr2=0x7f25280f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:52:59.432 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:59.429+0000 7f253021b700 1 --2- 172.21.15.137:0/878562830 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f25280f5410 0x7f25280f5830 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f2518009fb0 tx=0x7f2518009b30).stop 2023-11-15T01:52:59.432 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:59.429+0000 7f253021b700 1 -- 172.21.15.137:0/878562830 shutdown_connections 2023-11-15T01:52:59.433 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:59.429+0000 7f253021b700 1 --2- 172.21.15.137:0/878562830 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f25280f5410 0x7f25280f5830 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:52:59.433 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:59.429+0000 7f253021b700 1 -- 172.21.15.137:0/878562830 >> 172.21.15.137:0/878562830 conn(0x7f25280f0b70 msgr2=0x7f25280f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:52:59.433 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:59.429+0000 7f253021b700 1 -- 172.21.15.137:0/878562830 shutdown_connections 2023-11-15T01:52:59.433 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:59.429+0000 7f253021b700 1 -- 172.21.15.137:0/878562830 wait complete. 2023-11-15T01:52:59.434 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:59.430+0000 7f253021b700 1 Processor -- start 2023-11-15T01:52:59.434 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:59.430+0000 7f253021b700 1 -- start start 2023-11-15T01:52:59.434 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:59.430+0000 7f253021b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f25280f5410 0x7f2528106710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:52:59.435 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:59.430+0000 7f253021b700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2528106c00 con 0x7f25280f5410 2023-11-15T01:52:59.435 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:59.430+0000 7f252dfb7700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f25280f5410 0x7f2528106710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:52:59.435 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:59.430+0000 7f252dfb7700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f25280f5410 0x7f2528106710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:35038/0 (socket says 172.21.15.137:35038) 2023-11-15T01:52:59.436 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:59.430+0000 7f252dfb7700 1 -- 172.21.15.137:0/2227396094 learned_addr learned my addr 172.21.15.137:0/2227396094 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:52:59.436 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:59.430+0000 7f252dfb7700 1 -- 172.21.15.137:0/2227396094 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2518005040 con 0x7f25280f5410 2023-11-15T01:52:59.436 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:59.431+0000 7f252dfb7700 1 --2- 172.21.15.137:0/2227396094 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f25280f5410 0x7f2528106710 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f2518013510 tx=0x7f251800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:52:59.436 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:59.431+0000 7f251e7fc700 1 -- 172.21.15.137:0/2227396094 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2518009320 con 0x7f25280f5410 2023-11-15T01:52:59.437 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:59.431+0000 7f253021b700 1 -- 172.21.15.137:0/2227396094 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2528106e00 con 0x7f25280f5410 2023-11-15T01:52:59.437 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:59.431+0000 7f251e7fc700 1 -- 172.21.15.137:0/2227396094 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2518013f00 con 0x7f25280f5410 2023-11-15T01:52:59.437 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:59.431+0000 7f251e7fc700 1 -- 172.21.15.137:0/2227396094 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2518014d90 con 0x7f25280f5410 2023-11-15T01:52:59.437 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:59.431+0000 7f253021b700 1 -- 172.21.15.137:0/2227396094 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2528107290 con 0x7f25280f5410 2023-11-15T01:52:59.438 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:59.432+0000 7f251e7fc700 1 -- 172.21.15.137:0/2227396094 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f25180263a0 con 0x7f25280f5410 2023-11-15T01:52:59.439 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:59.432+0000 7f251e7fc700 1 --2- 172.21.15.137:0/2227396094 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f251402ff60 0x7f2514032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:52:59.439 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:59.432+0000 7f253021b700 1 -- 172.21.15.137:0/2227396094 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2528045ad0 con 0x7f25280f5410 2023-11-15T01:52:59.439 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:59.432+0000 7f251e7fc700 1 -- 172.21.15.137:0/2227396094 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f25180412c0 con 0x7f25280f5410 2023-11-15T01:52:59.439 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:59.432+0000 7f252d7b6700 1 --2- 172.21.15.137:0/2227396094 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f251402ff60 0x7f2514032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:52:59.439 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:59.432+0000 7f252d7b6700 1 --2- 172.21.15.137:0/2227396094 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f251402ff60 0x7f2514032420 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f25240096a0 tx=0x7f2524006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:52:59.440 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:59.437+0000 7f251e7fc700 1 -- 172.21.15.137:0/2227396094 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f251801cac0 con 0x7f25280f5410 2023-11-15T01:52:59.643 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:59.641+0000 7f253021b700 1 -- 172.21.15.137:0/2227396094 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f25280faf50 con 0x7f25280f5410 2023-11-15T01:52:59.643 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:59.642+0000 7f251e7fc700 1 -- 172.21.15.137:0/2227396094 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f251801cac0 con 0x7f25280f5410 2023-11-15T01:52:59.643 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:52:59.644 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:52:59.644 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:59.643+0000 7f253021b700 1 -- 172.21.15.137:0/2227396094 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f251402ff60 msgr2=0x7f2514032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:52:59.645 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:59.643+0000 7f253021b700 1 --2- 172.21.15.137:0/2227396094 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f251402ff60 0x7f2514032420 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f25240096a0 tx=0x7f2524006b40).stop 2023-11-15T01:52:59.645 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:59.643+0000 7f253021b700 1 -- 172.21.15.137:0/2227396094 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f25280f5410 msgr2=0x7f2528106710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:52:59.645 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:59.643+0000 7f253021b700 1 --2- 172.21.15.137:0/2227396094 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f25280f5410 0x7f2528106710 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f2518013510 tx=0x7f251800b040).stop 2023-11-15T01:52:59.645 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:59.643+0000 7f253021b700 1 -- 172.21.15.137:0/2227396094 shutdown_connections 2023-11-15T01:52:59.645 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:59.643+0000 7f253021b700 1 --2- 172.21.15.137:0/2227396094 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f251402ff60 0x7f2514032420 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:52:59.645 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:59.643+0000 7f253021b700 1 --2- 172.21.15.137:0/2227396094 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f25280f5410 0x7f2528106710 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:52:59.646 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:59.643+0000 7f253021b700 1 -- 172.21.15.137:0/2227396094 >> 172.21.15.137:0/2227396094 conn(0x7f25280f0b70 msgr2=0x7f252805fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:52:59.646 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:59.643+0000 7f253021b700 1 -- 172.21.15.137:0/2227396094 shutdown_connections 2023-11-15T01:52:59.646 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:52:59.643+0000 7f253021b700 1 -- 172.21.15.137:0/2227396094 wait complete. 2023-11-15T01:52:59.646 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:53:00.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:52:59 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:52:58.721589+0000 mgr.smithi055.szwngw (mgr.14164) 99 : cluster [DBG] pgmap v92: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:53:00.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:52:59 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:52:59.643028+0000 mon.smithi055 (mon.0) 182 : audit [DBG] from='client.? 172.21.15.137:0/2227396094' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:53:01.039 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:53:01.039 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:53:01.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:53:01 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:53:00.721995+0000 mgr.smithi055.szwngw (mgr.14164) 100 : cluster [DBG] pgmap v93: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:53:02.256 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:02.255+0000 7f0657ee9700 1 -- 172.21.15.137:0/3807490313 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f06500f6510 msgr2=0x7f06500f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:53:02.257 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:02.255+0000 7f0657ee9700 1 --2- 172.21.15.137:0/3807490313 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f06500f6510 0x7f06500f6930 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7f0640009fb0 tx=0x7f0640009b30).stop 2023-11-15T01:53:02.257 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:02.255+0000 7f0657ee9700 1 -- 172.21.15.137:0/3807490313 shutdown_connections 2023-11-15T01:53:02.257 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:02.255+0000 7f0657ee9700 1 --2- 172.21.15.137:0/3807490313 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f06500f6510 0x7f06500f6930 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:53:02.258 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:02.255+0000 7f0657ee9700 1 -- 172.21.15.137:0/3807490313 >> 172.21.15.137:0/3807490313 conn(0x7f06500f1c70 msgr2=0x7f06500f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:53:02.258 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:02.255+0000 7f0657ee9700 1 -- 172.21.15.137:0/3807490313 shutdown_connections 2023-11-15T01:53:02.258 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:02.255+0000 7f0657ee9700 1 -- 172.21.15.137:0/3807490313 wait complete. 2023-11-15T01:53:02.258 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:02.255+0000 7f0657ee9700 1 Processor -- start 2023-11-15T01:53:02.258 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:02.255+0000 7f0657ee9700 1 -- start start 2023-11-15T01:53:02.259 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:02.256+0000 7f0657ee9700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f06500f6510 0x7f0650106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:53:02.259 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:02.256+0000 7f0657ee9700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0650106c20 con 0x7f06500f6510 2023-11-15T01:53:02.259 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:02.256+0000 7f0655c85700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f06500f6510 0x7f0650106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:53:02.259 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:02.256+0000 7f0655c85700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f06500f6510 0x7f0650106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:35060/0 (socket says 172.21.15.137:35060) 2023-11-15T01:53:02.259 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:02.256+0000 7f0655c85700 1 -- 172.21.15.137:0/1425908381 learned_addr learned my addr 172.21.15.137:0/1425908381 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:53:02.261 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:02.256+0000 7f0655c85700 1 -- 172.21.15.137:0/1425908381 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0640005040 con 0x7f06500f6510 2023-11-15T01:53:02.261 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:02.257+0000 7f0655c85700 1 --2- 172.21.15.137:0/1425908381 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f06500f6510 0x7f0650106730 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f0640013510 tx=0x7f064000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:53:02.261 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:02.257+0000 7f06467fc700 1 -- 172.21.15.137:0/1425908381 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0640009320 con 0x7f06500f6510 2023-11-15T01:53:02.262 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:02.257+0000 7f0657ee9700 1 -- 172.21.15.137:0/1425908381 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0650106e20 con 0x7f06500f6510 2023-11-15T01:53:02.262 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:02.257+0000 7f06467fc700 1 -- 172.21.15.137:0/1425908381 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0640013f00 con 0x7f06500f6510 2023-11-15T01:53:02.262 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:02.257+0000 7f06467fc700 1 -- 172.21.15.137:0/1425908381 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0640014d90 con 0x7f06500f6510 2023-11-15T01:53:02.262 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:02.257+0000 7f0657ee9700 1 -- 172.21.15.137:0/1425908381 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06501072b0 con 0x7f06500f6510 2023-11-15T01:53:02.263 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:02.258+0000 7f06467fc700 1 -- 172.21.15.137:0/1425908381 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f06400095f0 con 0x7f06500f6510 2023-11-15T01:53:02.263 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:02.258+0000 7f0657ee9700 1 -- 172.21.15.137:0/1425908381 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0650045ad0 con 0x7f06500f6510 2023-11-15T01:53:02.263 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:02.258+0000 7f06467fc700 1 --2- 172.21.15.137:0/1425908381 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f063c02ff60 0x7f063c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:53:02.263 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:02.258+0000 7f06467fc700 1 -- 172.21.15.137:0/1425908381 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0640040fc0 con 0x7f06500f6510 2023-11-15T01:53:02.263 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:02.258+0000 7f0655484700 1 --2- 172.21.15.137:0/1425908381 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f063c02ff60 0x7f063c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:53:02.264 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:02.258+0000 7f0655484700 1 --2- 172.21.15.137:0/1425908381 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f063c02ff60 0x7f063c032420 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f064c0096a0 tx=0x7f064c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:53:02.264 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:02.263+0000 7f06467fc700 1 -- 172.21.15.137:0/1425908381 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0640014420 con 0x7f06500f6510 2023-11-15T01:53:02.467 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:02.467+0000 7f0657ee9700 1 -- 172.21.15.137:0/1425908381 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f06500fbf60 con 0x7f06500f6510 2023-11-15T01:53:02.468 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:02.467+0000 7f06467fc700 1 -- 172.21.15.137:0/1425908381 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0640010230 con 0x7f06500f6510 2023-11-15T01:53:02.468 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:53:02.469 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:53:02.469 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:02.469+0000 7f0657ee9700 1 -- 172.21.15.137:0/1425908381 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f063c02ff60 msgr2=0x7f063c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:53:02.470 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:02.469+0000 7f0657ee9700 1 --2- 172.21.15.137:0/1425908381 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f063c02ff60 0x7f063c032420 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f064c0096a0 tx=0x7f064c006b40).stop 2023-11-15T01:53:02.470 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:02.469+0000 7f0657ee9700 1 -- 172.21.15.137:0/1425908381 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f06500f6510 msgr2=0x7f0650106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:53:02.470 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:02.469+0000 7f0657ee9700 1 --2- 172.21.15.137:0/1425908381 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f06500f6510 0x7f0650106730 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f0640013510 tx=0x7f064000b040).stop 2023-11-15T01:53:02.470 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:02.469+0000 7f0657ee9700 1 -- 172.21.15.137:0/1425908381 shutdown_connections 2023-11-15T01:53:02.471 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:02.469+0000 7f0657ee9700 1 --2- 172.21.15.137:0/1425908381 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f063c02ff60 0x7f063c032420 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:53:02.471 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:02.469+0000 7f0657ee9700 1 --2- 172.21.15.137:0/1425908381 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f06500f6510 0x7f0650106730 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:53:02.471 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:02.469+0000 7f0657ee9700 1 -- 172.21.15.137:0/1425908381 >> 172.21.15.137:0/1425908381 conn(0x7f06500f1c70 msgr2=0x7f06500f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:53:02.471 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:02.469+0000 7f0657ee9700 1 -- 172.21.15.137:0/1425908381 shutdown_connections 2023-11-15T01:53:02.471 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:02.469+0000 7f0657ee9700 1 -- 172.21.15.137:0/1425908381 wait complete. 2023-11-15T01:53:02.472 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:53:02.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:53:02 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:53:02.468347+0000 mon.smithi055 (mon.0) 183 : audit [DBG] from='client.? 172.21.15.137:0/1425908381' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:53:03.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:53:03 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:53:02.722332+0000 mgr.smithi055.szwngw (mgr.14164) 101 : cluster [DBG] pgmap v94: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:53:03.855 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:53:03.856 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:53:04.973 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:04.972+0000 7fa769bd7700 1 -- 172.21.15.137:0/1260464290 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa764060200 msgr2=0x7fa7640f9170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:53:04.974 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:04.972+0000 7fa769bd7700 1 --2- 172.21.15.137:0/1260464290 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa764060200 0x7fa7640f9170 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7fa74c009fb0 tx=0x7fa74c009b30).stop 2023-11-15T01:53:04.974 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:04.972+0000 7fa769bd7700 1 -- 172.21.15.137:0/1260464290 shutdown_connections 2023-11-15T01:53:04.974 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:04.972+0000 7fa769bd7700 1 --2- 172.21.15.137:0/1260464290 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa764060200 0x7fa7640f9170 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:53:04.974 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:04.972+0000 7fa769bd7700 1 -- 172.21.15.137:0/1260464290 >> 172.21.15.137:0/1260464290 conn(0x7fa7640f0b70 msgr2=0x7fa7640f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:53:04.975 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:04.972+0000 7fa769bd7700 1 -- 172.21.15.137:0/1260464290 shutdown_connections 2023-11-15T01:53:04.975 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:04.972+0000 7fa769bd7700 1 -- 172.21.15.137:0/1260464290 wait complete. 2023-11-15T01:53:04.975 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:04.972+0000 7fa769bd7700 1 Processor -- start 2023-11-15T01:53:04.975 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:04.973+0000 7fa769bd7700 1 -- start start 2023-11-15T01:53:04.975 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:04.973+0000 7fa769bd7700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa764060200 0x7fa7640f6bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:53:04.976 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:04.973+0000 7fa769bd7700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7640f5070 con 0x7fa764060200 2023-11-15T01:53:04.976 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:04.973+0000 7fa7637fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa764060200 0x7fa7640f6bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:53:04.976 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:04.973+0000 7fa7637fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa764060200 0x7fa7640f6bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:35076/0 (socket says 172.21.15.137:35076) 2023-11-15T01:53:04.976 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:04.973+0000 7fa7637fe700 1 -- 172.21.15.137:0/821002954 learned_addr learned my addr 172.21.15.137:0/821002954 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:53:04.978 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:04.973+0000 7fa7637fe700 1 -- 172.21.15.137:0/821002954 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa74c005040 con 0x7fa764060200 2023-11-15T01:53:04.978 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:04.973+0000 7fa7637fe700 1 --2- 172.21.15.137:0/821002954 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa764060200 0x7fa7640f6bf0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7fa74c013510 tx=0x7fa74c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:53:04.978 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:04.974+0000 7fa75bfff700 1 -- 172.21.15.137:0/821002954 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa74c009320 con 0x7fa764060200 2023-11-15T01:53:04.978 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:04.974+0000 7fa769bd7700 1 -- 172.21.15.137:0/821002954 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7640f5210 con 0x7fa764060200 2023-11-15T01:53:04.979 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:04.974+0000 7fa75bfff700 1 -- 172.21.15.137:0/821002954 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa74c013f00 con 0x7fa764060200 2023-11-15T01:53:04.979 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:04.974+0000 7fa75bfff700 1 -- 172.21.15.137:0/821002954 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa74c014dc0 con 0x7fa764060200 2023-11-15T01:53:04.979 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:04.974+0000 7fa769bd7700 1 -- 172.21.15.137:0/821002954 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7640f56a0 con 0x7fa764060200 2023-11-15T01:53:04.979 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:04.974+0000 7fa75bfff700 1 -- 172.21.15.137:0/821002954 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa74c0095f0 con 0x7fa764060200 2023-11-15T01:53:04.979 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:04.974+0000 7fa769bd7700 1 -- 172.21.15.137:0/821002954 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa764059100 con 0x7fa764060200 2023-11-15T01:53:04.979 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:04.975+0000 7fa75bfff700 1 --2- 172.21.15.137:0/821002954 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fa75002ff60 0x7fa750032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:53:04.980 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:04.975+0000 7fa75bfff700 1 -- 172.21.15.137:0/821002954 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa74c0410a0 con 0x7fa764060200 2023-11-15T01:53:04.980 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:04.975+0000 7fa762ffd700 1 --2- 172.21.15.137:0/821002954 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fa75002ff60 0x7fa750032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:53:04.980 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:04.975+0000 7fa762ffd700 1 --2- 172.21.15.137:0/821002954 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fa75002ff60 0x7fa750032420 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7fa7540096a0 tx=0x7fa754006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:53:04.981 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:04.979+0000 7fa75bfff700 1 -- 172.21.15.137:0/821002954 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa74c014420 con 0x7fa764060200 2023-11-15T01:53:05.186 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:05.185+0000 7fa769bd7700 1 -- 172.21.15.137:0/821002954 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa764045ad0 con 0x7fa764060200 2023-11-15T01:53:05.186 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:05.186+0000 7fa75bfff700 1 -- 172.21.15.137:0/821002954 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa74c01c920 con 0x7fa764060200 2023-11-15T01:53:05.187 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:53:05.187 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:53:05.188 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:05.187+0000 7fa769bd7700 1 -- 172.21.15.137:0/821002954 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fa75002ff60 msgr2=0x7fa750032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:53:05.188 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:05.187+0000 7fa769bd7700 1 --2- 172.21.15.137:0/821002954 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fa75002ff60 0x7fa750032420 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7fa7540096a0 tx=0x7fa754006b40).stop 2023-11-15T01:53:05.188 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:05.187+0000 7fa769bd7700 1 -- 172.21.15.137:0/821002954 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa764060200 msgr2=0x7fa7640f6bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:53:05.189 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:05.187+0000 7fa769bd7700 1 --2- 172.21.15.137:0/821002954 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa764060200 0x7fa7640f6bf0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7fa74c013510 tx=0x7fa74c00b040).stop 2023-11-15T01:53:05.189 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:05.187+0000 7fa769bd7700 1 -- 172.21.15.137:0/821002954 shutdown_connections 2023-11-15T01:53:05.189 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:05.187+0000 7fa769bd7700 1 --2- 172.21.15.137:0/821002954 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fa75002ff60 0x7fa750032420 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:53:05.189 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:05.187+0000 7fa769bd7700 1 --2- 172.21.15.137:0/821002954 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa764060200 0x7fa7640f6bf0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:53:05.190 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:05.187+0000 7fa769bd7700 1 -- 172.21.15.137:0/821002954 >> 172.21.15.137:0/821002954 conn(0x7fa7640f0b70 msgr2=0x7fa7640f16b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:53:05.190 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:05.187+0000 7fa769bd7700 1 -- 172.21.15.137:0/821002954 shutdown_connections 2023-11-15T01:53:05.190 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:05.187+0000 7fa769bd7700 1 -- 172.21.15.137:0/821002954 wait complete. 2023-11-15T01:53:05.190 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:53:06.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:53:05 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:53:04.722661+0000 mgr.smithi055.szwngw (mgr.14164) 102 : cluster [DBG] pgmap v95: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:53:06.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:53:05 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:53:05.186688+0000 mon.smithi055 (mon.0) 184 : audit [DBG] from='client.? 172.21.15.137:0/821002954' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:53:06.561 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:53:06.562 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:53:07.707 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:07.704+0000 7f29c2e4a700 1 -- 172.21.15.137:0/751981916 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f29bc0f6510 msgr2=0x7f29bc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:53:07.707 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:07.705+0000 7f29c2e4a700 1 --2- 172.21.15.137:0/751981916 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f29bc0f6510 0x7f29bc0f6930 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7f29b0009fb0 tx=0x7f29b0009b30).stop 2023-11-15T01:53:07.707 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:07.705+0000 7f29c2e4a700 1 -- 172.21.15.137:0/751981916 shutdown_connections 2023-11-15T01:53:07.707 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:07.705+0000 7f29c2e4a700 1 --2- 172.21.15.137:0/751981916 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f29bc0f6510 0x7f29bc0f6930 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:53:07.708 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:07.705+0000 7f29c2e4a700 1 -- 172.21.15.137:0/751981916 >> 172.21.15.137:0/751981916 conn(0x7f29bc0f1c70 msgr2=0x7f29bc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:53:07.708 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:07.705+0000 7f29c2e4a700 1 -- 172.21.15.137:0/751981916 shutdown_connections 2023-11-15T01:53:07.708 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:07.705+0000 7f29c2e4a700 1 -- 172.21.15.137:0/751981916 wait complete. 2023-11-15T01:53:07.708 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:07.705+0000 7f29c2e4a700 1 Processor -- start 2023-11-15T01:53:07.708 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:07.705+0000 7f29c2e4a700 1 -- start start 2023-11-15T01:53:07.710 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:07.705+0000 7f29c2e4a700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f29bc0f6510 0x7f29bc06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:53:07.710 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:07.705+0000 7f29c2e4a700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29bc069e20 con 0x7f29bc0f6510 2023-11-15T01:53:07.711 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:07.706+0000 7f29c0be6700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f29bc0f6510 0x7f29bc06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:53:07.711 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:07.706+0000 7f29c0be6700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f29bc0f6510 0x7f29bc06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:35084/0 (socket says 172.21.15.137:35084) 2023-11-15T01:53:07.711 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:07.706+0000 7f29c0be6700 1 -- 172.21.15.137:0/2889533526 learned_addr learned my addr 172.21.15.137:0/2889533526 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:53:07.711 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:07.706+0000 7f29c0be6700 1 -- 172.21.15.137:0/2889533526 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29b0005040 con 0x7f29bc0f6510 2023-11-15T01:53:07.711 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:07.706+0000 7f29c0be6700 1 --2- 172.21.15.137:0/2889533526 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f29bc0f6510 0x7f29bc06b9a0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f29b0013510 tx=0x7f29b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:53:07.712 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:07.706+0000 7f29b97fa700 1 -- 172.21.15.137:0/2889533526 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f29b0009320 con 0x7f29bc0f6510 2023-11-15T01:53:07.712 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:07.707+0000 7f29c2e4a700 1 -- 172.21.15.137:0/2889533526 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29bc069fc0 con 0x7f29bc0f6510 2023-11-15T01:53:07.712 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:07.707+0000 7f29b97fa700 1 -- 172.21.15.137:0/2889533526 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f29b0013f00 con 0x7f29bc0f6510 2023-11-15T01:53:07.712 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:07.707+0000 7f29b97fa700 1 -- 172.21.15.137:0/2889533526 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f29b0014d90 con 0x7f29bc0f6510 2023-11-15T01:53:07.712 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:07.707+0000 7f29c2e4a700 1 -- 172.21.15.137:0/2889533526 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29bc06a450 con 0x7f29bc0f6510 2023-11-15T01:53:07.713 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:07.707+0000 7f29b97fa700 1 -- 172.21.15.137:0/2889533526 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f29b0014420 con 0x7f29bc0f6510 2023-11-15T01:53:07.713 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:07.707+0000 7f29c2e4a700 1 -- 172.21.15.137:0/2889533526 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29bc059100 con 0x7f29bc0f6510 2023-11-15T01:53:07.713 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:07.707+0000 7f29b97fa700 1 --2- 172.21.15.137:0/2889533526 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f29ac02ff60 0x7f29ac032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:53:07.713 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:07.708+0000 7f29b97fa700 1 -- 172.21.15.137:0/2889533526 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f29b0040ff0 con 0x7f29bc0f6510 2023-11-15T01:53:07.713 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:07.708+0000 7f29bbfff700 1 --2- 172.21.15.137:0/2889533526 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f29ac02ff60 0x7f29ac032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:53:07.713 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:07.708+0000 7f29bbfff700 1 --2- 172.21.15.137:0/2889533526 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f29ac02ff60 0x7f29ac032420 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f29a80096a0 tx=0x7f29a8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:53:07.714 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:07.712+0000 7f29b97fa700 1 -- 172.21.15.137:0/2889533526 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f29b0026bf0 con 0x7f29bc0f6510 2023-11-15T01:53:07.919 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:07.917+0000 7f29c2e4a700 1 -- 172.21.15.137:0/2889533526 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f29bc0fbf60 con 0x7f29bc0f6510 2023-11-15T01:53:07.920 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:07.918+0000 7f29b97fa700 1 -- 172.21.15.137:0/2889533526 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f29b001db70 con 0x7f29bc0f6510 2023-11-15T01:53:07.920 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:53:07.920 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:53:07.921 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:07.920+0000 7f29c2e4a700 1 -- 172.21.15.137:0/2889533526 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f29ac02ff60 msgr2=0x7f29ac032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:53:07.922 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:07.920+0000 7f29c2e4a700 1 --2- 172.21.15.137:0/2889533526 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f29ac02ff60 0x7f29ac032420 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f29a80096a0 tx=0x7f29a8006b40).stop 2023-11-15T01:53:07.922 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:07.920+0000 7f29c2e4a700 1 -- 172.21.15.137:0/2889533526 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f29bc0f6510 msgr2=0x7f29bc06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:53:07.922 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:07.920+0000 7f29c2e4a700 1 --2- 172.21.15.137:0/2889533526 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f29bc0f6510 0x7f29bc06b9a0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f29b0013510 tx=0x7f29b000b040).stop 2023-11-15T01:53:07.922 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:07.920+0000 7f29c2e4a700 1 -- 172.21.15.137:0/2889533526 shutdown_connections 2023-11-15T01:53:07.922 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:07.920+0000 7f29c2e4a700 1 --2- 172.21.15.137:0/2889533526 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f29ac02ff60 0x7f29ac032420 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:53:07.923 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:07.920+0000 7f29c2e4a700 1 --2- 172.21.15.137:0/2889533526 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f29bc0f6510 0x7f29bc06b9a0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:53:07.923 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:07.920+0000 7f29c2e4a700 1 -- 172.21.15.137:0/2889533526 >> 172.21.15.137:0/2889533526 conn(0x7f29bc0f1c70 msgr2=0x7f29bc0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:53:07.923 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:07.920+0000 7f29c2e4a700 1 -- 172.21.15.137:0/2889533526 shutdown_connections 2023-11-15T01:53:07.923 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:07.920+0000 7f29c2e4a700 1 -- 172.21.15.137:0/2889533526 wait complete. 2023-11-15T01:53:07.924 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:53:08.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:53:07 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:53:06.722876+0000 mgr.smithi055.szwngw (mgr.14164) 103 : cluster [DBG] pgmap v96: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:53:09.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:53:08 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:53:07.920279+0000 mon.smithi055 (mon.0) 185 : audit [DBG] from='client.? 172.21.15.137:0/2889533526' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:53:09.309 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:53:09.310 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:53:10.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:53:09 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:53:08.723149+0000 mgr.smithi055.szwngw (mgr.14164) 104 : cluster [DBG] pgmap v97: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:53:10.463 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:10.461+0000 7f5698f5b700 1 -- 172.21.15.137:0/339513766 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f56940f6510 msgr2=0x7f56940f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:53:10.464 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:10.461+0000 7f5698f5b700 1 --2- 172.21.15.137:0/339513766 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f56940f6510 0x7f56940f6930 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7f5684009fb0 tx=0x7f5684009b30).stop 2023-11-15T01:53:10.464 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:10.461+0000 7f5698f5b700 1 -- 172.21.15.137:0/339513766 shutdown_connections 2023-11-15T01:53:10.464 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:10.461+0000 7f5698f5b700 1 --2- 172.21.15.137:0/339513766 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f56940f6510 0x7f56940f6930 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:53:10.464 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:10.461+0000 7f5698f5b700 1 -- 172.21.15.137:0/339513766 >> 172.21.15.137:0/339513766 conn(0x7f56940f1c70 msgr2=0x7f56940f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:53:10.465 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:10.461+0000 7f5698f5b700 1 -- 172.21.15.137:0/339513766 shutdown_connections 2023-11-15T01:53:10.465 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:10.461+0000 7f5698f5b700 1 -- 172.21.15.137:0/339513766 wait complete. 2023-11-15T01:53:10.465 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:10.462+0000 7f5698f5b700 1 Processor -- start 2023-11-15T01:53:10.465 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:10.462+0000 7f5698f5b700 1 -- start start 2023-11-15T01:53:10.465 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:10.462+0000 7f5698f5b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f56940f6510 0x7f5694106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:53:10.466 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:10.462+0000 7f5698f5b700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5694106c20 con 0x7f56940f6510 2023-11-15T01:53:10.467 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:10.462+0000 7f569259c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f56940f6510 0x7f5694106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:53:10.467 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:10.462+0000 7f569259c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f56940f6510 0x7f5694106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:41678/0 (socket says 172.21.15.137:41678) 2023-11-15T01:53:10.468 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:10.463+0000 7f569259c700 1 -- 172.21.15.137:0/2178508746 learned_addr learned my addr 172.21.15.137:0/2178508746 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:53:10.468 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:10.463+0000 7f569259c700 1 -- 172.21.15.137:0/2178508746 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5684005040 con 0x7f56940f6510 2023-11-15T01:53:10.468 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:10.463+0000 7f569259c700 1 --2- 172.21.15.137:0/2178508746 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f56940f6510 0x7f5694106730 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f5684013510 tx=0x7f568400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:53:10.468 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:10.463+0000 7f568affd700 1 -- 172.21.15.137:0/2178508746 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5684009320 con 0x7f56940f6510 2023-11-15T01:53:10.468 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:10.463+0000 7f5698f5b700 1 -- 172.21.15.137:0/2178508746 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5694106e20 con 0x7f56940f6510 2023-11-15T01:53:10.469 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:10.463+0000 7f5698f5b700 1 -- 172.21.15.137:0/2178508746 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56941072b0 con 0x7f56940f6510 2023-11-15T01:53:10.469 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:10.463+0000 7f568affd700 1 -- 172.21.15.137:0/2178508746 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5684013f00 con 0x7f56940f6510 2023-11-15T01:53:10.469 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:10.463+0000 7f568affd700 1 -- 172.21.15.137:0/2178508746 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5684014cc0 con 0x7f56940f6510 2023-11-15T01:53:10.469 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:10.464+0000 7f568affd700 1 -- 172.21.15.137:0/2178508746 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f56840263a0 con 0x7f56940f6510 2023-11-15T01:53:10.470 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:10.464+0000 7f5698f5b700 1 -- 172.21.15.137:0/2178508746 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5678004fa0 con 0x7f56940f6510 2023-11-15T01:53:10.470 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:10.464+0000 7f568affd700 1 --2- 172.21.15.137:0/2178508746 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f568002ff60 0x7f5680032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:53:10.470 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:10.464+0000 7f568affd700 1 -- 172.21.15.137:0/2178508746 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f56840411c0 con 0x7f56940f6510 2023-11-15T01:53:10.470 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:10.464+0000 7f5691d9b700 1 --2- 172.21.15.137:0/2178508746 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f568002ff60 0x7f5680032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:53:10.470 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:10.465+0000 7f5691d9b700 1 --2- 172.21.15.137:0/2178508746 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f568002ff60 0x7f5680032420 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f567c0096a0 tx=0x7f567c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:53:10.471 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:10.469+0000 7f568affd700 1 -- 172.21.15.137:0/2178508746 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5684012330 con 0x7f56940f6510 2023-11-15T01:53:10.676 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:10.674+0000 7f5698f5b700 1 -- 172.21.15.137:0/2178508746 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5678005670 con 0x7f56940f6510 2023-11-15T01:53:10.677 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:10.675+0000 7f568affd700 1 -- 172.21.15.137:0/2178508746 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5684012330 con 0x7f56940f6510 2023-11-15T01:53:10.677 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:53:10.677 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:53:10.678 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:10.677+0000 7f5698f5b700 1 -- 172.21.15.137:0/2178508746 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f568002ff60 msgr2=0x7f5680032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:53:10.679 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:10.677+0000 7f5698f5b700 1 --2- 172.21.15.137:0/2178508746 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f568002ff60 0x7f5680032420 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f567c0096a0 tx=0x7f567c006b40).stop 2023-11-15T01:53:10.679 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:10.677+0000 7f5698f5b700 1 -- 172.21.15.137:0/2178508746 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f56940f6510 msgr2=0x7f5694106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:53:10.679 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:10.677+0000 7f5698f5b700 1 --2- 172.21.15.137:0/2178508746 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f56940f6510 0x7f5694106730 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f5684013510 tx=0x7f568400b040).stop 2023-11-15T01:53:10.679 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:10.677+0000 7f5698f5b700 1 -- 172.21.15.137:0/2178508746 shutdown_connections 2023-11-15T01:53:10.680 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:10.677+0000 7f5698f5b700 1 --2- 172.21.15.137:0/2178508746 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f568002ff60 0x7f5680032420 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:53:10.680 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:10.677+0000 7f5698f5b700 1 --2- 172.21.15.137:0/2178508746 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f56940f6510 0x7f5694106730 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:53:10.680 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:10.677+0000 7f5698f5b700 1 -- 172.21.15.137:0/2178508746 >> 172.21.15.137:0/2178508746 conn(0x7f56940f1c70 msgr2=0x7f56940f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:53:10.680 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:10.677+0000 7f5698f5b700 1 -- 172.21.15.137:0/2178508746 shutdown_connections 2023-11-15T01:53:10.680 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:10.677+0000 7f5698f5b700 1 -- 172.21.15.137:0/2178508746 wait complete. 2023-11-15T01:53:10.681 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:53:11.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:53:11 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:53:10.677347+0000 mon.smithi055 (mon.0) 186 : audit [DBG] from='client.? 172.21.15.137:0/2178508746' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:53:11.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:53:11 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:53:10.723452+0000 mgr.smithi055.szwngw (mgr.14164) 105 : cluster [DBG] pgmap v98: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:53:12.074 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:53:12.075 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:53:13.211 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:13.209+0000 7f38cefa6700 1 -- 172.21.15.137:0/569626484 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f38c80f6510 msgr2=0x7f38c80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:53:13.211 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:13.209+0000 7f38cefa6700 1 --2- 172.21.15.137:0/569626484 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f38c80f6510 0x7f38c80f6930 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7f38bc009fb0 tx=0x7f38bc009b30).stop 2023-11-15T01:53:13.211 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:13.209+0000 7f38cefa6700 1 -- 172.21.15.137:0/569626484 shutdown_connections 2023-11-15T01:53:13.212 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:13.209+0000 7f38cefa6700 1 --2- 172.21.15.137:0/569626484 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f38c80f6510 0x7f38c80f6930 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:53:13.212 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:13.209+0000 7f38cefa6700 1 -- 172.21.15.137:0/569626484 >> 172.21.15.137:0/569626484 conn(0x7f38c80f1c70 msgr2=0x7f38c80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:53:13.212 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:13.209+0000 7f38cefa6700 1 -- 172.21.15.137:0/569626484 shutdown_connections 2023-11-15T01:53:13.212 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:13.209+0000 7f38cefa6700 1 -- 172.21.15.137:0/569626484 wait complete. 2023-11-15T01:53:13.215 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:13.210+0000 7f38cefa6700 1 Processor -- start 2023-11-15T01:53:13.215 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:13.210+0000 7f38cefa6700 1 -- start start 2023-11-15T01:53:13.215 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:13.210+0000 7f38cefa6700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f38c80f6510 0x7f38c8106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:53:13.215 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:13.210+0000 7f38cefa6700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38c8106c20 con 0x7f38c80f6510 2023-11-15T01:53:13.215 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:13.210+0000 7f38ccd42700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f38c80f6510 0x7f38c8106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:53:13.216 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:13.210+0000 7f38ccd42700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f38c80f6510 0x7f38c8106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:41690/0 (socket says 172.21.15.137:41690) 2023-11-15T01:53:13.216 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:13.210+0000 7f38ccd42700 1 -- 172.21.15.137:0/2471269973 learned_addr learned my addr 172.21.15.137:0/2471269973 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:53:13.216 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:13.210+0000 7f38ccd42700 1 -- 172.21.15.137:0/2471269973 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38bc005040 con 0x7f38c80f6510 2023-11-15T01:53:13.216 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:13.211+0000 7f38ccd42700 1 --2- 172.21.15.137:0/2471269973 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f38c80f6510 0x7f38c8106730 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f38bc013510 tx=0x7f38bc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:53:13.217 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:13.211+0000 7f38c57fa700 1 -- 172.21.15.137:0/2471269973 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f38bc009320 con 0x7f38c80f6510 2023-11-15T01:53:13.217 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:13.211+0000 7f38cefa6700 1 -- 172.21.15.137:0/2471269973 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38c8106e20 con 0x7f38c80f6510 2023-11-15T01:53:13.217 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:13.211+0000 7f38c57fa700 1 -- 172.21.15.137:0/2471269973 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f38bc013f00 con 0x7f38c80f6510 2023-11-15T01:53:13.217 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:13.211+0000 7f38c57fa700 1 -- 172.21.15.137:0/2471269973 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f38bc014d90 con 0x7f38c80f6510 2023-11-15T01:53:13.218 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:13.211+0000 7f38cefa6700 1 -- 172.21.15.137:0/2471269973 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38c81072b0 con 0x7f38c80f6510 2023-11-15T01:53:13.218 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:13.212+0000 7f38cefa6700 1 -- 172.21.15.137:0/2471269973 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38c8045ad0 con 0x7f38c80f6510 2023-11-15T01:53:13.220 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:13.212+0000 7f38c57fa700 1 -- 172.21.15.137:0/2471269973 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f38bc0095f0 con 0x7f38c80f6510 2023-11-15T01:53:13.220 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:13.212+0000 7f38c57fa700 1 --2- 172.21.15.137:0/2471269973 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f38b802ff60 0x7f38b8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:53:13.220 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:13.212+0000 7f38c57fa700 1 -- 172.21.15.137:0/2471269973 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f38bc0410a0 con 0x7f38c80f6510 2023-11-15T01:53:13.220 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:13.212+0000 7f38c7fff700 1 --2- 172.21.15.137:0/2471269973 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f38b802ff60 0x7f38b8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:53:13.220 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:13.212+0000 7f38c7fff700 1 --2- 172.21.15.137:0/2471269973 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f38b802ff60 0x7f38b8032420 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f38b40096a0 tx=0x7f38b4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:53:13.222 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:13.217+0000 7f38c57fa700 1 -- 172.21.15.137:0/2471269973 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f38bc01d7f0 con 0x7f38c80f6510 2023-11-15T01:53:13.431 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:13.429+0000 7f38cefa6700 1 -- 172.21.15.137:0/2471269973 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f38c80fbf60 con 0x7f38c80f6510 2023-11-15T01:53:13.463 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:13.430+0000 7f38c57fa700 1 -- 172.21.15.137:0/2471269973 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f38bc010230 con 0x7f38c80f6510 2023-11-15T01:53:13.464 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:53:13.464 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:53:13.465 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:13.431+0000 7f38cefa6700 1 -- 172.21.15.137:0/2471269973 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f38b802ff60 msgr2=0x7f38b8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:53:13.465 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:13.431+0000 7f38cefa6700 1 --2- 172.21.15.137:0/2471269973 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f38b802ff60 0x7f38b8032420 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f38b40096a0 tx=0x7f38b4006b40).stop 2023-11-15T01:53:13.465 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:13.431+0000 7f38cefa6700 1 -- 172.21.15.137:0/2471269973 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f38c80f6510 msgr2=0x7f38c8106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:53:13.466 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:13.431+0000 7f38cefa6700 1 --2- 172.21.15.137:0/2471269973 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f38c80f6510 0x7f38c8106730 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f38bc013510 tx=0x7f38bc00b040).stop 2023-11-15T01:53:13.466 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:13.432+0000 7f38cefa6700 1 -- 172.21.15.137:0/2471269973 shutdown_connections 2023-11-15T01:53:13.466 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:13.432+0000 7f38cefa6700 1 --2- 172.21.15.137:0/2471269973 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f38b802ff60 0x7f38b8032420 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:53:13.466 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:13.432+0000 7f38cefa6700 1 --2- 172.21.15.137:0/2471269973 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f38c80f6510 0x7f38c8106730 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:53:13.467 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:13.432+0000 7f38cefa6700 1 -- 172.21.15.137:0/2471269973 >> 172.21.15.137:0/2471269973 conn(0x7f38c80f1c70 msgr2=0x7f38c80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:53:13.467 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:13.432+0000 7f38cefa6700 1 -- 172.21.15.137:0/2471269973 shutdown_connections 2023-11-15T01:53:13.467 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:13.432+0000 7f38cefa6700 1 -- 172.21.15.137:0/2471269973 wait complete. 2023-11-15T01:53:13.467 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:53:14.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:53:13 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:53:12.723715+0000 mgr.smithi055.szwngw (mgr.14164) 106 : cluster [DBG] pgmap v99: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:53:14.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:53:13 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:53:13.432010+0000 mon.smithi055 (mon.0) 187 : audit [DBG] from='client.? 172.21.15.137:0/2471269973' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:53:14.833 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:53:14.834 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:53:15.995 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:15.992+0000 7f15fb59e700 1 -- 172.21.15.137:0/1762287509 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f15fc0f51a0 msgr2=0x7f15fc0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:53:15.995 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:15.992+0000 7f15fb59e700 1 --2- 172.21.15.137:0/1762287509 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f15fc0f51a0 0x7f15fc0f55c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f15e4009fb0 tx=0x7f15e4009b30).stop 2023-11-15T01:53:15.995 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:15.992+0000 7f15fb59e700 1 -- 172.21.15.137:0/1762287509 shutdown_connections 2023-11-15T01:53:15.996 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:15.992+0000 7f15fb59e700 1 --2- 172.21.15.137:0/1762287509 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f15fc0f51a0 0x7f15fc0f55c0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:53:15.996 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:15.992+0000 7f15fb59e700 1 -- 172.21.15.137:0/1762287509 >> 172.21.15.137:0/1762287509 conn(0x7f15fc0f09a0 msgr2=0x7f15fc0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:53:15.996 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:15.993+0000 7f15fb59e700 1 -- 172.21.15.137:0/1762287509 shutdown_connections 2023-11-15T01:53:15.996 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:15.993+0000 7f15fb59e700 1 -- 172.21.15.137:0/1762287509 wait complete. 2023-11-15T01:53:15.996 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:15.993+0000 7f15fb59e700 1 Processor -- start 2023-11-15T01:53:15.997 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:15.993+0000 7f15fb59e700 1 -- start start 2023-11-15T01:53:15.997 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:15.993+0000 7f15fb59e700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f15fc0f51a0 0x7f15fc1091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:53:15.997 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:15.993+0000 7f15fb59e700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15fc060240 con 0x7f15fc0f51a0 2023-11-15T01:53:15.999 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:15.994+0000 7f15fa59c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f15fc0f51a0 0x7f15fc1091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:53:15.999 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:15.994+0000 7f15fa59c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f15fc0f51a0 0x7f15fc1091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:41706/0 (socket says 172.21.15.137:41706) 2023-11-15T01:53:16.000 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:15.994+0000 7f15fa59c700 1 -- 172.21.15.137:0/2658111491 learned_addr learned my addr 172.21.15.137:0/2658111491 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:53:16.000 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:15.994+0000 7f15fa59c700 1 -- 172.21.15.137:0/2658111491 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15e4005040 con 0x7f15fc0f51a0 2023-11-15T01:53:16.000 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:15.994+0000 7f15fa59c700 1 --2- 172.21.15.137:0/2658111491 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f15fc0f51a0 0x7f15fc1091b0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f15e4009830 tx=0x7f15e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:53:16.000 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:15.994+0000 7f15f2ffd700 1 -- 172.21.15.137:0/2658111491 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f15e4009320 con 0x7f15fc0f51a0 2023-11-15T01:53:16.001 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:15.994+0000 7f15f2ffd700 1 -- 172.21.15.137:0/2658111491 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f15e40136c0 con 0x7f15fc0f51a0 2023-11-15T01:53:16.001 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:15.994+0000 7f15fb59e700 1 -- 172.21.15.137:0/2658111491 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15fc060440 con 0x7f15fc0f51a0 2023-11-15T01:53:16.001 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:15.994+0000 7f15f2ffd700 1 -- 172.21.15.137:0/2658111491 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f15e401ee90 con 0x7f15fc0f51a0 2023-11-15T01:53:16.001 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:15.994+0000 7f15fb59e700 1 -- 172.21.15.137:0/2658111491 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15fc109860 con 0x7f15fc0f51a0 2023-11-15T01:53:16.001 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:15.995+0000 7f15f2ffd700 1 -- 172.21.15.137:0/2658111491 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f15e401e420 con 0x7f15fc0f51a0 2023-11-15T01:53:16.002 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:15.995+0000 7f15fb59e700 1 -- 172.21.15.137:0/2658111491 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15fc045ad0 con 0x7f15fc0f51a0 2023-11-15T01:53:16.002 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:15.995+0000 7f15f2ffd700 1 --2- 172.21.15.137:0/2658111491 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f15e802ff10 0x7f15e80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:53:16.002 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:15.995+0000 7f15f2ffd700 1 -- 172.21.15.137:0/2658111491 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f15e4041120 con 0x7f15fc0f51a0 2023-11-15T01:53:16.002 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:15.995+0000 7f15f9d9b700 1 --2- 172.21.15.137:0/2658111491 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f15e802ff10 0x7f15e80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:53:16.002 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:15.996+0000 7f15f9d9b700 1 --2- 172.21.15.137:0/2658111491 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f15e802ff10 0x7f15e80323d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f15ec0096a0 tx=0x7f15ec006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:53:16.004 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:16.001+0000 7f15f2ffd700 1 -- 172.21.15.137:0/2658111491 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f15e40095f0 con 0x7f15fc0f51a0 2023-11-15T01:53:16.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:53:15 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:53:14.724039+0000 mgr.smithi055.szwngw (mgr.14164) 107 : cluster [DBG] pgmap v100: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:53:16.210 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:16.209+0000 7f15fb59e700 1 -- 172.21.15.137:0/2658111491 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f15fc10a080 con 0x7f15fc0f51a0 2023-11-15T01:53:16.211 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:16.209+0000 7f15f2ffd700 1 -- 172.21.15.137:0/2658111491 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f15e401cb10 con 0x7f15fc0f51a0 2023-11-15T01:53:16.212 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:53:16.212 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:53:16.213 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:16.211+0000 7f15fb59e700 1 -- 172.21.15.137:0/2658111491 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f15e802ff10 msgr2=0x7f15e80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:53:16.213 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:16.211+0000 7f15fb59e700 1 --2- 172.21.15.137:0/2658111491 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f15e802ff10 0x7f15e80323d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f15ec0096a0 tx=0x7f15ec006b40).stop 2023-11-15T01:53:16.213 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:16.211+0000 7f15fb59e700 1 -- 172.21.15.137:0/2658111491 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f15fc0f51a0 msgr2=0x7f15fc1091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:53:16.213 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:16.211+0000 7f15fb59e700 1 --2- 172.21.15.137:0/2658111491 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f15fc0f51a0 0x7f15fc1091b0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f15e4009830 tx=0x7f15e400b040).stop 2023-11-15T01:53:16.213 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:16.211+0000 7f15fb59e700 1 -- 172.21.15.137:0/2658111491 shutdown_connections 2023-11-15T01:53:16.214 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:16.211+0000 7f15fb59e700 1 --2- 172.21.15.137:0/2658111491 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f15e802ff10 0x7f15e80323d0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:53:16.214 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:16.211+0000 7f15fb59e700 1 --2- 172.21.15.137:0/2658111491 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f15fc0f51a0 0x7f15fc1091b0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:53:16.214 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:16.211+0000 7f15fb59e700 1 -- 172.21.15.137:0/2658111491 >> 172.21.15.137:0/2658111491 conn(0x7f15fc0f09a0 msgr2=0x7f15fc0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:53:16.214 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:16.211+0000 7f15fb59e700 1 -- 172.21.15.137:0/2658111491 shutdown_connections 2023-11-15T01:53:16.214 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:16.211+0000 7f15fb59e700 1 -- 172.21.15.137:0/2658111491 wait complete. 2023-11-15T01:53:16.215 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:53:17.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:53:16 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:53:16.211565+0000 mon.smithi055 (mon.0) 188 : audit [DBG] from='client.? 172.21.15.137:0/2658111491' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:53:17.593 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:53:17.594 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:53:18.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:53:17 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:53:16.724310+0000 mgr.smithi055.szwngw (mgr.14164) 108 : cluster [DBG] pgmap v101: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:53:18.724 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:18.722+0000 7f7a5f40a700 1 -- 172.21.15.137:0/4089970454 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7a580f5410 msgr2=0x7f7a580f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:53:18.724 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:18.722+0000 7f7a5f40a700 1 --2- 172.21.15.137:0/4089970454 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7a580f5410 0x7f7a580f5830 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7f7a54009fb0 tx=0x7f7a54009b30).stop 2023-11-15T01:53:18.724 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:18.722+0000 7f7a5f40a700 1 -- 172.21.15.137:0/4089970454 shutdown_connections 2023-11-15T01:53:18.725 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:18.722+0000 7f7a5f40a700 1 --2- 172.21.15.137:0/4089970454 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7a580f5410 0x7f7a580f5830 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:53:18.725 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:18.722+0000 7f7a5f40a700 1 -- 172.21.15.137:0/4089970454 >> 172.21.15.137:0/4089970454 conn(0x7f7a580f0b70 msgr2=0x7f7a580f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:53:18.725 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:18.722+0000 7f7a5f40a700 1 -- 172.21.15.137:0/4089970454 shutdown_connections 2023-11-15T01:53:18.725 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:18.722+0000 7f7a5f40a700 1 -- 172.21.15.137:0/4089970454 wait complete. 2023-11-15T01:53:18.725 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:18.723+0000 7f7a5f40a700 1 Processor -- start 2023-11-15T01:53:18.725 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:18.723+0000 7f7a5f40a700 1 -- start start 2023-11-15T01:53:18.726 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:18.723+0000 7f7a5f40a700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7a580f5410 0x7f7a581093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:53:18.726 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:18.723+0000 7f7a5f40a700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a58060280 con 0x7f7a580f5410 2023-11-15T01:53:18.726 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:18.723+0000 7f7a5d1a6700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7a580f5410 0x7f7a581093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:53:18.726 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:18.723+0000 7f7a5d1a6700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7a580f5410 0x7f7a581093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:55304/0 (socket says 172.21.15.137:55304) 2023-11-15T01:53:18.726 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:18.723+0000 7f7a5d1a6700 1 -- 172.21.15.137:0/1375280819 learned_addr learned my addr 172.21.15.137:0/1375280819 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:53:18.728 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:18.724+0000 7f7a5d1a6700 1 -- 172.21.15.137:0/1375280819 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a54005040 con 0x7f7a580f5410 2023-11-15T01:53:18.728 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:18.724+0000 7f7a5d1a6700 1 --2- 172.21.15.137:0/1375280819 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7a580f5410 0x7f7a581093f0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f7a54013510 tx=0x7f7a5400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:53:18.728 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:18.724+0000 7f7a4dffb700 1 -- 172.21.15.137:0/1375280819 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a54009320 con 0x7f7a580f5410 2023-11-15T01:53:18.728 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:18.724+0000 7f7a5f40a700 1 -- 172.21.15.137:0/1375280819 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a58060480 con 0x7f7a580f5410 2023-11-15T01:53:18.729 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:18.724+0000 7f7a4dffb700 1 -- 172.21.15.137:0/1375280819 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7a54013f00 con 0x7f7a580f5410 2023-11-15T01:53:18.729 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:18.724+0000 7f7a4dffb700 1 -- 172.21.15.137:0/1375280819 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a54014d90 con 0x7f7a580f5410 2023-11-15T01:53:18.729 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:18.724+0000 7f7a5f40a700 1 -- 172.21.15.137:0/1375280819 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a58109b20 con 0x7f7a580f5410 2023-11-15T01:53:18.729 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:18.725+0000 7f7a4dffb700 1 -- 172.21.15.137:0/1375280819 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7a540095f0 con 0x7f7a580f5410 2023-11-15T01:53:18.729 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:18.725+0000 7f7a5f40a700 1 -- 172.21.15.137:0/1375280819 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a58045ad0 con 0x7f7a580f5410 2023-11-15T01:53:18.730 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:18.725+0000 7f7a4dffb700 1 --2- 172.21.15.137:0/1375280819 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f7a4402ff60 0x7f7a44032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:53:18.730 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:18.725+0000 7f7a4dffb700 1 -- 172.21.15.137:0/1375280819 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7a54040fc0 con 0x7f7a580f5410 2023-11-15T01:53:18.730 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:18.725+0000 7f7a5c9a5700 1 --2- 172.21.15.137:0/1375280819 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f7a4402ff60 0x7f7a44032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:53:18.730 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:18.726+0000 7f7a5c9a5700 1 --2- 172.21.15.137:0/1375280819 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f7a4402ff60 0x7f7a44032420 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f7a480096a0 tx=0x7f7a48006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:53:18.731 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:18.730+0000 7f7a4dffb700 1 -- 172.21.15.137:0/1375280819 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7a54014420 con 0x7f7a580f5410 2023-11-15T01:53:18.935 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:18.934+0000 7f7a5f40a700 1 -- 172.21.15.137:0/1375280819 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7a5810a370 con 0x7f7a580f5410 2023-11-15T01:53:18.936 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:18.935+0000 7f7a4dffb700 1 -- 172.21.15.137:0/1375280819 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7a54010230 con 0x7f7a580f5410 2023-11-15T01:53:18.937 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:53:18.937 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:53:18.939 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:18.936+0000 7f7a5f40a700 1 -- 172.21.15.137:0/1375280819 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f7a4402ff60 msgr2=0x7f7a44032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:53:18.939 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:18.936+0000 7f7a5f40a700 1 --2- 172.21.15.137:0/1375280819 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f7a4402ff60 0x7f7a44032420 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f7a480096a0 tx=0x7f7a48006b40).stop 2023-11-15T01:53:18.939 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:18.936+0000 7f7a5f40a700 1 -- 172.21.15.137:0/1375280819 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7a580f5410 msgr2=0x7f7a581093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:53:18.940 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:18.936+0000 7f7a5f40a700 1 --2- 172.21.15.137:0/1375280819 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7a580f5410 0x7f7a581093f0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f7a54013510 tx=0x7f7a5400b040).stop 2023-11-15T01:53:18.940 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:18.937+0000 7f7a5f40a700 1 -- 172.21.15.137:0/1375280819 shutdown_connections 2023-11-15T01:53:18.940 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:18.937+0000 7f7a5f40a700 1 --2- 172.21.15.137:0/1375280819 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f7a4402ff60 0x7f7a44032420 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:53:18.940 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:18.937+0000 7f7a5f40a700 1 --2- 172.21.15.137:0/1375280819 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7a580f5410 0x7f7a581093f0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:53:18.940 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:18.937+0000 7f7a5f40a700 1 -- 172.21.15.137:0/1375280819 >> 172.21.15.137:0/1375280819 conn(0x7f7a580f0b70 msgr2=0x7f7a580f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:53:18.941 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:18.937+0000 7f7a5f40a700 1 -- 172.21.15.137:0/1375280819 shutdown_connections 2023-11-15T01:53:18.941 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:18.937+0000 7f7a5f40a700 1 -- 172.21.15.137:0/1375280819 wait complete. 2023-11-15T01:53:18.941 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:53:20.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:53:19 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:53:18.724623+0000 mgr.smithi055.szwngw (mgr.14164) 109 : cluster [DBG] pgmap v102: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:53:20.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:53:19 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:53:18.936063+0000 mon.smithi055 (mon.0) 189 : audit [DBG] from='client.? 172.21.15.137:0/1375280819' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:53:20.327 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:53:20.328 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:53:21.453 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:21.451+0000 7f24339ca700 1 -- 172.21.15.137:0/2001474444 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f242c060220 msgr2=0x7f242c0f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:53:21.453 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:21.451+0000 7f24339ca700 1 --2- 172.21.15.137:0/2001474444 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f242c060220 0x7f242c0f9150 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f2428009fb0 tx=0x7f2428009b30).stop 2023-11-15T01:53:21.453 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:21.451+0000 7f24339ca700 1 -- 172.21.15.137:0/2001474444 shutdown_connections 2023-11-15T01:53:21.454 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:21.451+0000 7f24339ca700 1 --2- 172.21.15.137:0/2001474444 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f242c060220 0x7f242c0f9150 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:53:21.454 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:21.451+0000 7f24339ca700 1 -- 172.21.15.137:0/2001474444 >> 172.21.15.137:0/2001474444 conn(0x7f242c0f0b70 msgr2=0x7f242c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:53:21.454 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:21.451+0000 7f24339ca700 1 -- 172.21.15.137:0/2001474444 shutdown_connections 2023-11-15T01:53:21.454 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:21.451+0000 7f24339ca700 1 -- 172.21.15.137:0/2001474444 wait complete. 2023-11-15T01:53:21.455 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:21.452+0000 7f24339ca700 1 Processor -- start 2023-11-15T01:53:21.455 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:21.452+0000 7f24339ca700 1 -- start start 2023-11-15T01:53:21.455 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:21.452+0000 7f24339ca700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f242c060220 0x7f242c1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:53:21.455 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:21.452+0000 7f24339ca700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f242c106bd0 con 0x7f242c060220 2023-11-15T01:53:21.456 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:21.452+0000 7f2431766700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f242c060220 0x7f242c1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:53:21.456 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:21.452+0000 7f2431766700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f242c060220 0x7f242c1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:55318/0 (socket says 172.21.15.137:55318) 2023-11-15T01:53:21.456 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:21.452+0000 7f2431766700 1 -- 172.21.15.137:0/3613432612 learned_addr learned my addr 172.21.15.137:0/3613432612 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:53:21.459 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:21.452+0000 7f2431766700 1 -- 172.21.15.137:0/3613432612 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2428005040 con 0x7f242c060220 2023-11-15T01:53:21.459 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:21.453+0000 7f2431766700 1 --2- 172.21.15.137:0/3613432612 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f242c060220 0x7f242c1066e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f2428009830 tx=0x7f242800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:53:21.459 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:21.453+0000 7f2421ffb700 1 -- 172.21.15.137:0/3613432612 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2428009320 con 0x7f242c060220 2023-11-15T01:53:21.459 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:21.453+0000 7f24339ca700 1 -- 172.21.15.137:0/3613432612 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f242c106dd0 con 0x7f242c060220 2023-11-15T01:53:21.459 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:21.453+0000 7f2421ffb700 1 -- 172.21.15.137:0/3613432612 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f24280136c0 con 0x7f242c060220 2023-11-15T01:53:21.460 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:21.453+0000 7f2421ffb700 1 -- 172.21.15.137:0/3613432612 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f242801ee90 con 0x7f242c060220 2023-11-15T01:53:21.460 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:21.453+0000 7f24339ca700 1 -- 172.21.15.137:0/3613432612 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f242c107260 con 0x7f242c060220 2023-11-15T01:53:21.460 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:21.454+0000 7f2421ffb700 1 -- 172.21.15.137:0/3613432612 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2428014420 con 0x7f242c060220 2023-11-15T01:53:21.460 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:21.454+0000 7f2421ffb700 1 --2- 172.21.15.137:0/3613432612 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f241802ff10 0x7f24180323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:53:21.460 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:21.454+0000 7f24339ca700 1 -- 172.21.15.137:0/3613432612 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f242c045ad0 con 0x7f242c060220 2023-11-15T01:53:21.461 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:21.454+0000 7f2421ffb700 1 -- 172.21.15.137:0/3613432612 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2428041330 con 0x7f242c060220 2023-11-15T01:53:21.461 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:21.454+0000 7f2430f65700 1 --2- 172.21.15.137:0/3613432612 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f241802ff10 0x7f24180323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:53:21.461 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:21.455+0000 7f2430f65700 1 --2- 172.21.15.137:0/3613432612 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f241802ff10 0x7f24180323d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f241c0096a0 tx=0x7f241c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:53:21.462 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:21.459+0000 7f2421ffb700 1 -- 172.21.15.137:0/3613432612 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f242801e420 con 0x7f242c060220 2023-11-15T01:53:21.666 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:21.666+0000 7f24339ca700 1 -- 172.21.15.137:0/3613432612 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f242c108110 con 0x7f242c060220 2023-11-15T01:53:21.667 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:21.667+0000 7f2421ffb700 1 -- 172.21.15.137:0/3613432612 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f242802c2d0 con 0x7f242c060220 2023-11-15T01:53:21.667 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:53:21.668 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:53:21.669 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:21.668+0000 7f24339ca700 1 -- 172.21.15.137:0/3613432612 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f241802ff10 msgr2=0x7f24180323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:53:21.670 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:21.668+0000 7f24339ca700 1 --2- 172.21.15.137:0/3613432612 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f241802ff10 0x7f24180323d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f241c0096a0 tx=0x7f241c006b40).stop 2023-11-15T01:53:21.670 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:21.668+0000 7f24339ca700 1 -- 172.21.15.137:0/3613432612 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f242c060220 msgr2=0x7f242c1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:53:21.670 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:21.668+0000 7f24339ca700 1 --2- 172.21.15.137:0/3613432612 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f242c060220 0x7f242c1066e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f2428009830 tx=0x7f242800b040).stop 2023-11-15T01:53:21.670 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:21.668+0000 7f24339ca700 1 -- 172.21.15.137:0/3613432612 shutdown_connections 2023-11-15T01:53:21.670 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:21.669+0000 7f24339ca700 1 --2- 172.21.15.137:0/3613432612 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f241802ff10 0x7f24180323d0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:53:21.671 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:21.669+0000 7f24339ca700 1 --2- 172.21.15.137:0/3613432612 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f242c060220 0x7f242c1066e0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:53:21.671 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:21.669+0000 7f24339ca700 1 -- 172.21.15.137:0/3613432612 >> 172.21.15.137:0/3613432612 conn(0x7f242c0f0b70 msgr2=0x7f242c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:53:21.671 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:21.669+0000 7f24339ca700 1 -- 172.21.15.137:0/3613432612 shutdown_connections 2023-11-15T01:53:21.671 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:21.669+0000 7f24339ca700 1 -- 172.21.15.137:0/3613432612 wait complete. 2023-11-15T01:53:21.672 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:53:21.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:53:21 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:53:20.724962+0000 mgr.smithi055.szwngw (mgr.14164) 110 : cluster [DBG] pgmap v103: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:53:22.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:53:22 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:53:21.667833+0000 mon.smithi055 (mon.0) 190 : audit [DBG] from='client.? 172.21.15.137:0/3613432612' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:53:23.070 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:53:23.071 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:53:23.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:53:23 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:53:22.725286+0000 mgr.smithi055.szwngw (mgr.14164) 111 : cluster [DBG] pgmap v104: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:53:24.207 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:24.205+0000 7f430987d700 1 -- 172.21.15.137:0/3869869061 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f43040f6510 msgr2=0x7f43040f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:53:24.207 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:24.205+0000 7f430987d700 1 --2- 172.21.15.137:0/3869869061 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f43040f6510 0x7f43040f6930 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7f42f8009fb0 tx=0x7f42f8009b30).stop 2023-11-15T01:53:24.208 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:24.205+0000 7f430987d700 1 -- 172.21.15.137:0/3869869061 shutdown_connections 2023-11-15T01:53:24.208 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:24.205+0000 7f430987d700 1 --2- 172.21.15.137:0/3869869061 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f43040f6510 0x7f43040f6930 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:53:24.208 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:24.205+0000 7f430987d700 1 -- 172.21.15.137:0/3869869061 >> 172.21.15.137:0/3869869061 conn(0x7f43040f1c70 msgr2=0x7f43040f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:53:24.209 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:24.205+0000 7f430987d700 1 -- 172.21.15.137:0/3869869061 shutdown_connections 2023-11-15T01:53:24.209 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:24.205+0000 7f430987d700 1 -- 172.21.15.137:0/3869869061 wait complete. 2023-11-15T01:53:24.209 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:24.206+0000 7f430987d700 1 Processor -- start 2023-11-15T01:53:24.209 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:24.206+0000 7f430987d700 1 -- start start 2023-11-15T01:53:24.210 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:24.206+0000 7f430987d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f43040f6510 0x7f4304106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:53:24.210 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:24.206+0000 7f430987d700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4304106c20 con 0x7f43040f6510 2023-11-15T01:53:24.213 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:24.206+0000 7f4302ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f43040f6510 0x7f4304106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:53:24.214 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:24.206+0000 7f4302ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f43040f6510 0x7f4304106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:55342/0 (socket says 172.21.15.137:55342) 2023-11-15T01:53:24.214 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:24.206+0000 7f4302ffd700 1 -- 172.21.15.137:0/2958882754 learned_addr learned my addr 172.21.15.137:0/2958882754 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:53:24.214 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:24.206+0000 7f4302ffd700 1 -- 172.21.15.137:0/2958882754 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42f8005040 con 0x7f43040f6510 2023-11-15T01:53:24.215 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:24.207+0000 7f4302ffd700 1 --2- 172.21.15.137:0/2958882754 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f43040f6510 0x7f4304106730 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f42f8013510 tx=0x7f42f800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:53:24.215 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:24.207+0000 7f42eb7fe700 1 -- 172.21.15.137:0/2958882754 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f42f8009320 con 0x7f43040f6510 2023-11-15T01:53:24.215 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:24.207+0000 7f430987d700 1 -- 172.21.15.137:0/2958882754 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4304106e20 con 0x7f43040f6510 2023-11-15T01:53:24.216 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:24.207+0000 7f42eb7fe700 1 -- 172.21.15.137:0/2958882754 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f42f8013f00 con 0x7f43040f6510 2023-11-15T01:53:24.216 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:24.207+0000 7f42eb7fe700 1 -- 172.21.15.137:0/2958882754 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f42f8014d90 con 0x7f43040f6510 2023-11-15T01:53:24.216 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:24.207+0000 7f430987d700 1 -- 172.21.15.137:0/2958882754 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43041072b0 con 0x7f43040f6510 2023-11-15T01:53:24.217 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:24.208+0000 7f42eb7fe700 1 -- 172.21.15.137:0/2958882754 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f42f80095f0 con 0x7f43040f6510 2023-11-15T01:53:24.217 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:24.208+0000 7f430987d700 1 -- 172.21.15.137:0/2958882754 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4304045ad0 con 0x7f43040f6510 2023-11-15T01:53:24.217 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:24.208+0000 7f42eb7fe700 1 --2- 172.21.15.137:0/2958882754 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f42ec02ff60 0x7f42ec032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:53:24.218 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:24.208+0000 7f42eb7fe700 1 -- 172.21.15.137:0/2958882754 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f42f8041000 con 0x7f43040f6510 2023-11-15T01:53:24.218 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:24.208+0000 7f43027fc700 1 --2- 172.21.15.137:0/2958882754 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f42ec02ff60 0x7f42ec032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:53:24.218 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:24.209+0000 7f43027fc700 1 --2- 172.21.15.137:0/2958882754 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f42ec02ff60 0x7f42ec032420 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f42f40096a0 tx=0x7f42f4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:53:24.219 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:24.213+0000 7f42eb7fe700 1 -- 172.21.15.137:0/2958882754 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f42f8014420 con 0x7f43040f6510 2023-11-15T01:53:24.421 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:24.420+0000 7f430987d700 1 -- 172.21.15.137:0/2958882754 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f43040fbf60 con 0x7f43040f6510 2023-11-15T01:53:24.423 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:24.420+0000 7f42eb7fe700 1 -- 172.21.15.137:0/2958882754 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f42f801d7f0 con 0x7f43040f6510 2023-11-15T01:53:24.425 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:53:24.425 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:53:24.428 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:24.422+0000 7f430987d700 1 -- 172.21.15.137:0/2958882754 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f42ec02ff60 msgr2=0x7f42ec032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:53:24.429 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:24.422+0000 7f430987d700 1 --2- 172.21.15.137:0/2958882754 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f42ec02ff60 0x7f42ec032420 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f42f40096a0 tx=0x7f42f4006b40).stop 2023-11-15T01:53:24.429 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:24.422+0000 7f430987d700 1 -- 172.21.15.137:0/2958882754 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f43040f6510 msgr2=0x7f4304106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:53:24.430 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:24.422+0000 7f430987d700 1 --2- 172.21.15.137:0/2958882754 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f43040f6510 0x7f4304106730 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f42f8013510 tx=0x7f42f800b040).stop 2023-11-15T01:53:24.430 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:24.422+0000 7f430987d700 1 -- 172.21.15.137:0/2958882754 shutdown_connections 2023-11-15T01:53:24.433 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:24.422+0000 7f430987d700 1 --2- 172.21.15.137:0/2958882754 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f42ec02ff60 0x7f42ec032420 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:53:24.433 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:24.422+0000 7f430987d700 1 --2- 172.21.15.137:0/2958882754 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f43040f6510 0x7f4304106730 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:53:24.433 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:24.422+0000 7f430987d700 1 -- 172.21.15.137:0/2958882754 >> 172.21.15.137:0/2958882754 conn(0x7f43040f1c70 msgr2=0x7f43040f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:53:24.433 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:24.422+0000 7f430987d700 1 -- 172.21.15.137:0/2958882754 shutdown_connections 2023-11-15T01:53:24.433 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:24.422+0000 7f430987d700 1 -- 172.21.15.137:0/2958882754 wait complete. 2023-11-15T01:53:24.434 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:53:24.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:53:24 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:53:24.421430+0000 mon.smithi055 (mon.0) 191 : audit [DBG] from='client.? 172.21.15.137:0/2958882754' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:53:25.813 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:53:25.814 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:53:25.827 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:53:25 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:53:24.725591+0000 mgr.smithi055.szwngw (mgr.14164) 112 : cluster [DBG] pgmap v105: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:53:26.869 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:26.868+0000 7fbe5a16e700 1 -- 172.21.15.137:0/1131175262 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbe540f6510 msgr2=0x7fbe540f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:53:26.870 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:26.868+0000 7fbe5a16e700 1 --2- 172.21.15.137:0/1131175262 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbe540f6510 0x7fbe540f6930 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7fbe48009fb0 tx=0x7fbe48009b30).stop 2023-11-15T01:53:26.870 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:26.868+0000 7fbe5a16e700 1 -- 172.21.15.137:0/1131175262 shutdown_connections 2023-11-15T01:53:26.870 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:26.868+0000 7fbe5a16e700 1 --2- 172.21.15.137:0/1131175262 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbe540f6510 0x7fbe540f6930 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:53:26.870 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:26.868+0000 7fbe5a16e700 1 -- 172.21.15.137:0/1131175262 >> 172.21.15.137:0/1131175262 conn(0x7fbe540f1c70 msgr2=0x7fbe540f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:53:26.871 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:26.868+0000 7fbe5a16e700 1 -- 172.21.15.137:0/1131175262 shutdown_connections 2023-11-15T01:53:26.871 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:26.868+0000 7fbe5a16e700 1 -- 172.21.15.137:0/1131175262 wait complete. 2023-11-15T01:53:26.871 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:26.868+0000 7fbe5a16e700 1 Processor -- start 2023-11-15T01:53:26.871 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:26.869+0000 7fbe5a16e700 1 -- start start 2023-11-15T01:53:26.871 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:26.869+0000 7fbe5a16e700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbe540f6510 0x7fbe54106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:53:26.872 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:26.869+0000 7fbe5a16e700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe54106c20 con 0x7fbe540f6510 2023-11-15T01:53:26.873 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:26.869+0000 7fbe537fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbe540f6510 0x7fbe54106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:53:26.874 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:26.869+0000 7fbe537fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbe540f6510 0x7fbe54106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:55364/0 (socket says 172.21.15.137:55364) 2023-11-15T01:53:26.874 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:26.869+0000 7fbe537fe700 1 -- 172.21.15.137:0/3177614241 learned_addr learned my addr 172.21.15.137:0/3177614241 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:53:26.874 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:26.869+0000 7fbe537fe700 1 -- 172.21.15.137:0/3177614241 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe48005040 con 0x7fbe540f6510 2023-11-15T01:53:26.874 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:26.870+0000 7fbe537fe700 1 --2- 172.21.15.137:0/3177614241 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbe540f6510 0x7fbe54106730 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7fbe48013510 tx=0x7fbe4800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:53:26.875 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:26.870+0000 7fbe3bfff700 1 -- 172.21.15.137:0/3177614241 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe48009320 con 0x7fbe540f6510 2023-11-15T01:53:26.875 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:26.870+0000 7fbe3bfff700 1 -- 172.21.15.137:0/3177614241 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbe48013f00 con 0x7fbe540f6510 2023-11-15T01:53:26.875 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:26.870+0000 7fbe5a16e700 1 -- 172.21.15.137:0/3177614241 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe54106e20 con 0x7fbe540f6510 2023-11-15T01:53:26.875 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:26.870+0000 7fbe3bfff700 1 -- 172.21.15.137:0/3177614241 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe48014d90 con 0x7fbe540f6510 2023-11-15T01:53:26.875 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:26.870+0000 7fbe5a16e700 1 -- 172.21.15.137:0/3177614241 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe541072b0 con 0x7fbe540f6510 2023-11-15T01:53:26.876 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:26.871+0000 7fbe3bfff700 1 -- 172.21.15.137:0/3177614241 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbe480095f0 con 0x7fbe540f6510 2023-11-15T01:53:26.876 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:26.871+0000 7fbe5a16e700 1 -- 172.21.15.137:0/3177614241 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe54045ad0 con 0x7fbe540f6510 2023-11-15T01:53:26.876 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:26.871+0000 7fbe3bfff700 1 --2- 172.21.15.137:0/3177614241 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fbe3c02ff60 0x7fbe3c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:53:26.876 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:26.871+0000 7fbe3bfff700 1 -- 172.21.15.137:0/3177614241 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbe48040fe0 con 0x7fbe540f6510 2023-11-15T01:53:26.877 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:26.871+0000 7fbe52ffd700 1 --2- 172.21.15.137:0/3177614241 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fbe3c02ff60 0x7fbe3c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:53:26.877 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:26.872+0000 7fbe52ffd700 1 --2- 172.21.15.137:0/3177614241 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fbe3c02ff60 0x7fbe3c032420 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7fbe440096a0 tx=0x7fbe44006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:53:26.877 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:26.876+0000 7fbe3bfff700 1 -- 172.21.15.137:0/3177614241 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbe4801d7f0 con 0x7fbe540f6510 2023-11-15T01:53:27.081 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:27.080+0000 7fbe5a16e700 1 -- 172.21.15.137:0/3177614241 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbe540fbf60 con 0x7fbe540f6510 2023-11-15T01:53:27.082 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:27.081+0000 7fbe3bfff700 1 -- 172.21.15.137:0/3177614241 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbe48010230 con 0x7fbe540f6510 2023-11-15T01:53:27.082 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:53:27.083 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:53:27.084 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:27.083+0000 7fbe5a16e700 1 -- 172.21.15.137:0/3177614241 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fbe3c02ff60 msgr2=0x7fbe3c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:53:27.084 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:27.083+0000 7fbe5a16e700 1 --2- 172.21.15.137:0/3177614241 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fbe3c02ff60 0x7fbe3c032420 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7fbe440096a0 tx=0x7fbe44006b40).stop 2023-11-15T01:53:27.084 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:27.083+0000 7fbe5a16e700 1 -- 172.21.15.137:0/3177614241 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbe540f6510 msgr2=0x7fbe54106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:53:27.085 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:27.083+0000 7fbe5a16e700 1 --2- 172.21.15.137:0/3177614241 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbe540f6510 0x7fbe54106730 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7fbe48013510 tx=0x7fbe4800b040).stop 2023-11-15T01:53:27.085 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:27.083+0000 7fbe5a16e700 1 -- 172.21.15.137:0/3177614241 shutdown_connections 2023-11-15T01:53:27.085 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:27.083+0000 7fbe5a16e700 1 --2- 172.21.15.137:0/3177614241 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fbe3c02ff60 0x7fbe3c032420 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:53:27.086 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:27.083+0000 7fbe5a16e700 1 --2- 172.21.15.137:0/3177614241 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbe540f6510 0x7fbe54106730 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:53:27.086 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:27.083+0000 7fbe5a16e700 1 -- 172.21.15.137:0/3177614241 >> 172.21.15.137:0/3177614241 conn(0x7fbe540f1c70 msgr2=0x7fbe540f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:53:27.086 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:27.083+0000 7fbe5a16e700 1 -- 172.21.15.137:0/3177614241 shutdown_connections 2023-11-15T01:53:27.086 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:27.083+0000 7fbe5a16e700 1 -- 172.21.15.137:0/3177614241 wait complete. 2023-11-15T01:53:27.087 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:53:28.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:53:27 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:53:26.725857+0000 mgr.smithi055.szwngw (mgr.14164) 113 : cluster [DBG] pgmap v106: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:53:28.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:53:27 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:53:27.081982+0000 mon.smithi055 (mon.0) 192 : audit [DBG] from='client.? 172.21.15.137:0/3177614241' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:53:28.496 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:53:28.496 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:53:29.654 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:29.653+0000 7fc3bd697700 1 -- 172.21.15.137:0/4165117216 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc3b80f5410 msgr2=0x7fc3b80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:53:29.654 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:29.653+0000 7fc3bd697700 1 --2- 172.21.15.137:0/4165117216 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc3b80f5410 0x7fc3b80f5830 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7fc3ac009fb0 tx=0x7fc3ac009b30).stop 2023-11-15T01:53:29.655 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:29.653+0000 7fc3bd697700 1 -- 172.21.15.137:0/4165117216 shutdown_connections 2023-11-15T01:53:29.655 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:29.653+0000 7fc3bd697700 1 --2- 172.21.15.137:0/4165117216 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc3b80f5410 0x7fc3b80f5830 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:53:29.655 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:29.653+0000 7fc3bd697700 1 -- 172.21.15.137:0/4165117216 >> 172.21.15.137:0/4165117216 conn(0x7fc3b80f0b70 msgr2=0x7fc3b80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:53:29.655 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:29.653+0000 7fc3bd697700 1 -- 172.21.15.137:0/4165117216 shutdown_connections 2023-11-15T01:53:29.656 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:29.653+0000 7fc3bd697700 1 -- 172.21.15.137:0/4165117216 wait complete. 2023-11-15T01:53:29.658 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:29.654+0000 7fc3bd697700 1 Processor -- start 2023-11-15T01:53:29.658 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:29.654+0000 7fc3bd697700 1 -- start start 2023-11-15T01:53:29.658 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:29.654+0000 7fc3bd697700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc3b80f5410 0x7fc3b8106760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:53:29.658 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:29.654+0000 7fc3bd697700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b8106c50 con 0x7fc3b80f5410 2023-11-15T01:53:29.658 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:29.654+0000 7fc3b6ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc3b80f5410 0x7fc3b8106760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:53:29.659 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:29.654+0000 7fc3b6ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc3b80f5410 0x7fc3b8106760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:48378/0 (socket says 172.21.15.137:48378) 2023-11-15T01:53:29.659 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:29.654+0000 7fc3b6ffd700 1 -- 172.21.15.137:0/145778920 learned_addr learned my addr 172.21.15.137:0/145778920 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:53:29.659 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:29.655+0000 7fc3b6ffd700 1 -- 172.21.15.137:0/145778920 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3ac005040 con 0x7fc3b80f5410 2023-11-15T01:53:29.659 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:29.655+0000 7fc3b6ffd700 1 --2- 172.21.15.137:0/145778920 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc3b80f5410 0x7fc3b8106760 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7fc3ac009f80 tx=0x7fc3ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:53:29.659 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:29.655+0000 7fc39f7fe700 1 -- 172.21.15.137:0/145778920 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc3ac009320 con 0x7fc3b80f5410 2023-11-15T01:53:29.660 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:29.655+0000 7fc39f7fe700 1 -- 172.21.15.137:0/145778920 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc3ac013ec0 con 0x7fc3b80f5410 2023-11-15T01:53:29.660 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:29.655+0000 7fc3bd697700 1 -- 172.21.15.137:0/145778920 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3b8106e50 con 0x7fc3b80f5410 2023-11-15T01:53:29.660 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:29.655+0000 7fc39f7fe700 1 -- 172.21.15.137:0/145778920 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc3ac01ece0 con 0x7fc3b80f5410 2023-11-15T01:53:29.660 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:29.655+0000 7fc3bd697700 1 -- 172.21.15.137:0/145778920 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3b81072e0 con 0x7fc3b80f5410 2023-11-15T01:53:29.660 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:29.656+0000 7fc39f7fe700 1 -- 172.21.15.137:0/145778920 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc3ac01e420 con 0x7fc3b80f5410 2023-11-15T01:53:29.661 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:29.656+0000 7fc3bd697700 1 -- 172.21.15.137:0/145778920 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3b8045ad0 con 0x7fc3b80f5410 2023-11-15T01:53:29.661 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:29.656+0000 7fc39f7fe700 1 --2- 172.21.15.137:0/145778920 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fc3a002ff60 0x7fc3a0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:53:29.661 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:29.656+0000 7fc39f7fe700 1 -- 172.21.15.137:0/145778920 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc3ac040e20 con 0x7fc3b80f5410 2023-11-15T01:53:29.661 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:29.656+0000 7fc3b67fc700 1 --2- 172.21.15.137:0/145778920 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fc3a002ff60 0x7fc3a0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:53:29.661 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:29.657+0000 7fc3b67fc700 1 --2- 172.21.15.137:0/145778920 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fc3a002ff60 0x7fc3a0032420 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7fc3a80096a0 tx=0x7fc3a8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:53:29.662 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:29.661+0000 7fc39f7fe700 1 -- 172.21.15.137:0/145778920 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc3ac01e6f0 con 0x7fc3b80f5410 2023-11-15T01:53:29.869 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:29.868+0000 7fc3bd697700 1 -- 172.21.15.137:0/145778920 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc3b8108190 con 0x7fc3b80f5410 2023-11-15T01:53:29.870 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:29.869+0000 7fc39f7fe700 1 -- 172.21.15.137:0/145778920 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc3ac02c2d0 con 0x7fc3b80f5410 2023-11-15T01:53:29.870 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:53:29.871 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:53:29.871 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:29.871+0000 7fc3bd697700 1 -- 172.21.15.137:0/145778920 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fc3a002ff60 msgr2=0x7fc3a0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:53:29.872 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:29.871+0000 7fc3bd697700 1 --2- 172.21.15.137:0/145778920 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fc3a002ff60 0x7fc3a0032420 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7fc3a80096a0 tx=0x7fc3a8006b40).stop 2023-11-15T01:53:29.872 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:29.871+0000 7fc3bd697700 1 -- 172.21.15.137:0/145778920 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc3b80f5410 msgr2=0x7fc3b8106760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:53:29.872 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:29.871+0000 7fc3bd697700 1 --2- 172.21.15.137:0/145778920 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc3b80f5410 0x7fc3b8106760 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7fc3ac009f80 tx=0x7fc3ac00b040).stop 2023-11-15T01:53:29.872 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:29.871+0000 7fc3bd697700 1 -- 172.21.15.137:0/145778920 shutdown_connections 2023-11-15T01:53:29.873 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:29.871+0000 7fc3bd697700 1 --2- 172.21.15.137:0/145778920 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fc3a002ff60 0x7fc3a0032420 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:53:29.873 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:29.871+0000 7fc3bd697700 1 --2- 172.21.15.137:0/145778920 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc3b80f5410 0x7fc3b8106760 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:53:29.873 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:29.871+0000 7fc3bd697700 1 -- 172.21.15.137:0/145778920 >> 172.21.15.137:0/145778920 conn(0x7fc3b80f0b70 msgr2=0x7fc3b805fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:53:29.873 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:29.871+0000 7fc3bd697700 1 -- 172.21.15.137:0/145778920 shutdown_connections 2023-11-15T01:53:29.874 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:29.871+0000 7fc3bd697700 1 -- 172.21.15.137:0/145778920 wait complete. 2023-11-15T01:53:29.874 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:53:30.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:53:29 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:53:28.726109+0000 mgr.smithi055.szwngw (mgr.14164) 114 : cluster [DBG] pgmap v107: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:53:31.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:53:30 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:53:29.870119+0000 mon.smithi055 (mon.0) 193 : audit [DBG] from='client.? 172.21.15.137:0/145778920' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:53:31.261 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:53:31.261 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:53:32.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:53:31 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:53:30.726402+0000 mgr.smithi055.szwngw (mgr.14164) 115 : cluster [DBG] pgmap v108: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:53:32.352 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:32.350+0000 7faaa3c47700 1 -- 172.21.15.137:0/517887353 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faa9c0f7b60 msgr2=0x7faa9c0f7f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:53:32.352 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:32.350+0000 7faaa3c47700 1 --2- 172.21.15.137:0/517887353 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faa9c0f7b60 0x7faa9c0f7f80 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7faa8c009fb0 tx=0x7faa8c009b30).stop 2023-11-15T01:53:32.353 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:32.350+0000 7faaa3c47700 1 -- 172.21.15.137:0/517887353 shutdown_connections 2023-11-15T01:53:32.353 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:32.350+0000 7faaa3c47700 1 --2- 172.21.15.137:0/517887353 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faa9c0f7b60 0x7faa9c0f7f80 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:53:32.353 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:32.350+0000 7faaa3c47700 1 -- 172.21.15.137:0/517887353 >> 172.21.15.137:0/517887353 conn(0x7faa9c0f02e0 msgr2=0x7faa9c0f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:53:32.353 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:32.351+0000 7faaa3c47700 1 -- 172.21.15.137:0/517887353 shutdown_connections 2023-11-15T01:53:32.353 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:32.351+0000 7faaa3c47700 1 -- 172.21.15.137:0/517887353 wait complete. 2023-11-15T01:53:32.353 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:32.351+0000 7faaa3c47700 1 Processor -- start 2023-11-15T01:53:32.354 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:32.351+0000 7faaa3c47700 1 -- start start 2023-11-15T01:53:32.354 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:32.351+0000 7faaa3c47700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faa9c0f7b60 0x7faa9c108950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:53:32.354 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:32.351+0000 7faaa3c47700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa9c108e40 con 0x7faa9c0f7b60 2023-11-15T01:53:32.354 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:32.351+0000 7faaa19e3700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faa9c0f7b60 0x7faa9c108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:53:32.354 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:32.352+0000 7faaa19e3700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faa9c0f7b60 0x7faa9c108950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:48406/0 (socket says 172.21.15.137:48406) 2023-11-15T01:53:32.355 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:32.352+0000 7faaa19e3700 1 -- 172.21.15.137:0/1386895186 learned_addr learned my addr 172.21.15.137:0/1386895186 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:53:32.356 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:32.352+0000 7faaa19e3700 1 -- 172.21.15.137:0/1386895186 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa8c005040 con 0x7faa9c0f7b60 2023-11-15T01:53:32.356 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:32.352+0000 7faaa19e3700 1 --2- 172.21.15.137:0/1386895186 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faa9c0f7b60 0x7faa9c108950 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7faa8c013510 tx=0x7faa8c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:53:32.356 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:32.352+0000 7faa927fc700 1 -- 172.21.15.137:0/1386895186 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faa8c009320 con 0x7faa9c0f7b60 2023-11-15T01:53:32.357 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:32.352+0000 7faa927fc700 1 -- 172.21.15.137:0/1386895186 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faa8c013f00 con 0x7faa9c0f7b60 2023-11-15T01:53:32.357 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:32.352+0000 7faaa3c47700 1 -- 172.21.15.137:0/1386895186 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa9c109040 con 0x7faa9c0f7b60 2023-11-15T01:53:32.357 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:32.352+0000 7faa927fc700 1 -- 172.21.15.137:0/1386895186 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faa8c014d90 con 0x7faa9c0f7b60 2023-11-15T01:53:32.357 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:32.352+0000 7faaa3c47700 1 -- 172.21.15.137:0/1386895186 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa9c1094d0 con 0x7faa9c0f7b60 2023-11-15T01:53:32.357 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:32.353+0000 7faa927fc700 1 -- 172.21.15.137:0/1386895186 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7faa8c014420 con 0x7faa9c0f7b60 2023-11-15T01:53:32.358 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:32.353+0000 7faaa3c47700 1 -- 172.21.15.137:0/1386895186 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa9c045ad0 con 0x7faa9c0f7b60 2023-11-15T01:53:32.358 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:32.353+0000 7faa927fc700 1 --2- 172.21.15.137:0/1386895186 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7faa8802ff60 0x7faa88032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:53:32.358 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:32.353+0000 7faa927fc700 1 -- 172.21.15.137:0/1386895186 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7faa8c041000 con 0x7faa9c0f7b60 2023-11-15T01:53:32.358 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:32.353+0000 7faaa11e2700 1 --2- 172.21.15.137:0/1386895186 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7faa8802ff60 0x7faa88032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:53:32.358 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:32.354+0000 7faaa11e2700 1 --2- 172.21.15.137:0/1386895186 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7faa8802ff60 0x7faa88032420 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7faa980096a0 tx=0x7faa98006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:53:32.359 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:32.358+0000 7faa927fc700 1 -- 172.21.15.137:0/1386895186 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7faa8c0095f0 con 0x7faa9c0f7b60 2023-11-15T01:53:32.566 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:32.565+0000 7faaa3c47700 1 -- 172.21.15.137:0/1386895186 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7faa9c0f6ac0 con 0x7faa9c0f7b60 2023-11-15T01:53:32.566 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:32.566+0000 7faa927fc700 1 -- 172.21.15.137:0/1386895186 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7faa8c01db70 con 0x7faa9c0f7b60 2023-11-15T01:53:32.567 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:53:32.567 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:53:32.568 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:32.568+0000 7faaa3c47700 1 -- 172.21.15.137:0/1386895186 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7faa8802ff60 msgr2=0x7faa88032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:53:32.569 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:32.568+0000 7faaa3c47700 1 --2- 172.21.15.137:0/1386895186 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7faa8802ff60 0x7faa88032420 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7faa980096a0 tx=0x7faa98006b40).stop 2023-11-15T01:53:32.569 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:32.568+0000 7faaa3c47700 1 -- 172.21.15.137:0/1386895186 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faa9c0f7b60 msgr2=0x7faa9c108950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:53:32.569 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:32.568+0000 7faaa3c47700 1 --2- 172.21.15.137:0/1386895186 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faa9c0f7b60 0x7faa9c108950 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7faa8c013510 tx=0x7faa8c00b040).stop 2023-11-15T01:53:32.569 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:32.568+0000 7faaa3c47700 1 -- 172.21.15.137:0/1386895186 shutdown_connections 2023-11-15T01:53:32.569 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:32.568+0000 7faaa3c47700 1 --2- 172.21.15.137:0/1386895186 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7faa8802ff60 0x7faa88032420 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:53:32.570 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:32.568+0000 7faaa3c47700 1 --2- 172.21.15.137:0/1386895186 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faa9c0f7b60 0x7faa9c108950 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:53:32.570 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:32.568+0000 7faaa3c47700 1 -- 172.21.15.137:0/1386895186 >> 172.21.15.137:0/1386895186 conn(0x7faa9c0f02e0 msgr2=0x7faa9c0f1010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:53:32.570 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:32.568+0000 7faaa3c47700 1 -- 172.21.15.137:0/1386895186 shutdown_connections 2023-11-15T01:53:32.570 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:32.568+0000 7faaa3c47700 1 -- 172.21.15.137:0/1386895186 wait complete. 2023-11-15T01:53:32.571 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:53:33.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:53:32 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:53:32.566918+0000 mon.smithi055 (mon.0) 194 : audit [DBG] from='client.? 172.21.15.137:0/1386895186' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:53:33.961 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:53:33.962 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:53:34.077 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:53:33 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:53:32.726721+0000 mgr.smithi055.szwngw (mgr.14164) 116 : cluster [DBG] pgmap v109: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:53:35.076 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:35.074+0000 7f43f467f700 1 -- 172.21.15.137:0/809958480 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f43ec0f5410 msgr2=0x7f43ec0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:53:35.076 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:35.074+0000 7f43f467f700 1 --2- 172.21.15.137:0/809958480 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f43ec0f5410 0x7f43ec0f5830 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7f43dc009fb0 tx=0x7f43dc009b30).stop 2023-11-15T01:53:35.077 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:35.074+0000 7f43f467f700 1 -- 172.21.15.137:0/809958480 shutdown_connections 2023-11-15T01:53:35.077 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:35.074+0000 7f43f467f700 1 --2- 172.21.15.137:0/809958480 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f43ec0f5410 0x7f43ec0f5830 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:53:35.077 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:35.074+0000 7f43f467f700 1 -- 172.21.15.137:0/809958480 >> 172.21.15.137:0/809958480 conn(0x7f43ec0f0b70 msgr2=0x7f43ec0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:53:35.077 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:35.074+0000 7f43f467f700 1 -- 172.21.15.137:0/809958480 shutdown_connections 2023-11-15T01:53:35.077 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:35.074+0000 7f43f467f700 1 -- 172.21.15.137:0/809958480 wait complete. 2023-11-15T01:53:35.078 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:35.075+0000 7f43f467f700 1 Processor -- start 2023-11-15T01:53:35.078 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:35.075+0000 7f43f467f700 1 -- start start 2023-11-15T01:53:35.078 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:35.075+0000 7f43f467f700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f43ec0f5410 0x7f43ec104520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:53:35.078 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:35.075+0000 7f43f467f700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43ec104a10 con 0x7f43ec0f5410 2023-11-15T01:53:35.078 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:35.075+0000 7f43f241b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f43ec0f5410 0x7f43ec104520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:53:35.079 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:35.075+0000 7f43f241b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f43ec0f5410 0x7f43ec104520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:48418/0 (socket says 172.21.15.137:48418) 2023-11-15T01:53:35.079 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:35.075+0000 7f43f241b700 1 -- 172.21.15.137:0/1575799313 learned_addr learned my addr 172.21.15.137:0/1575799313 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:53:35.079 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:35.076+0000 7f43f241b700 1 -- 172.21.15.137:0/1575799313 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43dc005040 con 0x7f43ec0f5410 2023-11-15T01:53:35.079 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:35.076+0000 7f43f241b700 1 --2- 172.21.15.137:0/1575799313 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f43ec0f5410 0x7f43ec104520 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f43dc014000 tx=0x7f43dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:53:35.080 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:35.076+0000 7f43e2ffd700 1 -- 172.21.15.137:0/1575799313 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f43dc0095f0 con 0x7f43ec0f5410 2023-11-15T01:53:35.081 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:35.076+0000 7f43e2ffd700 1 -- 172.21.15.137:0/1575799313 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f43dc007e70 con 0x7f43ec0f5410 2023-11-15T01:53:35.081 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:35.076+0000 7f43f467f700 1 -- 172.21.15.137:0/1575799313 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43ec104c10 con 0x7f43ec0f5410 2023-11-15T01:53:35.081 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:35.076+0000 7f43e2ffd700 1 -- 172.21.15.137:0/1575799313 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f43dc014ea0 con 0x7f43ec0f5410 2023-11-15T01:53:35.081 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:35.076+0000 7f43f467f700 1 -- 172.21.15.137:0/1575799313 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43ec1050a0 con 0x7f43ec0f5410 2023-11-15T01:53:35.082 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:35.077+0000 7f43e2ffd700 1 -- 172.21.15.137:0/1575799313 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f43dc025070 con 0x7f43ec0f5410 2023-11-15T01:53:35.082 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:35.077+0000 7f43e2ffd700 1 --2- 172.21.15.137:0/1575799313 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f43d80321b0 0x7f43d8034670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:53:35.082 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:35.077+0000 7f43e2ffd700 1 -- 172.21.15.137:0/1575799313 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f43dc040c00 con 0x7f43ec0f5410 2023-11-15T01:53:35.083 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:35.077+0000 7f43f467f700 1 -- 172.21.15.137:0/1575799313 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43ec045ad0 con 0x7f43ec0f5410 2023-11-15T01:53:35.083 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:35.077+0000 7f43f1c1a700 1 --2- 172.21.15.137:0/1575799313 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f43d80321b0 0x7f43d8034670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:53:35.083 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:35.078+0000 7f43f1c1a700 1 --2- 172.21.15.137:0/1575799313 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f43d80321b0 0x7f43d8034670 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f43e80096a0 tx=0x7f43e8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:53:35.083 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:35.082+0000 7f43e2ffd700 1 -- 172.21.15.137:0/1575799313 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f43dc026350 con 0x7f43ec0f5410 2023-11-15T01:53:35.286 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:35.285+0000 7f43f467f700 1 -- 172.21.15.137:0/1575799313 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f43ec0f9e70 con 0x7f43ec0f5410 2023-11-15T01:53:35.286 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:35.286+0000 7f43e2ffd700 1 -- 172.21.15.137:0/1575799313 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f43dc012330 con 0x7f43ec0f5410 2023-11-15T01:53:35.287 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:53:35.287 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:53:35.288 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:35.287+0000 7f43f467f700 1 -- 172.21.15.137:0/1575799313 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f43d80321b0 msgr2=0x7f43d8034670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:53:35.288 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:35.287+0000 7f43f467f700 1 --2- 172.21.15.137:0/1575799313 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f43d80321b0 0x7f43d8034670 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f43e80096a0 tx=0x7f43e8006b40).stop 2023-11-15T01:53:35.289 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:35.287+0000 7f43f467f700 1 -- 172.21.15.137:0/1575799313 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f43ec0f5410 msgr2=0x7f43ec104520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:53:35.289 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:35.287+0000 7f43f467f700 1 --2- 172.21.15.137:0/1575799313 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f43ec0f5410 0x7f43ec104520 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f43dc014000 tx=0x7f43dc00b040).stop 2023-11-15T01:53:35.289 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:35.287+0000 7f43f467f700 1 -- 172.21.15.137:0/1575799313 shutdown_connections 2023-11-15T01:53:35.289 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:35.287+0000 7f43f467f700 1 --2- 172.21.15.137:0/1575799313 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f43d80321b0 0x7f43d8034670 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:53:35.289 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:35.288+0000 7f43f467f700 1 --2- 172.21.15.137:0/1575799313 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f43ec0f5410 0x7f43ec104520 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:53:35.290 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:35.288+0000 7f43f467f700 1 -- 172.21.15.137:0/1575799313 >> 172.21.15.137:0/1575799313 conn(0x7f43ec0f0b70 msgr2=0x7f43ec0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:53:35.290 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:35.288+0000 7f43f467f700 1 -- 172.21.15.137:0/1575799313 shutdown_connections 2023-11-15T01:53:35.290 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:35.288+0000 7f43f467f700 1 -- 172.21.15.137:0/1575799313 wait complete. 2023-11-15T01:53:35.290 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:53:36.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:53:35 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:53:34.727059+0000 mgr.smithi055.szwngw (mgr.14164) 117 : cluster [DBG] pgmap v110: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:53:36.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:53:35 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:53:35.286963+0000 mon.smithi055 (mon.0) 195 : audit [DBG] from='client.? 172.21.15.137:0/1575799313' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:53:36.668 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:53:36.669 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:53:37.790 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:37.790+0000 7f1a34f81700 1 -- 172.21.15.137:0/1364395012 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1a300f51a0 msgr2=0x7f1a300f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:53:37.791 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:37.790+0000 7f1a34f81700 1 --2- 172.21.15.137:0/1364395012 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1a300f51a0 0x7f1a300f55c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7f1a20009fb0 tx=0x7f1a20009b30).stop 2023-11-15T01:53:37.791 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:37.790+0000 7f1a34f81700 1 -- 172.21.15.137:0/1364395012 shutdown_connections 2023-11-15T01:53:37.791 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:37.790+0000 7f1a34f81700 1 --2- 172.21.15.137:0/1364395012 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1a300f51a0 0x7f1a300f55c0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:53:37.791 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:37.790+0000 7f1a34f81700 1 -- 172.21.15.137:0/1364395012 >> 172.21.15.137:0/1364395012 conn(0x7f1a300f09a0 msgr2=0x7f1a300f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:53:37.792 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:37.790+0000 7f1a34f81700 1 -- 172.21.15.137:0/1364395012 shutdown_connections 2023-11-15T01:53:37.792 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:37.790+0000 7f1a34f81700 1 -- 172.21.15.137:0/1364395012 wait complete. 2023-11-15T01:53:37.792 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:37.790+0000 7f1a34f81700 1 Processor -- start 2023-11-15T01:53:37.794 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:37.790+0000 7f1a34f81700 1 -- start start 2023-11-15T01:53:37.794 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:37.790+0000 7f1a34f81700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1a300f51a0 0x7f1a30109170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:53:37.794 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:37.790+0000 7f1a34f81700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a30060280 con 0x7f1a300f51a0 2023-11-15T01:53:37.795 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:37.791+0000 7f1a2f7fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1a300f51a0 0x7f1a30109170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:53:37.795 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:37.791+0000 7f1a2f7fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1a300f51a0 0x7f1a30109170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:48440/0 (socket says 172.21.15.137:48440) 2023-11-15T01:53:37.795 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:37.791+0000 7f1a2f7fe700 1 -- 172.21.15.137:0/2335394932 learned_addr learned my addr 172.21.15.137:0/2335394932 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:53:37.795 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:37.791+0000 7f1a2f7fe700 1 -- 172.21.15.137:0/2335394932 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a20005040 con 0x7f1a300f51a0 2023-11-15T01:53:37.795 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:37.791+0000 7f1a2f7fe700 1 --2- 172.21.15.137:0/2335394932 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1a300f51a0 0x7f1a30109170 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f1a20013510 tx=0x7f1a2000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:53:37.796 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:37.791+0000 7f1a17fff700 1 -- 172.21.15.137:0/2335394932 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a20009320 con 0x7f1a300f51a0 2023-11-15T01:53:37.796 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:37.791+0000 7f1a17fff700 1 -- 172.21.15.137:0/2335394932 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1a20013f00 con 0x7f1a300f51a0 2023-11-15T01:53:37.796 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:37.791+0000 7f1a34f81700 1 -- 172.21.15.137:0/2335394932 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a30060480 con 0x7f1a300f51a0 2023-11-15T01:53:37.796 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:37.791+0000 7f1a17fff700 1 -- 172.21.15.137:0/2335394932 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a20014dc0 con 0x7f1a300f51a0 2023-11-15T01:53:37.796 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:37.791+0000 7f1a34f81700 1 -- 172.21.15.137:0/2335394932 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a301098a0 con 0x7f1a300f51a0 2023-11-15T01:53:37.797 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:37.792+0000 7f1a17fff700 1 -- 172.21.15.137:0/2335394932 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1a200095f0 con 0x7f1a300f51a0 2023-11-15T01:53:37.797 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:37.792+0000 7f1a34f81700 1 -- 172.21.15.137:0/2335394932 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a30045ad0 con 0x7f1a300f51a0 2023-11-15T01:53:37.797 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:37.793+0000 7f1a17fff700 1 --2- 172.21.15.137:0/2335394932 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f1a1802ff60 0x7f1a18032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:53:37.797 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:37.793+0000 7f1a17fff700 1 -- 172.21.15.137:0/2335394932 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1a20041000 con 0x7f1a300f51a0 2023-11-15T01:53:37.797 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:37.793+0000 7f1a2effd700 1 --2- 172.21.15.137:0/2335394932 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f1a1802ff60 0x7f1a18032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:53:37.797 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:37.793+0000 7f1a2effd700 1 --2- 172.21.15.137:0/2335394932 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f1a1802ff60 0x7f1a18032420 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f1a240096a0 tx=0x7f1a24006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:53:37.798 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:37.798+0000 7f1a17fff700 1 -- 172.21.15.137:0/2335394932 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1a20014420 con 0x7f1a300f51a0 2023-11-15T01:53:38.003 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:38.002+0000 7f1a34f81700 1 -- 172.21.15.137:0/2335394932 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1a300fabe0 con 0x7f1a300f51a0 2023-11-15T01:53:38.003 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:38.003+0000 7f1a17fff700 1 -- 172.21.15.137:0/2335394932 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1a20010230 con 0x7f1a300f51a0 2023-11-15T01:53:38.004 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:53:38.004 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:53:38.005 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:38.004+0000 7f1a34f81700 1 -- 172.21.15.137:0/2335394932 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f1a1802ff60 msgr2=0x7f1a18032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:53:38.005 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:38.005+0000 7f1a34f81700 1 --2- 172.21.15.137:0/2335394932 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f1a1802ff60 0x7f1a18032420 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f1a240096a0 tx=0x7f1a24006b40).stop 2023-11-15T01:53:38.006 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:38.005+0000 7f1a34f81700 1 -- 172.21.15.137:0/2335394932 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1a300f51a0 msgr2=0x7f1a30109170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:53:38.006 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:38.005+0000 7f1a34f81700 1 --2- 172.21.15.137:0/2335394932 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1a300f51a0 0x7f1a30109170 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f1a20013510 tx=0x7f1a2000b040).stop 2023-11-15T01:53:38.006 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:38.005+0000 7f1a34f81700 1 -- 172.21.15.137:0/2335394932 shutdown_connections 2023-11-15T01:53:38.006 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:38.005+0000 7f1a34f81700 1 --2- 172.21.15.137:0/2335394932 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f1a1802ff60 0x7f1a18032420 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:53:38.006 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:38.005+0000 7f1a34f81700 1 --2- 172.21.15.137:0/2335394932 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1a300f51a0 0x7f1a30109170 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:53:38.007 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:38.005+0000 7f1a34f81700 1 -- 172.21.15.137:0/2335394932 >> 172.21.15.137:0/2335394932 conn(0x7f1a300f09a0 msgr2=0x7f1a300f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:53:38.007 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:38.005+0000 7f1a34f81700 1 -- 172.21.15.137:0/2335394932 shutdown_connections 2023-11-15T01:53:38.007 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:38.005+0000 7f1a34f81700 1 -- 172.21.15.137:0/2335394932 wait complete. 2023-11-15T01:53:38.008 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:53:38.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:53:37 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:53:36.727339+0000 mgr.smithi055.szwngw (mgr.14164) 118 : cluster [DBG] pgmap v111: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:53:39.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:53:38 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:53:38.003979+0000 mon.smithi055 (mon.0) 196 : audit [DBG] from='client.? 172.21.15.137:0/2335394932' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:53:39.394 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:53:39.394 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:53:40.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:53:39 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:53:38.727700+0000 mgr.smithi055.szwngw (mgr.14164) 119 : cluster [DBG] pgmap v112: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:53:40.521 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:40.518+0000 7f3b23db0700 1 -- 172.21.15.137:0/13938060 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3b1c0f5410 msgr2=0x7f3b1c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:53:40.522 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:40.518+0000 7f3b23db0700 1 --2- 172.21.15.137:0/13938060 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3b1c0f5410 0x7f3b1c0f5830 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7f3b0c009fb0 tx=0x7f3b0c009b30).stop 2023-11-15T01:53:40.522 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:40.518+0000 7f3b23db0700 1 -- 172.21.15.137:0/13938060 shutdown_connections 2023-11-15T01:53:40.522 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:40.518+0000 7f3b23db0700 1 --2- 172.21.15.137:0/13938060 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3b1c0f5410 0x7f3b1c0f5830 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:53:40.523 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:40.518+0000 7f3b23db0700 1 -- 172.21.15.137:0/13938060 >> 172.21.15.137:0/13938060 conn(0x7f3b1c0f0b70 msgr2=0x7f3b1c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:53:40.523 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:40.518+0000 7f3b23db0700 1 -- 172.21.15.137:0/13938060 shutdown_connections 2023-11-15T01:53:40.523 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:40.518+0000 7f3b23db0700 1 -- 172.21.15.137:0/13938060 wait complete. 2023-11-15T01:53:40.523 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:40.519+0000 7f3b23db0700 1 Processor -- start 2023-11-15T01:53:40.523 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:40.519+0000 7f3b23db0700 1 -- start start 2023-11-15T01:53:40.523 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:40.519+0000 7f3b23db0700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3b1c106830 0x7f3b1c106c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:53:40.524 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:40.519+0000 7f3b23db0700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b1c107140 con 0x7f3b1c106830 2023-11-15T01:53:40.524 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:40.519+0000 7f3b21b4c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3b1c106830 0x7f3b1c106c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:53:40.524 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:40.519+0000 7f3b21b4c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3b1c106830 0x7f3b1c106c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:43612/0 (socket says 172.21.15.137:43612) 2023-11-15T01:53:40.525 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:40.519+0000 7f3b21b4c700 1 -- 172.21.15.137:0/1205744175 learned_addr learned my addr 172.21.15.137:0/1205744175 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:53:40.525 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:40.520+0000 7f3b21b4c700 1 -- 172.21.15.137:0/1205744175 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b0c005040 con 0x7f3b1c106830 2023-11-15T01:53:40.525 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:40.520+0000 7f3b21b4c700 1 --2- 172.21.15.137:0/1205744175 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3b1c106830 0x7f3b1c106c50 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f3b0c009f80 tx=0x7f3b0c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:53:40.525 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:40.520+0000 7f3b127fc700 1 -- 172.21.15.137:0/1205744175 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3b0c0136c0 con 0x7f3b1c106830 2023-11-15T01:53:40.525 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:40.520+0000 7f3b23db0700 1 -- 172.21.15.137:0/1205744175 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b1c107340 con 0x7f3b1c106830 2023-11-15T01:53:40.526 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:40.520+0000 7f3b127fc700 1 -- 172.21.15.137:0/1205744175 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3b0c0095f0 con 0x7f3b1c106830 2023-11-15T01:53:40.526 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:40.520+0000 7f3b127fc700 1 -- 172.21.15.137:0/1205744175 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3b0c014420 con 0x7f3b1c106830 2023-11-15T01:53:40.526 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:40.520+0000 7f3b23db0700 1 -- 172.21.15.137:0/1205744175 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b1c107e90 con 0x7f3b1c106830 2023-11-15T01:53:40.526 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:40.521+0000 7f3b127fc700 1 -- 172.21.15.137:0/1205744175 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3b0c025060 con 0x7f3b1c106830 2023-11-15T01:53:40.530 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:40.521+0000 7f3b23db0700 1 -- 172.21.15.137:0/1205744175 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b1c059100 con 0x7f3b1c106830 2023-11-15T01:53:40.530 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:40.521+0000 7f3b127fc700 1 --2- 172.21.15.137:0/1205744175 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f3b0802ff60 0x7f3b08032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:53:40.530 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:40.521+0000 7f3b127fc700 1 -- 172.21.15.137:0/1205744175 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3b0c040d80 con 0x7f3b1c106830 2023-11-15T01:53:40.531 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:40.521+0000 7f3b2134b700 1 --2- 172.21.15.137:0/1205744175 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f3b0802ff60 0x7f3b08032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:53:40.531 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:40.522+0000 7f3b2134b700 1 --2- 172.21.15.137:0/1205744175 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f3b0802ff60 0x7f3b08032420 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f3b180096a0 tx=0x7f3b18006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:53:40.532 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:40.526+0000 7f3b127fc700 1 -- 172.21.15.137:0/1205744175 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3b0c01cb60 con 0x7f3b1c106830 2023-11-15T01:53:40.733 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:40.733+0000 7f3b23db0700 1 -- 172.21.15.137:0/1205744175 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3b1c108650 con 0x7f3b1c106830 2023-11-15T01:53:40.734 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:40.733+0000 7f3b127fc700 1 -- 172.21.15.137:0/1205744175 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3b0c042070 con 0x7f3b1c106830 2023-11-15T01:53:40.735 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:53:40.735 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:53:40.736 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:40.735+0000 7f3b23db0700 1 -- 172.21.15.137:0/1205744175 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f3b0802ff60 msgr2=0x7f3b08032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:53:40.736 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:40.735+0000 7f3b23db0700 1 --2- 172.21.15.137:0/1205744175 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f3b0802ff60 0x7f3b08032420 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f3b180096a0 tx=0x7f3b18006b40).stop 2023-11-15T01:53:40.736 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:40.735+0000 7f3b23db0700 1 -- 172.21.15.137:0/1205744175 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3b1c106830 msgr2=0x7f3b1c106c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:53:40.736 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:40.735+0000 7f3b23db0700 1 --2- 172.21.15.137:0/1205744175 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3b1c106830 0x7f3b1c106c50 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f3b0c009f80 tx=0x7f3b0c009b30).stop 2023-11-15T01:53:40.737 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:40.735+0000 7f3b23db0700 1 -- 172.21.15.137:0/1205744175 shutdown_connections 2023-11-15T01:53:40.737 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:40.735+0000 7f3b23db0700 1 --2- 172.21.15.137:0/1205744175 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f3b0802ff60 0x7f3b08032420 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:53:40.737 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:40.735+0000 7f3b23db0700 1 --2- 172.21.15.137:0/1205744175 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3b1c106830 0x7f3b1c106c50 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:53:40.737 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:40.735+0000 7f3b23db0700 1 -- 172.21.15.137:0/1205744175 >> 172.21.15.137:0/1205744175 conn(0x7f3b1c0f0b70 msgr2=0x7f3b1c05fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:53:40.737 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:40.735+0000 7f3b23db0700 1 -- 172.21.15.137:0/1205744175 shutdown_connections 2023-11-15T01:53:40.738 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:40.735+0000 7f3b23db0700 1 -- 172.21.15.137:0/1205744175 wait complete. 2023-11-15T01:53:40.738 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:53:41.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:53:41 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:53:40.727989+0000 mgr.smithi055.szwngw (mgr.14164) 120 : cluster [DBG] pgmap v113: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:53:41.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:53:41 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:53:40.734715+0000 mon.smithi055 (mon.0) 197 : audit [DBG] from='client.? 172.21.15.137:0/1205744175' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:53:42.107 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:53:42.108 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:53:43.216 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:43.215+0000 7fe8cd983700 1 -- 172.21.15.137:0/1259720259 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe8c80f62c0 msgr2=0x7fe8c80f66e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:53:43.216 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:43.215+0000 7fe8cd983700 1 --2- 172.21.15.137:0/1259720259 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe8c80f62c0 0x7fe8c80f66e0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7fe8b8009fb0 tx=0x7fe8b8009b30).stop 2023-11-15T01:53:43.216 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:43.215+0000 7fe8cd983700 1 -- 172.21.15.137:0/1259720259 shutdown_connections 2023-11-15T01:53:43.217 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:43.215+0000 7fe8cd983700 1 --2- 172.21.15.137:0/1259720259 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe8c80f62c0 0x7fe8c80f66e0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:53:43.217 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:43.215+0000 7fe8cd983700 1 -- 172.21.15.137:0/1259720259 >> 172.21.15.137:0/1259720259 conn(0x7fe8c80f1a20 msgr2=0x7fe8c80f3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:53:43.217 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:43.215+0000 7fe8cd983700 1 -- 172.21.15.137:0/1259720259 shutdown_connections 2023-11-15T01:53:43.217 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:43.215+0000 7fe8cd983700 1 -- 172.21.15.137:0/1259720259 wait complete. 2023-11-15T01:53:43.220 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:43.215+0000 7fe8cd983700 1 Processor -- start 2023-11-15T01:53:43.220 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:43.216+0000 7fe8cd983700 1 -- start start 2023-11-15T01:53:43.220 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:43.216+0000 7fe8cd983700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe8c80f62c0 0x7fe8c8106580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:53:43.220 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:43.216+0000 7fe8cd983700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8c8106a70 con 0x7fe8c80f62c0 2023-11-15T01:53:43.221 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:43.216+0000 7fe8c6ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe8c80f62c0 0x7fe8c8106580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:53:43.221 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:43.216+0000 7fe8c6ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe8c80f62c0 0x7fe8c8106580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:43632/0 (socket says 172.21.15.137:43632) 2023-11-15T01:53:43.221 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:43.216+0000 7fe8c6ffd700 1 -- 172.21.15.137:0/3529762554 learned_addr learned my addr 172.21.15.137:0/3529762554 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:53:43.221 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:43.217+0000 7fe8c6ffd700 1 -- 172.21.15.137:0/3529762554 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8b8005040 con 0x7fe8c80f62c0 2023-11-15T01:53:43.222 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:43.217+0000 7fe8c6ffd700 1 --2- 172.21.15.137:0/3529762554 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe8c80f62c0 0x7fe8c8106580 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7fe8b8013510 tx=0x7fe8b800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:53:43.222 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:43.217+0000 7fe8af7fe700 1 -- 172.21.15.137:0/3529762554 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe8b8009320 con 0x7fe8c80f62c0 2023-11-15T01:53:43.222 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:43.217+0000 7fe8af7fe700 1 -- 172.21.15.137:0/3529762554 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe8b8013f00 con 0x7fe8c80f62c0 2023-11-15T01:53:43.222 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:43.217+0000 7fe8cd983700 1 -- 172.21.15.137:0/3529762554 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8c8106c70 con 0x7fe8c80f62c0 2023-11-15T01:53:43.222 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:43.217+0000 7fe8af7fe700 1 -- 172.21.15.137:0/3529762554 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe8b8014d90 con 0x7fe8c80f62c0 2023-11-15T01:53:43.223 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:43.217+0000 7fe8cd983700 1 -- 172.21.15.137:0/3529762554 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8c8107100 con 0x7fe8c80f62c0 2023-11-15T01:53:43.223 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:43.218+0000 7fe8af7fe700 1 -- 172.21.15.137:0/3529762554 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe8b80095f0 con 0x7fe8c80f62c0 2023-11-15T01:53:43.223 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:43.218+0000 7fe8cd983700 1 -- 172.21.15.137:0/3529762554 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8c8045ad0 con 0x7fe8c80f62c0 2023-11-15T01:53:43.223 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:43.218+0000 7fe8af7fe700 1 --2- 172.21.15.137:0/3529762554 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fe8b002ff60 0x7fe8b0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:53:43.223 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:43.218+0000 7fe8af7fe700 1 -- 172.21.15.137:0/3529762554 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe8b80410c0 con 0x7fe8c80f62c0 2023-11-15T01:53:43.224 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:43.218+0000 7fe8c67fc700 1 --2- 172.21.15.137:0/3529762554 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fe8b002ff60 0x7fe8b0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:53:43.224 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:43.219+0000 7fe8c67fc700 1 --2- 172.21.15.137:0/3529762554 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fe8b002ff60 0x7fe8b0032420 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7fe8bc0096a0 tx=0x7fe8bc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:53:43.224 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:43.223+0000 7fe8af7fe700 1 -- 172.21.15.137:0/3529762554 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe8b8014420 con 0x7fe8c80f62c0 2023-11-15T01:53:43.429 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:43.428+0000 7fe8cd983700 1 -- 172.21.15.137:0/3529762554 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe8c8107f00 con 0x7fe8c80f62c0 2023-11-15T01:53:43.429 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:43.429+0000 7fe8af7fe700 1 -- 172.21.15.137:0/3529762554 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe8b801d7f0 con 0x7fe8c80f62c0 2023-11-15T01:53:43.430 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:53:43.430 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:53:43.431 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:43.430+0000 7fe8cd983700 1 -- 172.21.15.137:0/3529762554 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fe8b002ff60 msgr2=0x7fe8b0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:53:43.431 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:43.430+0000 7fe8cd983700 1 --2- 172.21.15.137:0/3529762554 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fe8b002ff60 0x7fe8b0032420 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7fe8bc0096a0 tx=0x7fe8bc006b40).stop 2023-11-15T01:53:43.431 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:43.430+0000 7fe8cd983700 1 -- 172.21.15.137:0/3529762554 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe8c80f62c0 msgr2=0x7fe8c8106580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:53:43.432 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:43.430+0000 7fe8cd983700 1 --2- 172.21.15.137:0/3529762554 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe8c80f62c0 0x7fe8c8106580 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7fe8b8013510 tx=0x7fe8b800b040).stop 2023-11-15T01:53:43.432 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:43.430+0000 7fe8cd983700 1 -- 172.21.15.137:0/3529762554 shutdown_connections 2023-11-15T01:53:43.432 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:43.430+0000 7fe8cd983700 1 --2- 172.21.15.137:0/3529762554 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fe8b002ff60 0x7fe8b0032420 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:53:43.432 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:43.430+0000 7fe8cd983700 1 --2- 172.21.15.137:0/3529762554 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe8c80f62c0 0x7fe8c8106580 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:53:43.432 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:43.430+0000 7fe8cd983700 1 -- 172.21.15.137:0/3529762554 >> 172.21.15.137:0/3529762554 conn(0x7fe8c80f1a20 msgr2=0x7fe8c80f2580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:53:43.433 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:43.430+0000 7fe8cd983700 1 -- 172.21.15.137:0/3529762554 shutdown_connections 2023-11-15T01:53:43.433 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:43.430+0000 7fe8cd983700 1 -- 172.21.15.137:0/3529762554 wait complete. 2023-11-15T01:53:43.433 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:53:44.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:53:43 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:53:42.728333+0000 mgr.smithi055.szwngw (mgr.14164) 121 : cluster [DBG] pgmap v114: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:53:44.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:53:43 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:53:42.851257+0000 mon.smithi055 (mon.0) 198 : audit [INF] from='mgr.14164 172.21.15.55:0/59463371' entity='mgr.smithi055.szwngw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi055.szwngw/mirror_snapshot_schedule"}]: dispatch 2023-11-15T01:53:44.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:53:43 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:53:42.853940+0000 mon.smithi055 (mon.0) 199 : audit [INF] from='mgr.14164 172.21.15.55:0/59463371' entity='mgr.smithi055.szwngw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi055.szwngw/trash_purge_schedule"}]: dispatch 2023-11-15T01:53:44.079 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:53:43 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:53:43.429739+0000 mon.smithi055 (mon.0) 200 : audit [DBG] from='client.? 172.21.15.137:0/3529762554' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:53:44.805 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:53:44.806 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:53:45.924 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:45.922+0000 7f5f48b31700 1 -- 172.21.15.137:0/3786277576 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5f440f5410 msgr2=0x7f5f440f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:53:45.924 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:45.922+0000 7f5f48b31700 1 --2- 172.21.15.137:0/3786277576 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5f440f5410 0x7f5f440f5830 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7f5f2c009fb0 tx=0x7f5f2c009b30).stop 2023-11-15T01:53:45.924 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:45.922+0000 7f5f48b31700 1 -- 172.21.15.137:0/3786277576 shutdown_connections 2023-11-15T01:53:45.925 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:45.922+0000 7f5f48b31700 1 --2- 172.21.15.137:0/3786277576 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5f440f5410 0x7f5f440f5830 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:53:45.925 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:45.922+0000 7f5f48b31700 1 -- 172.21.15.137:0/3786277576 >> 172.21.15.137:0/3786277576 conn(0x7f5f440f0b70 msgr2=0x7f5f440f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:53:45.925 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:45.923+0000 7f5f48b31700 1 -- 172.21.15.137:0/3786277576 shutdown_connections 2023-11-15T01:53:45.925 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:45.923+0000 7f5f48b31700 1 -- 172.21.15.137:0/3786277576 wait complete. 2023-11-15T01:53:45.925 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:45.923+0000 7f5f48b31700 1 Processor -- start 2023-11-15T01:53:45.926 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:45.923+0000 7f5f48b31700 1 -- start start 2023-11-15T01:53:45.926 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:45.923+0000 7f5f48b31700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5f440f5410 0x7f5f44109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:53:45.926 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:45.923+0000 7f5f48b31700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f44060240 con 0x7f5f440f5410 2023-11-15T01:53:45.926 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:45.923+0000 7f5f4259c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5f440f5410 0x7f5f44109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:53:45.926 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:45.923+0000 7f5f4259c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5f440f5410 0x7f5f44109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:43646/0 (socket says 172.21.15.137:43646) 2023-11-15T01:53:45.927 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:45.923+0000 7f5f4259c700 1 -- 172.21.15.137:0/3968459043 learned_addr learned my addr 172.21.15.137:0/3968459043 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:53:45.928 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:45.924+0000 7f5f4259c700 1 -- 172.21.15.137:0/3968459043 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f2c005040 con 0x7f5f440f5410 2023-11-15T01:53:45.928 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:45.924+0000 7f5f4259c700 1 --2- 172.21.15.137:0/3968459043 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5f440f5410 0x7f5f44109430 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f5f2c013510 tx=0x7f5f2c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:53:45.928 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:45.924+0000 7f5f3affd700 1 -- 172.21.15.137:0/3968459043 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5f2c009320 con 0x7f5f440f5410 2023-11-15T01:53:45.929 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:45.924+0000 7f5f48b31700 1 -- 172.21.15.137:0/3968459043 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f44060440 con 0x7f5f440f5410 2023-11-15T01:53:45.929 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:45.924+0000 7f5f3affd700 1 -- 172.21.15.137:0/3968459043 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5f2c013f00 con 0x7f5f440f5410 2023-11-15T01:53:45.929 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:45.924+0000 7f5f3affd700 1 -- 172.21.15.137:0/3968459043 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5f2c014d90 con 0x7f5f440f5410 2023-11-15T01:53:45.929 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:45.924+0000 7f5f48b31700 1 -- 172.21.15.137:0/3968459043 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f44109ae0 con 0x7f5f440f5410 2023-11-15T01:53:45.929 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:45.925+0000 7f5f3affd700 1 -- 172.21.15.137:0/3968459043 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5f2c0263a0 con 0x7f5f440f5410 2023-11-15T01:53:45.930 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:45.925+0000 7f5f48b31700 1 -- 172.21.15.137:0/3968459043 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f44045ad0 con 0x7f5f440f5410 2023-11-15T01:53:45.930 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:45.925+0000 7f5f3affd700 1 --2- 172.21.15.137:0/3968459043 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f5f3002ff60 0x7f5f30032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:53:45.930 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:45.925+0000 7f5f3affd700 1 -- 172.21.15.137:0/3968459043 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5f2c041310 con 0x7f5f440f5410 2023-11-15T01:53:45.931 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:45.925+0000 7f5f41d9b700 1 --2- 172.21.15.137:0/3968459043 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f5f3002ff60 0x7f5f30032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:53:45.931 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:45.926+0000 7f5f41d9b700 1 --2- 172.21.15.137:0/3968459043 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f5f3002ff60 0x7f5f30032420 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f5f340096a0 tx=0x7f5f34006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:53:45.931 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:45.930+0000 7f5f3affd700 1 -- 172.21.15.137:0/3968459043 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5f2c0126f0 con 0x7f5f440f5410 2023-11-15T01:53:46.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:53:45 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:53:44.728699+0000 mgr.smithi055.szwngw (mgr.14164) 122 : cluster [DBG] pgmap v115: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:53:46.137 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:46.136+0000 7f5f48b31700 1 -- 172.21.15.137:0/3968459043 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5f440fae60 con 0x7f5f440f5410 2023-11-15T01:53:46.139 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:46.137+0000 7f5f3affd700 1 -- 172.21.15.137:0/3968459043 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5f2c012330 con 0x7f5f440f5410 2023-11-15T01:53:46.140 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:53:46.140 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:53:46.141 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:46.139+0000 7f5f48b31700 1 -- 172.21.15.137:0/3968459043 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f5f3002ff60 msgr2=0x7f5f30032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:53:46.141 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:46.139+0000 7f5f48b31700 1 --2- 172.21.15.137:0/3968459043 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f5f3002ff60 0x7f5f30032420 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f5f340096a0 tx=0x7f5f34006b40).stop 2023-11-15T01:53:46.141 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:46.139+0000 7f5f48b31700 1 -- 172.21.15.137:0/3968459043 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5f440f5410 msgr2=0x7f5f44109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:53:46.141 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:46.139+0000 7f5f48b31700 1 --2- 172.21.15.137:0/3968459043 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5f440f5410 0x7f5f44109430 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f5f2c013510 tx=0x7f5f2c00b040).stop 2023-11-15T01:53:46.142 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:46.139+0000 7f5f48b31700 1 -- 172.21.15.137:0/3968459043 shutdown_connections 2023-11-15T01:53:46.142 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:46.139+0000 7f5f48b31700 1 --2- 172.21.15.137:0/3968459043 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f5f3002ff60 0x7f5f30032420 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:53:46.142 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:46.139+0000 7f5f48b31700 1 --2- 172.21.15.137:0/3968459043 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5f440f5410 0x7f5f44109430 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:53:46.142 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:46.139+0000 7f5f48b31700 1 -- 172.21.15.137:0/3968459043 >> 172.21.15.137:0/3968459043 conn(0x7f5f440f0b70 msgr2=0x7f5f440f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:53:46.142 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:46.139+0000 7f5f48b31700 1 -- 172.21.15.137:0/3968459043 shutdown_connections 2023-11-15T01:53:46.143 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:46.139+0000 7f5f48b31700 1 -- 172.21.15.137:0/3968459043 wait complete. 2023-11-15T01:53:46.143 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:53:47.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:53:46 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:53:46.138245+0000 mon.smithi055 (mon.0) 201 : audit [DBG] from='client.? 172.21.15.137:0/3968459043' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:53:47.513 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:53:47.514 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:53:48.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:53:47 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:53:46.728973+0000 mgr.smithi055.szwngw (mgr.14164) 123 : cluster [DBG] pgmap v116: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:53:48.650 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:48.649+0000 7f9d86299700 1 -- 172.21.15.137:0/3237916399 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9d80060220 msgr2=0x7f9d800f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:53:48.650 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:48.649+0000 7f9d86299700 1 --2- 172.21.15.137:0/3237916399 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9d80060220 0x7f9d800f9150 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7f9d68009fb0 tx=0x7f9d68009b30).stop 2023-11-15T01:53:48.651 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:48.649+0000 7f9d86299700 1 -- 172.21.15.137:0/3237916399 shutdown_connections 2023-11-15T01:53:48.651 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:48.649+0000 7f9d86299700 1 --2- 172.21.15.137:0/3237916399 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9d80060220 0x7f9d800f9150 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:53:48.651 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:48.649+0000 7f9d86299700 1 -- 172.21.15.137:0/3237916399 >> 172.21.15.137:0/3237916399 conn(0x7f9d800f0b70 msgr2=0x7f9d800f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:53:48.651 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:48.649+0000 7f9d86299700 1 -- 172.21.15.137:0/3237916399 shutdown_connections 2023-11-15T01:53:48.651 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:48.649+0000 7f9d86299700 1 -- 172.21.15.137:0/3237916399 wait complete. 2023-11-15T01:53:48.653 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:48.650+0000 7f9d86299700 1 Processor -- start 2023-11-15T01:53:48.653 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:48.650+0000 7f9d86299700 1 -- start start 2023-11-15T01:53:48.654 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:48.650+0000 7f9d86299700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9d80060220 0x7f9d801066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:53:48.654 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:48.650+0000 7f9d86299700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d80106bd0 con 0x7f9d80060220 2023-11-15T01:53:48.654 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:48.650+0000 7f9d7ffff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9d80060220 0x7f9d801066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:53:48.654 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:48.650+0000 7f9d7ffff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9d80060220 0x7f9d801066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:46032/0 (socket says 172.21.15.137:46032) 2023-11-15T01:53:48.654 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:48.650+0000 7f9d7ffff700 1 -- 172.21.15.137:0/1374734848 learned_addr learned my addr 172.21.15.137:0/1374734848 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:53:48.655 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:48.651+0000 7f9d7ffff700 1 -- 172.21.15.137:0/1374734848 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d68005040 con 0x7f9d80060220 2023-11-15T01:53:48.655 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:48.651+0000 7f9d7ffff700 1 --2- 172.21.15.137:0/1374734848 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9d80060220 0x7f9d801066e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f9d68013510 tx=0x7f9d6800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:53:48.655 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:48.651+0000 7f9d7cff9700 1 -- 172.21.15.137:0/1374734848 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d68009320 con 0x7f9d80060220 2023-11-15T01:53:48.655 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:48.651+0000 7f9d86299700 1 -- 172.21.15.137:0/1374734848 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d80106dd0 con 0x7f9d80060220 2023-11-15T01:53:48.655 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:48.651+0000 7f9d7cff9700 1 -- 172.21.15.137:0/1374734848 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9d68013f00 con 0x7f9d80060220 2023-11-15T01:53:48.655 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:48.651+0000 7f9d7cff9700 1 -- 172.21.15.137:0/1374734848 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d68014dc0 con 0x7f9d80060220 2023-11-15T01:53:48.656 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:48.651+0000 7f9d86299700 1 -- 172.21.15.137:0/1374734848 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d80107260 con 0x7f9d80060220 2023-11-15T01:53:48.656 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:48.652+0000 7f9d7cff9700 1 -- 172.21.15.137:0/1374734848 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9d680095f0 con 0x7f9d80060220 2023-11-15T01:53:48.656 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:48.652+0000 7f9d86299700 1 -- 172.21.15.137:0/1374734848 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d80045ad0 con 0x7f9d80060220 2023-11-15T01:53:48.656 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:48.652+0000 7f9d7cff9700 1 --2- 172.21.15.137:0/1374734848 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f9d6c02ff60 0x7f9d6c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:53:48.657 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:48.652+0000 7f9d7cff9700 1 -- 172.21.15.137:0/1374734848 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9d68041000 con 0x7f9d80060220 2023-11-15T01:53:48.657 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:48.652+0000 7f9d7f7fe700 1 --2- 172.21.15.137:0/1374734848 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f9d6c02ff60 0x7f9d6c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:53:48.657 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:48.653+0000 7f9d7f7fe700 1 --2- 172.21.15.137:0/1374734848 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f9d6c02ff60 0x7f9d6c032420 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f9d700096a0 tx=0x7f9d70006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:53:48.657 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:48.657+0000 7f9d7cff9700 1 -- 172.21.15.137:0/1374734848 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9d6801c790 con 0x7f9d80060220 2023-11-15T01:53:48.868 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:48.865+0000 7f9d86299700 1 -- 172.21.15.137:0/1374734848 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9d801080b0 con 0x7f9d80060220 2023-11-15T01:53:48.869 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:48.865+0000 7f9d7cff9700 1 -- 172.21.15.137:0/1374734848 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9d6801c790 con 0x7f9d80060220 2023-11-15T01:53:48.869 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:53:48.869 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:53:48.870 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:48.867+0000 7f9d86299700 1 -- 172.21.15.137:0/1374734848 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f9d6c02ff60 msgr2=0x7f9d6c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:53:48.871 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:48.867+0000 7f9d86299700 1 --2- 172.21.15.137:0/1374734848 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f9d6c02ff60 0x7f9d6c032420 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f9d700096a0 tx=0x7f9d70006b40).stop 2023-11-15T01:53:48.871 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:48.867+0000 7f9d86299700 1 -- 172.21.15.137:0/1374734848 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9d80060220 msgr2=0x7f9d801066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:53:48.871 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:48.867+0000 7f9d86299700 1 --2- 172.21.15.137:0/1374734848 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9d80060220 0x7f9d801066e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f9d68013510 tx=0x7f9d6800b040).stop 2023-11-15T01:53:48.871 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:48.867+0000 7f9d86299700 1 -- 172.21.15.137:0/1374734848 shutdown_connections 2023-11-15T01:53:48.872 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:48.867+0000 7f9d86299700 1 --2- 172.21.15.137:0/1374734848 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f9d6c02ff60 0x7f9d6c032420 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:53:48.872 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:48.867+0000 7f9d86299700 1 --2- 172.21.15.137:0/1374734848 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9d80060220 0x7f9d801066e0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:53:48.872 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:48.867+0000 7f9d86299700 1 -- 172.21.15.137:0/1374734848 >> 172.21.15.137:0/1374734848 conn(0x7f9d800f0b70 msgr2=0x7f9d800f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:53:48.873 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:48.867+0000 7f9d86299700 1 -- 172.21.15.137:0/1374734848 shutdown_connections 2023-11-15T01:53:48.873 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:48.867+0000 7f9d86299700 1 -- 172.21.15.137:0/1374734848 wait complete. 2023-11-15T01:53:48.873 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:53:50.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:53:49 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:53:48.729302+0000 mgr.smithi055.szwngw (mgr.14164) 124 : cluster [DBG] pgmap v117: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:53:50.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:53:49 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:53:48.866446+0000 mon.smithi055 (mon.0) 202 : audit [DBG] from='client.? 172.21.15.137:0/1374734848' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:53:50.240 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:53:50.241 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:53:51.356 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:51.354+0000 7f28ac305700 1 -- 172.21.15.137:0/2755612394 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f28a40f5410 msgr2=0x7f28a40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:53:51.356 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:51.354+0000 7f28ac305700 1 --2- 172.21.15.137:0/2755612394 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f28a40f5410 0x7f28a40f5830 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7f2894009fb0 tx=0x7f2894009b30).stop 2023-11-15T01:53:51.356 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:51.354+0000 7f28ac305700 1 -- 172.21.15.137:0/2755612394 shutdown_connections 2023-11-15T01:53:51.357 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:51.354+0000 7f28ac305700 1 --2- 172.21.15.137:0/2755612394 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f28a40f5410 0x7f28a40f5830 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:53:51.357 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:51.354+0000 7f28ac305700 1 -- 172.21.15.137:0/2755612394 >> 172.21.15.137:0/2755612394 conn(0x7f28a40f0b70 msgr2=0x7f28a40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:53:51.357 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:51.354+0000 7f28ac305700 1 -- 172.21.15.137:0/2755612394 shutdown_connections 2023-11-15T01:53:51.357 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:51.354+0000 7f28ac305700 1 -- 172.21.15.137:0/2755612394 wait complete. 2023-11-15T01:53:51.357 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:51.355+0000 7f28ac305700 1 Processor -- start 2023-11-15T01:53:51.358 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:51.355+0000 7f28ac305700 1 -- start start 2023-11-15T01:53:51.358 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:51.355+0000 7f28ac305700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f28a40f5410 0x7f28a4105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:53:51.358 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:51.355+0000 7f28ac305700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28a4105b20 con 0x7f28a40f5410 2023-11-15T01:53:51.358 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:51.355+0000 7f28aa0a1700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f28a40f5410 0x7f28a4105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:53:51.358 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:51.355+0000 7f28aa0a1700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f28a40f5410 0x7f28a4105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:46054/0 (socket says 172.21.15.137:46054) 2023-11-15T01:53:51.359 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:51.355+0000 7f28aa0a1700 1 -- 172.21.15.137:0/1076615806 learned_addr learned my addr 172.21.15.137:0/1076615806 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:53:51.360 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:51.356+0000 7f28aa0a1700 1 -- 172.21.15.137:0/1076615806 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2894005040 con 0x7f28a40f5410 2023-11-15T01:53:51.361 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:51.356+0000 7f28aa0a1700 1 --2- 172.21.15.137:0/1076615806 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f28a40f5410 0x7f28a4105630 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f2894013510 tx=0x7f289400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:53:51.361 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:51.356+0000 7f289affd700 1 -- 172.21.15.137:0/1076615806 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2894009320 con 0x7f28a40f5410 2023-11-15T01:53:51.361 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:51.356+0000 7f28ac305700 1 -- 172.21.15.137:0/1076615806 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28a4105d20 con 0x7f28a40f5410 2023-11-15T01:53:51.361 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:51.356+0000 7f289affd700 1 -- 172.21.15.137:0/1076615806 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2894013f00 con 0x7f28a40f5410 2023-11-15T01:53:51.361 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:51.356+0000 7f289affd700 1 -- 172.21.15.137:0/1076615806 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2894014d90 con 0x7f28a40f5410 2023-11-15T01:53:51.362 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:51.356+0000 7f28ac305700 1 -- 172.21.15.137:0/1076615806 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28a41061b0 con 0x7f28a40f5410 2023-11-15T01:53:51.362 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:51.357+0000 7f289affd700 1 -- 172.21.15.137:0/1076615806 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f28940095f0 con 0x7f28a40f5410 2023-11-15T01:53:51.362 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:51.357+0000 7f28ac305700 1 -- 172.21.15.137:0/1076615806 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28a4045ad0 con 0x7f28a40f5410 2023-11-15T01:53:51.362 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:51.357+0000 7f289affd700 1 --2- 172.21.15.137:0/1076615806 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f289002ff60 0x7f2890032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:53:51.362 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:51.357+0000 7f289affd700 1 -- 172.21.15.137:0/1076615806 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2894041000 con 0x7f28a40f5410 2023-11-15T01:53:51.363 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:51.357+0000 7f28a98a0700 1 --2- 172.21.15.137:0/1076615806 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f289002ff60 0x7f2890032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:53:51.363 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:51.358+0000 7f28a98a0700 1 --2- 172.21.15.137:0/1076615806 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f289002ff60 0x7f2890032420 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f28a00096a0 tx=0x7f28a0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:53:51.363 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:51.362+0000 7f289affd700 1 -- 172.21.15.137:0/1076615806 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2894014420 con 0x7f28a40f5410 2023-11-15T01:53:51.567 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:51.567+0000 7f28ac305700 1 -- 172.21.15.137:0/1076615806 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f28a40fae60 con 0x7f28a40f5410 2023-11-15T01:53:51.568 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:51.568+0000 7f289affd700 1 -- 172.21.15.137:0/1076615806 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2894010230 con 0x7f28a40f5410 2023-11-15T01:53:51.569 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:53:51.569 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:53:51.570 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:51.569+0000 7f28ac305700 1 -- 172.21.15.137:0/1076615806 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f289002ff60 msgr2=0x7f2890032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:53:51.570 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:51.569+0000 7f28ac305700 1 --2- 172.21.15.137:0/1076615806 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f289002ff60 0x7f2890032420 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f28a00096a0 tx=0x7f28a0006b40).stop 2023-11-15T01:53:51.570 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:51.569+0000 7f28ac305700 1 -- 172.21.15.137:0/1076615806 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f28a40f5410 msgr2=0x7f28a4105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:53:51.571 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:51.569+0000 7f28ac305700 1 --2- 172.21.15.137:0/1076615806 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f28a40f5410 0x7f28a4105630 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f2894013510 tx=0x7f289400b040).stop 2023-11-15T01:53:51.571 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:51.569+0000 7f28ac305700 1 -- 172.21.15.137:0/1076615806 shutdown_connections 2023-11-15T01:53:51.571 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:51.569+0000 7f28ac305700 1 --2- 172.21.15.137:0/1076615806 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f289002ff60 0x7f2890032420 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:53:51.571 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:51.569+0000 7f28ac305700 1 --2- 172.21.15.137:0/1076615806 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f28a40f5410 0x7f28a4105630 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:53:51.571 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:51.569+0000 7f28ac305700 1 -- 172.21.15.137:0/1076615806 >> 172.21.15.137:0/1076615806 conn(0x7f28a40f0b70 msgr2=0x7f28a40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:53:51.572 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:51.569+0000 7f28ac305700 1 -- 172.21.15.137:0/1076615806 shutdown_connections 2023-11-15T01:53:51.572 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:51.569+0000 7f28ac305700 1 -- 172.21.15.137:0/1076615806 wait complete. 2023-11-15T01:53:51.572 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:53:51.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:53:51 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:53:50.729588+0000 mgr.smithi055.szwngw (mgr.14164) 125 : cluster [DBG] pgmap v118: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:53:52.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:53:52 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:53:51.568739+0000 mon.smithi055 (mon.0) 203 : audit [DBG] from='client.? 172.21.15.137:0/1076615806' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:53:52.939 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:53:52.939 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:53:53.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:53:53 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:53:52.729822+0000 mgr.smithi055.szwngw (mgr.14164) 126 : cluster [DBG] pgmap v119: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:53:54.060 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:54.058+0000 7fc2a8aa8700 1 -- 172.21.15.137:0/952410713 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc2a40f5410 msgr2=0x7fc2a40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:53:54.060 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:54.058+0000 7fc2a8aa8700 1 --2- 172.21.15.137:0/952410713 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc2a40f5410 0x7fc2a40f5830 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7fc294009fb0 tx=0x7fc294009b30).stop 2023-11-15T01:53:54.060 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:54.058+0000 7fc2a8aa8700 1 -- 172.21.15.137:0/952410713 shutdown_connections 2023-11-15T01:53:54.061 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:54.058+0000 7fc2a8aa8700 1 --2- 172.21.15.137:0/952410713 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc2a40f5410 0x7fc2a40f5830 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:53:54.061 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:54.058+0000 7fc2a8aa8700 1 -- 172.21.15.137:0/952410713 >> 172.21.15.137:0/952410713 conn(0x7fc2a40f0b70 msgr2=0x7fc2a40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:53:54.061 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:54.058+0000 7fc2a8aa8700 1 -- 172.21.15.137:0/952410713 shutdown_connections 2023-11-15T01:53:54.061 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:54.058+0000 7fc2a8aa8700 1 -- 172.21.15.137:0/952410713 wait complete. 2023-11-15T01:53:54.061 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:54.058+0000 7fc2a8aa8700 1 Processor -- start 2023-11-15T01:53:54.062 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:54.059+0000 7fc2a8aa8700 1 -- start start 2023-11-15T01:53:54.062 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:54.059+0000 7fc2a8aa8700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc2a40f5410 0x7fc2a4105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:53:54.062 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:54.059+0000 7fc2a8aa8700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a4105b20 con 0x7fc2a40f5410 2023-11-15T01:53:54.062 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:54.059+0000 7fc2a259c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc2a40f5410 0x7fc2a4105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:53:54.062 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:54.059+0000 7fc2a259c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc2a40f5410 0x7fc2a4105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:46072/0 (socket says 172.21.15.137:46072) 2023-11-15T01:53:54.063 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:54.059+0000 7fc2a259c700 1 -- 172.21.15.137:0/3828809107 learned_addr learned my addr 172.21.15.137:0/3828809107 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:53:54.063 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:54.059+0000 7fc2a259c700 1 -- 172.21.15.137:0/3828809107 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc294005040 con 0x7fc2a40f5410 2023-11-15T01:53:54.063 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:54.059+0000 7fc2a259c700 1 --2- 172.21.15.137:0/3828809107 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc2a40f5410 0x7fc2a4105630 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7fc294013510 tx=0x7fc29400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:53:54.064 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:54.060+0000 7fc292ffd700 1 -- 172.21.15.137:0/3828809107 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc294009320 con 0x7fc2a40f5410 2023-11-15T01:53:54.065 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:54.060+0000 7fc2a8aa8700 1 -- 172.21.15.137:0/3828809107 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2a4105d20 con 0x7fc2a40f5410 2023-11-15T01:53:54.065 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:54.060+0000 7fc292ffd700 1 -- 172.21.15.137:0/3828809107 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc294013f00 con 0x7fc2a40f5410 2023-11-15T01:53:54.065 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:54.060+0000 7fc292ffd700 1 -- 172.21.15.137:0/3828809107 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc294014d90 con 0x7fc2a40f5410 2023-11-15T01:53:54.065 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:54.060+0000 7fc2a8aa8700 1 -- 172.21.15.137:0/3828809107 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2a41061b0 con 0x7fc2a40f5410 2023-11-15T01:53:54.065 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:54.061+0000 7fc292ffd700 1 -- 172.21.15.137:0/3828809107 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc2940095f0 con 0x7fc2a40f5410 2023-11-15T01:53:54.066 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:54.061+0000 7fc2a8aa8700 1 -- 172.21.15.137:0/3828809107 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2a4045ad0 con 0x7fc2a40f5410 2023-11-15T01:53:54.066 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:54.061+0000 7fc292ffd700 1 --2- 172.21.15.137:0/3828809107 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fc28c02ff60 0x7fc28c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:53:54.066 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:54.061+0000 7fc292ffd700 1 -- 172.21.15.137:0/3828809107 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc294040fe0 con 0x7fc2a40f5410 2023-11-15T01:53:54.066 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:54.061+0000 7fc2a1d9b700 1 --2- 172.21.15.137:0/3828809107 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fc28c02ff60 0x7fc28c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:53:54.067 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:54.061+0000 7fc2a1d9b700 1 --2- 172.21.15.137:0/3828809107 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fc28c02ff60 0x7fc28c032420 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fc2980096a0 tx=0x7fc298006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:53:54.067 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:54.066+0000 7fc292ffd700 1 -- 172.21.15.137:0/3828809107 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc294014420 con 0x7fc2a40f5410 2023-11-15T01:53:54.272 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:54.271+0000 7fc2a8aa8700 1 -- 172.21.15.137:0/3828809107 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc2a40fae60 con 0x7fc2a40f5410 2023-11-15T01:53:54.272 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:54.271+0000 7fc292ffd700 1 -- 172.21.15.137:0/3828809107 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc294010230 con 0x7fc2a40f5410 2023-11-15T01:53:54.273 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:53:54.273 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:53:54.274 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:54.273+0000 7fc2a8aa8700 1 -- 172.21.15.137:0/3828809107 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fc28c02ff60 msgr2=0x7fc28c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:53:54.275 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:54.273+0000 7fc2a8aa8700 1 --2- 172.21.15.137:0/3828809107 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fc28c02ff60 0x7fc28c032420 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fc2980096a0 tx=0x7fc298006b40).stop 2023-11-15T01:53:54.275 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:54.273+0000 7fc2a8aa8700 1 -- 172.21.15.137:0/3828809107 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc2a40f5410 msgr2=0x7fc2a4105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:53:54.275 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:54.273+0000 7fc2a8aa8700 1 --2- 172.21.15.137:0/3828809107 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc2a40f5410 0x7fc2a4105630 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7fc294013510 tx=0x7fc29400b040).stop 2023-11-15T01:53:54.275 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:54.273+0000 7fc2a8aa8700 1 -- 172.21.15.137:0/3828809107 shutdown_connections 2023-11-15T01:53:54.275 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:54.273+0000 7fc2a8aa8700 1 --2- 172.21.15.137:0/3828809107 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fc28c02ff60 0x7fc28c032420 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:53:54.276 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:54.273+0000 7fc2a8aa8700 1 --2- 172.21.15.137:0/3828809107 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc2a40f5410 0x7fc2a4105630 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:53:54.276 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:54.273+0000 7fc2a8aa8700 1 -- 172.21.15.137:0/3828809107 >> 172.21.15.137:0/3828809107 conn(0x7fc2a40f0b70 msgr2=0x7fc2a40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:53:54.276 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:54.273+0000 7fc2a8aa8700 1 -- 172.21.15.137:0/3828809107 shutdown_connections 2023-11-15T01:53:54.276 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:54.273+0000 7fc2a8aa8700 1 -- 172.21.15.137:0/3828809107 wait complete. 2023-11-15T01:53:54.276 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:53:54.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:53:54 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:53:54.272668+0000 mon.smithi055 (mon.0) 204 : audit [DBG] from='client.? 172.21.15.137:0/3828809107' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:53:55.663 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:53:55.664 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:53:55.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:53:55 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:53:54.730073+0000 mgr.smithi055.szwngw (mgr.14164) 127 : cluster [DBG] pgmap v120: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:53:56.792 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:56.790+0000 7f2a8db5f700 1 -- 172.21.15.137:0/2067619773 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2a880f6510 msgr2=0x7f2a880f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:53:56.792 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:56.790+0000 7f2a8db5f700 1 --2- 172.21.15.137:0/2067619773 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2a880f6510 0x7f2a880f6930 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7f2a7c009fb0 tx=0x7f2a7c009b30).stop 2023-11-15T01:53:56.793 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:56.790+0000 7f2a8db5f700 1 -- 172.21.15.137:0/2067619773 shutdown_connections 2023-11-15T01:53:56.793 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:56.790+0000 7f2a8db5f700 1 --2- 172.21.15.137:0/2067619773 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2a880f6510 0x7f2a880f6930 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:53:56.793 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:56.790+0000 7f2a8db5f700 1 -- 172.21.15.137:0/2067619773 >> 172.21.15.137:0/2067619773 conn(0x7f2a880f1c70 msgr2=0x7f2a880f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:53:56.793 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:56.790+0000 7f2a8db5f700 1 -- 172.21.15.137:0/2067619773 shutdown_connections 2023-11-15T01:53:56.793 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:56.790+0000 7f2a8db5f700 1 -- 172.21.15.137:0/2067619773 wait complete. 2023-11-15T01:53:56.794 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:56.791+0000 7f2a8db5f700 1 Processor -- start 2023-11-15T01:53:56.794 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:56.791+0000 7f2a8db5f700 1 -- start start 2023-11-15T01:53:56.794 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:56.791+0000 7f2a8db5f700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2a880f6510 0x7f2a8806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:53:56.794 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:56.791+0000 7f2a8db5f700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a88069e20 con 0x7f2a880f6510 2023-11-15T01:53:56.794 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:56.791+0000 7f2a877fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2a880f6510 0x7f2a8806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:53:56.795 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:56.791+0000 7f2a877fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2a880f6510 0x7f2a8806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:46092/0 (socket says 172.21.15.137:46092) 2023-11-15T01:53:56.795 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:56.792+0000 7f2a877fe700 1 -- 172.21.15.137:0/3098449376 learned_addr learned my addr 172.21.15.137:0/3098449376 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:53:56.795 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:56.792+0000 7f2a877fe700 1 -- 172.21.15.137:0/3098449376 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a7c005040 con 0x7f2a880f6510 2023-11-15T01:53:56.796 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:56.792+0000 7f2a877fe700 1 --2- 172.21.15.137:0/3098449376 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2a880f6510 0x7f2a8806b9a0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f2a7c014000 tx=0x7f2a7c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:53:56.797 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:56.792+0000 7f2a6ffff700 1 -- 172.21.15.137:0/3098449376 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2a7c0095f0 con 0x7f2a880f6510 2023-11-15T01:53:56.797 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:56.792+0000 7f2a6ffff700 1 -- 172.21.15.137:0/3098449376 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2a7c007e70 con 0x7f2a880f6510 2023-11-15T01:53:56.797 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:56.792+0000 7f2a8db5f700 1 -- 172.21.15.137:0/3098449376 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a88069fc0 con 0x7f2a880f6510 2023-11-15T01:53:56.797 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:56.792+0000 7f2a6ffff700 1 -- 172.21.15.137:0/3098449376 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2a7c014ea0 con 0x7f2a880f6510 2023-11-15T01:53:56.797 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:56.792+0000 7f2a8db5f700 1 -- 172.21.15.137:0/3098449376 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a8806a450 con 0x7f2a880f6510 2023-11-15T01:53:56.798 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:56.793+0000 7f2a8db5f700 1 -- 172.21.15.137:0/3098449376 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a88059100 con 0x7f2a880f6510 2023-11-15T01:53:56.798 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:56.793+0000 7f2a6ffff700 1 -- 172.21.15.137:0/3098449376 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2a7c025070 con 0x7f2a880f6510 2023-11-15T01:53:56.798 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:56.793+0000 7f2a6ffff700 1 --2- 172.21.15.137:0/3098449376 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f2a7002ff60 0x7f2a70032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:53:56.798 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:56.793+0000 7f2a6ffff700 1 -- 172.21.15.137:0/3098449376 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2a7c040940 con 0x7f2a880f6510 2023-11-15T01:53:56.798 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:56.794+0000 7f2a86ffd700 1 --2- 172.21.15.137:0/3098449376 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f2a7002ff60 0x7f2a70032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:53:56.799 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:56.794+0000 7f2a86ffd700 1 --2- 172.21.15.137:0/3098449376 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f2a7002ff60 0x7f2a70032420 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f2a780096a0 tx=0x7f2a78006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:53:56.799 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:56.798+0000 7f2a6ffff700 1 -- 172.21.15.137:0/3098449376 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2a7c0079c0 con 0x7f2a880f6510 2023-11-15T01:53:57.005 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:57.005+0000 7f2a8db5f700 1 -- 172.21.15.137:0/3098449376 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2a88045ad0 con 0x7f2a880f6510 2023-11-15T01:53:57.006 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:57.005+0000 7f2a6ffff700 1 -- 172.21.15.137:0/3098449376 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2a7c00f300 con 0x7f2a880f6510 2023-11-15T01:53:57.006 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:53:57.007 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:53:57.008 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:57.007+0000 7f2a8db5f700 1 -- 172.21.15.137:0/3098449376 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f2a7002ff60 msgr2=0x7f2a70032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:53:57.008 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:57.007+0000 7f2a8db5f700 1 --2- 172.21.15.137:0/3098449376 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f2a7002ff60 0x7f2a70032420 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f2a780096a0 tx=0x7f2a78006b40).stop 2023-11-15T01:53:57.008 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:57.007+0000 7f2a8db5f700 1 -- 172.21.15.137:0/3098449376 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2a880f6510 msgr2=0x7f2a8806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:53:57.008 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:57.007+0000 7f2a8db5f700 1 --2- 172.21.15.137:0/3098449376 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2a880f6510 0x7f2a8806b9a0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f2a7c014000 tx=0x7f2a7c00b040).stop 2023-11-15T01:53:57.009 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:57.007+0000 7f2a8db5f700 1 -- 172.21.15.137:0/3098449376 shutdown_connections 2023-11-15T01:53:57.009 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:57.007+0000 7f2a8db5f700 1 --2- 172.21.15.137:0/3098449376 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f2a7002ff60 0x7f2a70032420 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:53:57.009 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:57.007+0000 7f2a8db5f700 1 --2- 172.21.15.137:0/3098449376 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2a880f6510 0x7f2a8806b9a0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:53:57.009 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:57.007+0000 7f2a8db5f700 1 -- 172.21.15.137:0/3098449376 >> 172.21.15.137:0/3098449376 conn(0x7f2a880f1c70 msgr2=0x7f2a880f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:53:57.010 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:57.007+0000 7f2a8db5f700 1 -- 172.21.15.137:0/3098449376 shutdown_connections 2023-11-15T01:53:57.010 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:57.007+0000 7f2a8db5f700 1 -- 172.21.15.137:0/3098449376 wait complete. 2023-11-15T01:53:57.010 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:53:58.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:53:57 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:53:56.730380+0000 mgr.smithi055.szwngw (mgr.14164) 128 : cluster [DBG] pgmap v121: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:53:58.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:53:57 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:53:57.006547+0000 mon.smithi055 (mon.0) 205 : audit [DBG] from='client.? 172.21.15.137:0/3098449376' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:53:58.342 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:53:58.342 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:53:59.466 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:59.465+0000 7ff59f218700 1 -- 172.21.15.137:0/3317846385 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff5980f51a0 msgr2=0x7ff5980f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:53:59.467 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:59.465+0000 7ff59f218700 1 --2- 172.21.15.137:0/3317846385 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff5980f51a0 0x7ff5980f55c0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7ff594009fb0 tx=0x7ff594009b30).stop 2023-11-15T01:53:59.467 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:59.465+0000 7ff59f218700 1 -- 172.21.15.137:0/3317846385 shutdown_connections 2023-11-15T01:53:59.467 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:59.465+0000 7ff59f218700 1 --2- 172.21.15.137:0/3317846385 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff5980f51a0 0x7ff5980f55c0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:53:59.467 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:59.465+0000 7ff59f218700 1 -- 172.21.15.137:0/3317846385 >> 172.21.15.137:0/3317846385 conn(0x7ff5980f09a0 msgr2=0x7ff5980f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:53:59.468 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:59.465+0000 7ff59f218700 1 -- 172.21.15.137:0/3317846385 shutdown_connections 2023-11-15T01:53:59.468 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:59.465+0000 7ff59f218700 1 -- 172.21.15.137:0/3317846385 wait complete. 2023-11-15T01:53:59.468 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:59.465+0000 7ff59f218700 1 Processor -- start 2023-11-15T01:53:59.468 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:59.465+0000 7ff59f218700 1 -- start start 2023-11-15T01:53:59.468 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:59.466+0000 7ff59f218700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff5980f51a0 0x7ff5981091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:53:59.469 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:59.466+0000 7ff59f218700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff598060240 con 0x7ff5980f51a0 2023-11-15T01:53:59.469 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:59.467+0000 7ff59e216700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff5980f51a0 0x7ff5981091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:53:59.470 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:59.467+0000 7ff59e216700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff5980f51a0 0x7ff5981091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:50326/0 (socket says 172.21.15.137:50326) 2023-11-15T01:53:59.470 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:59.467+0000 7ff59e216700 1 -- 172.21.15.137:0/2572512170 learned_addr learned my addr 172.21.15.137:0/2572512170 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:53:59.470 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:59.468+0000 7ff59e216700 1 -- 172.21.15.137:0/2572512170 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff594005040 con 0x7ff5980f51a0 2023-11-15T01:53:59.470 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:59.469+0000 7ff59e216700 1 --2- 172.21.15.137:0/2572512170 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff5980f51a0 0x7ff5981091b0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7ff594013510 tx=0x7ff59400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:53:59.471 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:59.469+0000 7ff58effd700 1 -- 172.21.15.137:0/2572512170 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff594009320 con 0x7ff5980f51a0 2023-11-15T01:53:59.471 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:59.469+0000 7ff58effd700 1 -- 172.21.15.137:0/2572512170 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff594013f00 con 0x7ff5980f51a0 2023-11-15T01:53:59.472 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:59.469+0000 7ff59f218700 1 -- 172.21.15.137:0/2572512170 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff598060440 con 0x7ff5980f51a0 2023-11-15T01:53:59.472 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:59.469+0000 7ff58effd700 1 -- 172.21.15.137:0/2572512170 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff594014dc0 con 0x7ff5980f51a0 2023-11-15T01:53:59.472 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:59.469+0000 7ff59f218700 1 -- 172.21.15.137:0/2572512170 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff598109860 con 0x7ff5980f51a0 2023-11-15T01:53:59.472 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:59.470+0000 7ff59f218700 1 -- 172.21.15.137:0/2572512170 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff598045ad0 con 0x7ff5980f51a0 2023-11-15T01:53:59.472 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:59.470+0000 7ff58effd700 1 -- 172.21.15.137:0/2572512170 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff5940095f0 con 0x7ff5980f51a0 2023-11-15T01:53:59.473 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:59.470+0000 7ff58effd700 1 --2- 172.21.15.137:0/2572512170 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7ff58402ff60 0x7ff584032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:53:59.473 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:59.470+0000 7ff58effd700 1 -- 172.21.15.137:0/2572512170 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff594041040 con 0x7ff5980f51a0 2023-11-15T01:53:59.473 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:59.471+0000 7ff59da15700 1 --2- 172.21.15.137:0/2572512170 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7ff58402ff60 0x7ff584032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:53:59.473 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:59.471+0000 7ff59da15700 1 --2- 172.21.15.137:0/2572512170 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7ff58402ff60 0x7ff584032420 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7ff5880096a0 tx=0x7ff588006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:53:59.476 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:59.475+0000 7ff58effd700 1 -- 172.21.15.137:0/2572512170 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff594014420 con 0x7ff5980f51a0 2023-11-15T01:53:59.684 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:59.683+0000 7ff59f218700 1 -- 172.21.15.137:0/2572512170 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff59810a080 con 0x7ff5980f51a0 2023-11-15T01:53:59.685 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:59.684+0000 7ff58effd700 1 -- 172.21.15.137:0/2572512170 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff594010230 con 0x7ff5980f51a0 2023-11-15T01:53:59.686 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:53:59.686 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:53:59.687 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:59.685+0000 7ff59f218700 1 -- 172.21.15.137:0/2572512170 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7ff58402ff60 msgr2=0x7ff584032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:53:59.687 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:59.685+0000 7ff59f218700 1 --2- 172.21.15.137:0/2572512170 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7ff58402ff60 0x7ff584032420 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7ff5880096a0 tx=0x7ff588006b40).stop 2023-11-15T01:53:59.688 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:59.686+0000 7ff59f218700 1 -- 172.21.15.137:0/2572512170 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff5980f51a0 msgr2=0x7ff5981091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:53:59.688 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:59.686+0000 7ff59f218700 1 --2- 172.21.15.137:0/2572512170 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff5980f51a0 0x7ff5981091b0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7ff594013510 tx=0x7ff59400b040).stop 2023-11-15T01:53:59.688 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:59.686+0000 7ff59f218700 1 -- 172.21.15.137:0/2572512170 shutdown_connections 2023-11-15T01:53:59.688 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:59.686+0000 7ff59f218700 1 --2- 172.21.15.137:0/2572512170 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7ff58402ff60 0x7ff584032420 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:53:59.689 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:59.686+0000 7ff59f218700 1 --2- 172.21.15.137:0/2572512170 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff5980f51a0 0x7ff5981091b0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:53:59.689 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:59.686+0000 7ff59f218700 1 -- 172.21.15.137:0/2572512170 >> 172.21.15.137:0/2572512170 conn(0x7ff5980f09a0 msgr2=0x7ff5980f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:53:59.689 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:59.686+0000 7ff59f218700 1 -- 172.21.15.137:0/2572512170 shutdown_connections 2023-11-15T01:53:59.689 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:53:59.686+0000 7ff59f218700 1 -- 172.21.15.137:0/2572512170 wait complete. 2023-11-15T01:53:59.690 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:54:00.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:53:59 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:53:58.730760+0000 mgr.smithi055.szwngw (mgr.14164) 129 : cluster [DBG] pgmap v122: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:54:00.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:53:59 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:53:59.684967+0000 mon.smithi055 (mon.0) 206 : audit [DBG] from='client.? 172.21.15.137:0/2572512170' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:54:01.074 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:54:01.075 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:54:01.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:54:01 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:54:00.731009+0000 mgr.smithi055.szwngw (mgr.14164) 130 : cluster [DBG] pgmap v123: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:54:02.245 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:02.244+0000 7f3e3d916700 1 -- 172.21.15.137:0/1254073519 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3e380f4b80 msgr2=0x7f3e380f4fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:54:02.246 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:02.244+0000 7f3e3d916700 1 --2- 172.21.15.137:0/1254073519 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3e380f4b80 0x7f3e380f4fa0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7f3e2c009fb0 tx=0x7f3e2c009b30).stop 2023-11-15T01:54:02.246 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:02.244+0000 7f3e3d916700 1 -- 172.21.15.137:0/1254073519 shutdown_connections 2023-11-15T01:54:02.246 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:02.244+0000 7f3e3d916700 1 --2- 172.21.15.137:0/1254073519 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3e380f4b80 0x7f3e380f4fa0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:54:02.246 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:02.244+0000 7f3e3d916700 1 -- 172.21.15.137:0/1254073519 >> 172.21.15.137:0/1254073519 conn(0x7f3e380f02e0 msgr2=0x7f3e380f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:54:02.247 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:02.244+0000 7f3e3d916700 1 -- 172.21.15.137:0/1254073519 shutdown_connections 2023-11-15T01:54:02.247 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:02.244+0000 7f3e3d916700 1 -- 172.21.15.137:0/1254073519 wait complete. 2023-11-15T01:54:02.247 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:02.245+0000 7f3e3d916700 1 Processor -- start 2023-11-15T01:54:02.247 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:02.245+0000 7f3e3d916700 1 -- start start 2023-11-15T01:54:02.247 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:02.245+0000 7f3e3d916700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3e380f4b80 0x7f3e380fcb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:54:02.248 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:02.245+0000 7f3e3d916700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e380faf90 con 0x7f3e380f4b80 2023-11-15T01:54:02.249 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:02.245+0000 7f3e36ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3e380f4b80 0x7f3e380fcb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:54:02.249 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:02.245+0000 7f3e36ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3e380f4b80 0x7f3e380fcb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:50338/0 (socket says 172.21.15.137:50338) 2023-11-15T01:54:02.249 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:02.245+0000 7f3e36ffd700 1 -- 172.21.15.137:0/1769957973 learned_addr learned my addr 172.21.15.137:0/1769957973 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:54:02.250 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:02.245+0000 7f3e36ffd700 1 -- 172.21.15.137:0/1769957973 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e2c005040 con 0x7f3e380f4b80 2023-11-15T01:54:02.250 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:02.246+0000 7f3e36ffd700 1 --2- 172.21.15.137:0/1769957973 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3e380f4b80 0x7f3e380fcb10 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f3e2c013510 tx=0x7f3e2c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:54:02.250 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:02.246+0000 7f3e1f7fe700 1 -- 172.21.15.137:0/1769957973 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3e2c009320 con 0x7f3e380f4b80 2023-11-15T01:54:02.250 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:02.246+0000 7f3e3d916700 1 -- 172.21.15.137:0/1769957973 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e380fb130 con 0x7f3e380f4b80 2023-11-15T01:54:02.251 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:02.246+0000 7f3e1f7fe700 1 -- 172.21.15.137:0/1769957973 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3e2c013f00 con 0x7f3e380f4b80 2023-11-15T01:54:02.251 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:02.246+0000 7f3e1f7fe700 1 -- 172.21.15.137:0/1769957973 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3e2c014d90 con 0x7f3e380f4b80 2023-11-15T01:54:02.251 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:02.246+0000 7f3e3d916700 1 -- 172.21.15.137:0/1769957973 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e380fb5c0 con 0x7f3e380f4b80 2023-11-15T01:54:02.251 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:02.247+0000 7f3e1f7fe700 1 -- 172.21.15.137:0/1769957973 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3e2c014420 con 0x7f3e380f4b80 2023-11-15T01:54:02.251 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:02.247+0000 7f3e3d916700 1 -- 172.21.15.137:0/1769957973 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e38059100 con 0x7f3e380f4b80 2023-11-15T01:54:02.252 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:02.247+0000 7f3e1f7fe700 1 --2- 172.21.15.137:0/1769957973 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f3e2002ff60 0x7f3e20032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:54:02.252 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:02.247+0000 7f3e1f7fe700 1 -- 172.21.15.137:0/1769957973 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3e2c040fb0 con 0x7f3e380f4b80 2023-11-15T01:54:02.252 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:02.247+0000 7f3e367fc700 1 --2- 172.21.15.137:0/1769957973 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f3e2002ff60 0x7f3e20032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:54:02.252 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:02.247+0000 7f3e367fc700 1 --2- 172.21.15.137:0/1769957973 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f3e2002ff60 0x7f3e20032420 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f3e280096a0 tx=0x7f3e28006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:54:02.253 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:02.252+0000 7f3e1f7fe700 1 -- 172.21.15.137:0/1769957973 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3e2c0095f0 con 0x7f3e380f4b80 2023-11-15T01:54:02.458 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:02.458+0000 7f3e3d916700 1 -- 172.21.15.137:0/1769957973 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3e38045ad0 con 0x7f3e380f4b80 2023-11-15T01:54:02.459 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:02.458+0000 7f3e1f7fe700 1 -- 172.21.15.137:0/1769957973 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3e2c01db70 con 0x7f3e380f4b80 2023-11-15T01:54:02.459 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:54:02.460 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:54:02.460 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:02.460+0000 7f3e3d916700 1 -- 172.21.15.137:0/1769957973 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f3e2002ff60 msgr2=0x7f3e20032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:54:02.461 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:02.460+0000 7f3e3d916700 1 --2- 172.21.15.137:0/1769957973 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f3e2002ff60 0x7f3e20032420 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f3e280096a0 tx=0x7f3e28006b40).stop 2023-11-15T01:54:02.461 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:02.460+0000 7f3e3d916700 1 -- 172.21.15.137:0/1769957973 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3e380f4b80 msgr2=0x7f3e380fcb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:54:02.461 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:02.460+0000 7f3e3d916700 1 --2- 172.21.15.137:0/1769957973 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3e380f4b80 0x7f3e380fcb10 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f3e2c013510 tx=0x7f3e2c00b040).stop 2023-11-15T01:54:02.461 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:02.460+0000 7f3e3d916700 1 -- 172.21.15.137:0/1769957973 shutdown_connections 2023-11-15T01:54:02.461 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:02.460+0000 7f3e3d916700 1 --2- 172.21.15.137:0/1769957973 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f3e2002ff60 0x7f3e20032420 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:54:02.462 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:02.460+0000 7f3e3d916700 1 --2- 172.21.15.137:0/1769957973 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3e380f4b80 0x7f3e380fcb10 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:54:02.462 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:02.460+0000 7f3e3d916700 1 -- 172.21.15.137:0/1769957973 >> 172.21.15.137:0/1769957973 conn(0x7f3e380f02e0 msgr2=0x7f3e380f1500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:54:02.462 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:02.460+0000 7f3e3d916700 1 -- 172.21.15.137:0/1769957973 shutdown_connections 2023-11-15T01:54:02.462 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:02.460+0000 7f3e3d916700 1 -- 172.21.15.137:0/1769957973 wait complete. 2023-11-15T01:54:02.463 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:54:02.829 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:54:02 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:54:02.459360+0000 mon.smithi055 (mon.0) 207 : audit [DBG] from='client.? 172.21.15.137:0/1769957973' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:54:03.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:54:03 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:54:02.731349+0000 mgr.smithi055.szwngw (mgr.14164) 131 : cluster [DBG] pgmap v124: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:54:03.852 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:54:03.853 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:54:05.015 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:05.012+0000 7f880ec5f700 1 -- 172.21.15.137:0/4011193478 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f88080f6480 msgr2=0x7f88080f68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:54:05.015 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:05.012+0000 7f880ec5f700 1 --2- 172.21.15.137:0/4011193478 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f88080f6480 0x7f88080f68a0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7f87fc009fb0 tx=0x7f87fc009b30).stop 2023-11-15T01:54:05.015 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:05.012+0000 7f880ec5f700 1 -- 172.21.15.137:0/4011193478 shutdown_connections 2023-11-15T01:54:05.016 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:05.012+0000 7f880ec5f700 1 --2- 172.21.15.137:0/4011193478 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f88080f6480 0x7f88080f68a0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:54:05.016 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:05.012+0000 7f880ec5f700 1 -- 172.21.15.137:0/4011193478 >> 172.21.15.137:0/4011193478 conn(0x7f88080f1c00 msgr2=0x7f88080f4020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:54:05.016 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:05.012+0000 7f880ec5f700 1 -- 172.21.15.137:0/4011193478 shutdown_connections 2023-11-15T01:54:05.016 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:05.012+0000 7f880ec5f700 1 -- 172.21.15.137:0/4011193478 wait complete. 2023-11-15T01:54:05.017 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:05.013+0000 7f880ec5f700 1 Processor -- start 2023-11-15T01:54:05.017 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:05.013+0000 7f880ec5f700 1 -- start start 2023-11-15T01:54:05.017 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:05.013+0000 7f880ec5f700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f88080f6480 0x7f88081066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:54:05.017 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:05.013+0000 7f880ec5f700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8808106bc0 con 0x7f88080f6480 2023-11-15T01:54:05.019 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:05.014+0000 7f880c9fb700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f88080f6480 0x7f88081066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:54:05.019 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:05.014+0000 7f880c9fb700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f88080f6480 0x7f88081066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:50350/0 (socket says 172.21.15.137:50350) 2023-11-15T01:54:05.020 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:05.014+0000 7f880c9fb700 1 -- 172.21.15.137:0/299211397 learned_addr learned my addr 172.21.15.137:0/299211397 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:54:05.020 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:05.014+0000 7f880c9fb700 1 -- 172.21.15.137:0/299211397 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87fc005040 con 0x7f88080f6480 2023-11-15T01:54:05.020 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:05.014+0000 7f880c9fb700 1 --2- 172.21.15.137:0/299211397 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f88080f6480 0x7f88081066d0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f87fc014000 tx=0x7f87fc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:54:05.021 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:05.014+0000 7f88057fa700 1 -- 172.21.15.137:0/299211397 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f87fc0095f0 con 0x7f88080f6480 2023-11-15T01:54:05.021 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:05.014+0000 7f880ec5f700 1 -- 172.21.15.137:0/299211397 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8808106dc0 con 0x7f88080f6480 2023-11-15T01:54:05.021 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:05.014+0000 7f88057fa700 1 -- 172.21.15.137:0/299211397 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f87fc007e70 con 0x7f88080f6480 2023-11-15T01:54:05.021 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:05.014+0000 7f88057fa700 1 -- 172.21.15.137:0/299211397 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f87fc014ea0 con 0x7f88080f6480 2023-11-15T01:54:05.021 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:05.014+0000 7f880ec5f700 1 -- 172.21.15.137:0/299211397 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8808107250 con 0x7f88080f6480 2023-11-15T01:54:05.022 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:05.015+0000 7f880ec5f700 1 -- 172.21.15.137:0/299211397 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8808045ad0 con 0x7f88080f6480 2023-11-15T01:54:05.022 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:05.015+0000 7f88057fa700 1 -- 172.21.15.137:0/299211397 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f87fc025070 con 0x7f88080f6480 2023-11-15T01:54:05.022 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:05.016+0000 7f88057fa700 1 --2- 172.21.15.137:0/299211397 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f87f802ff60 0x7f87f8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:54:05.022 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:05.016+0000 7f88057fa700 1 -- 172.21.15.137:0/299211397 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f87fc040a30 con 0x7f88080f6480 2023-11-15T01:54:05.023 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:05.016+0000 7f8807fff700 1 --2- 172.21.15.137:0/299211397 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f87f802ff60 0x7f87f8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:54:05.023 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:05.016+0000 7f8807fff700 1 --2- 172.21.15.137:0/299211397 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f87f802ff60 0x7f87f8032420 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f87f40096a0 tx=0x7f87f4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:54:05.023 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:05.021+0000 7f88057fa700 1 -- 172.21.15.137:0/299211397 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f87fc0079c0 con 0x7f88080f6480 2023-11-15T01:54:05.230 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:05.229+0000 7f880ec5f700 1 -- 172.21.15.137:0/299211397 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8808108100 con 0x7f88080f6480 2023-11-15T01:54:05.231 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:05.230+0000 7f88057fa700 1 -- 172.21.15.137:0/299211397 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f87fc00f300 con 0x7f88080f6480 2023-11-15T01:54:05.232 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:54:05.232 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:54:05.233 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:05.231+0000 7f880ec5f700 1 -- 172.21.15.137:0/299211397 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f87f802ff60 msgr2=0x7f87f8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:54:05.233 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:05.231+0000 7f880ec5f700 1 --2- 172.21.15.137:0/299211397 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f87f802ff60 0x7f87f8032420 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f87f40096a0 tx=0x7f87f4006b40).stop 2023-11-15T01:54:05.233 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:05.231+0000 7f880ec5f700 1 -- 172.21.15.137:0/299211397 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f88080f6480 msgr2=0x7f88081066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:54:05.233 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:05.231+0000 7f880ec5f700 1 --2- 172.21.15.137:0/299211397 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f88080f6480 0x7f88081066d0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f87fc014000 tx=0x7f87fc00b040).stop 2023-11-15T01:54:05.234 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:05.231+0000 7f880ec5f700 1 -- 172.21.15.137:0/299211397 shutdown_connections 2023-11-15T01:54:05.234 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:05.231+0000 7f880ec5f700 1 --2- 172.21.15.137:0/299211397 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f87f802ff60 0x7f87f8032420 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:54:05.234 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:05.231+0000 7f880ec5f700 1 --2- 172.21.15.137:0/299211397 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f88080f6480 0x7f88081066d0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:54:05.234 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:05.231+0000 7f880ec5f700 1 -- 172.21.15.137:0/299211397 >> 172.21.15.137:0/299211397 conn(0x7f88080f1c00 msgr2=0x7f88080f2720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:54:05.234 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:05.231+0000 7f880ec5f700 1 -- 172.21.15.137:0/299211397 shutdown_connections 2023-11-15T01:54:05.235 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:05.231+0000 7f880ec5f700 1 -- 172.21.15.137:0/299211397 wait complete. 2023-11-15T01:54:05.235 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:54:06.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:54:05 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:54:04.731692+0000 mgr.smithi055.szwngw (mgr.14164) 132 : cluster [DBG] pgmap v125: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:54:06.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:54:05 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:54:05.231630+0000 mon.smithi055 (mon.0) 208 : audit [DBG] from='client.? 172.21.15.137:0/299211397' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:54:06.616 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:54:06.617 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:54:07.767 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:07.764+0000 7fc0a39f5700 1 -- 172.21.15.137:0/2241306609 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc09c0f5410 msgr2=0x7fc09c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:54:07.767 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:07.764+0000 7fc0a39f5700 1 --2- 172.21.15.137:0/2241306609 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc09c0f5410 0x7fc09c0f5830 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7fc08c009fb0 tx=0x7fc08c009b30).stop 2023-11-15T01:54:07.767 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:07.764+0000 7fc0a39f5700 1 -- 172.21.15.137:0/2241306609 shutdown_connections 2023-11-15T01:54:07.767 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:07.764+0000 7fc0a39f5700 1 --2- 172.21.15.137:0/2241306609 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc09c0f5410 0x7fc09c0f5830 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:54:07.768 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:07.764+0000 7fc0a39f5700 1 -- 172.21.15.137:0/2241306609 >> 172.21.15.137:0/2241306609 conn(0x7fc09c0f0b70 msgr2=0x7fc09c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:54:07.768 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:07.764+0000 7fc0a39f5700 1 -- 172.21.15.137:0/2241306609 shutdown_connections 2023-11-15T01:54:07.768 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:07.764+0000 7fc0a39f5700 1 -- 172.21.15.137:0/2241306609 wait complete. 2023-11-15T01:54:07.768 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:07.764+0000 7fc0a39f5700 1 Processor -- start 2023-11-15T01:54:07.768 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:07.764+0000 7fc0a39f5700 1 -- start start 2023-11-15T01:54:07.769 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:07.764+0000 7fc0a39f5700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc09c0f5410 0x7fc09c1093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:54:07.769 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:07.764+0000 7fc0a39f5700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc09c060280 con 0x7fc09c0f5410 2023-11-15T01:54:07.770 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:07.766+0000 7fc0a1791700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc09c0f5410 0x7fc09c1093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:54:07.770 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:07.766+0000 7fc0a1791700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc09c0f5410 0x7fc09c1093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:50368/0 (socket says 172.21.15.137:50368) 2023-11-15T01:54:07.770 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:07.766+0000 7fc0a1791700 1 -- 172.21.15.137:0/208783795 learned_addr learned my addr 172.21.15.137:0/208783795 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:54:07.770 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:07.768+0000 7fc0a1791700 1 -- 172.21.15.137:0/208783795 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc08c005040 con 0x7fc09c0f5410 2023-11-15T01:54:07.771 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:07.768+0000 7fc0a1791700 1 --2- 172.21.15.137:0/208783795 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc09c0f5410 0x7fc09c1093f0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7fc08c014000 tx=0x7fc08c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:54:07.771 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:07.768+0000 7fc091ffb700 1 -- 172.21.15.137:0/208783795 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc08c0095f0 con 0x7fc09c0f5410 2023-11-15T01:54:07.771 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:07.768+0000 7fc0a39f5700 1 -- 172.21.15.137:0/208783795 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc09c060480 con 0x7fc09c0f5410 2023-11-15T01:54:07.771 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:07.768+0000 7fc091ffb700 1 -- 172.21.15.137:0/208783795 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc08c007e70 con 0x7fc09c0f5410 2023-11-15T01:54:07.771 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:07.768+0000 7fc091ffb700 1 -- 172.21.15.137:0/208783795 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc08c014ea0 con 0x7fc09c0f5410 2023-11-15T01:54:07.772 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:07.769+0000 7fc0a39f5700 1 -- 172.21.15.137:0/208783795 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc09c109b20 con 0x7fc09c0f5410 2023-11-15T01:54:07.773 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:07.770+0000 7fc0a39f5700 1 -- 172.21.15.137:0/208783795 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc09c045ad0 con 0x7fc09c0f5410 2023-11-15T01:54:07.773 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:07.770+0000 7fc091ffb700 1 -- 172.21.15.137:0/208783795 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc08c025070 con 0x7fc09c0f5410 2023-11-15T01:54:07.773 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:07.770+0000 7fc091ffb700 1 --2- 172.21.15.137:0/208783795 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fc08802ffb0 0x7fc088032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:54:07.773 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:07.770+0000 7fc091ffb700 1 -- 172.21.15.137:0/208783795 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc08c040900 con 0x7fc09c0f5410 2023-11-15T01:54:07.773 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:07.770+0000 7fc0a0f90700 1 --2- 172.21.15.137:0/208783795 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fc08802ffb0 0x7fc088032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:54:07.774 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:07.771+0000 7fc0a0f90700 1 --2- 172.21.15.137:0/208783795 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fc08802ffb0 0x7fc088032470 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7fc0980096a0 tx=0x7fc098006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:54:07.776 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:07.775+0000 7fc091ffb700 1 -- 172.21.15.137:0/208783795 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc08c0079c0 con 0x7fc09c0f5410 2023-11-15T01:54:07.982 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:07.980+0000 7fc0a39f5700 1 -- 172.21.15.137:0/208783795 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc09c10a340 con 0x7fc09c0f5410 2023-11-15T01:54:07.983 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:07.982+0000 7fc091ffb700 1 -- 172.21.15.137:0/208783795 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc08c00f300 con 0x7fc09c0f5410 2023-11-15T01:54:07.983 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:54:07.983 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:54:07.984 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:07.982+0000 7fc0a39f5700 1 -- 172.21.15.137:0/208783795 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fc08802ffb0 msgr2=0x7fc088032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:54:07.985 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:07.982+0000 7fc0a39f5700 1 --2- 172.21.15.137:0/208783795 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fc08802ffb0 0x7fc088032470 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7fc0980096a0 tx=0x7fc098006b40).stop 2023-11-15T01:54:07.985 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:07.982+0000 7fc0a39f5700 1 -- 172.21.15.137:0/208783795 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc09c0f5410 msgr2=0x7fc09c1093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:54:07.985 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:07.982+0000 7fc0a39f5700 1 --2- 172.21.15.137:0/208783795 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc09c0f5410 0x7fc09c1093f0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7fc08c014000 tx=0x7fc08c00b040).stop 2023-11-15T01:54:07.985 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:07.982+0000 7fc0a39f5700 1 -- 172.21.15.137:0/208783795 shutdown_connections 2023-11-15T01:54:07.986 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:07.982+0000 7fc0a39f5700 1 --2- 172.21.15.137:0/208783795 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fc08802ffb0 0x7fc088032470 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:54:07.986 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:07.982+0000 7fc0a39f5700 1 --2- 172.21.15.137:0/208783795 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc09c0f5410 0x7fc09c1093f0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:54:07.986 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:07.982+0000 7fc0a39f5700 1 -- 172.21.15.137:0/208783795 >> 172.21.15.137:0/208783795 conn(0x7fc09c0f0b70 msgr2=0x7fc09c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:54:07.986 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:07.982+0000 7fc0a39f5700 1 -- 172.21.15.137:0/208783795 shutdown_connections 2023-11-15T01:54:07.986 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:07.982+0000 7fc0a39f5700 1 -- 172.21.15.137:0/208783795 wait complete. 2023-11-15T01:54:07.987 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:54:08.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:54:07 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:54:06.732029+0000 mgr.smithi055.szwngw (mgr.14164) 133 : cluster [DBG] pgmap v126: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:54:09.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:54:08 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:54:07.982962+0000 mon.smithi055 (mon.0) 209 : audit [DBG] from='client.? 172.21.15.137:0/208783795' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:54:09.377 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:54:09.378 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:54:10.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:54:09 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:54:08.732417+0000 mgr.smithi055.szwngw (mgr.14164) 134 : cluster [DBG] pgmap v127: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:54:10.532 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:10.530+0000 7fbc693ef700 1 -- 172.21.15.137:0/4184224785 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbc640f6510 msgr2=0x7fbc640f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:54:10.533 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:10.530+0000 7fbc693ef700 1 --2- 172.21.15.137:0/4184224785 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbc640f6510 0x7fbc640f6930 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7fbc54009fb0 tx=0x7fbc54009b30).stop 2023-11-15T01:54:10.533 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:10.530+0000 7fbc693ef700 1 -- 172.21.15.137:0/4184224785 shutdown_connections 2023-11-15T01:54:10.533 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:10.530+0000 7fbc693ef700 1 --2- 172.21.15.137:0/4184224785 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbc640f6510 0x7fbc640f6930 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:54:10.534 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:10.530+0000 7fbc693ef700 1 -- 172.21.15.137:0/4184224785 >> 172.21.15.137:0/4184224785 conn(0x7fbc640f1c70 msgr2=0x7fbc640f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:54:10.534 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:10.530+0000 7fbc693ef700 1 -- 172.21.15.137:0/4184224785 shutdown_connections 2023-11-15T01:54:10.534 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:10.530+0000 7fbc693ef700 1 -- 172.21.15.137:0/4184224785 wait complete. 2023-11-15T01:54:10.534 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:10.531+0000 7fbc693ef700 1 Processor -- start 2023-11-15T01:54:10.534 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:10.531+0000 7fbc693ef700 1 -- start start 2023-11-15T01:54:10.535 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:10.531+0000 7fbc693ef700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbc640f6510 0x7fbc64106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:54:10.535 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:10.531+0000 7fbc693ef700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc64106c20 con 0x7fbc640f6510 2023-11-15T01:54:10.535 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:10.531+0000 7fbc62ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbc640f6510 0x7fbc64106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:54:10.535 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:10.531+0000 7fbc62ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbc640f6510 0x7fbc64106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:56018/0 (socket says 172.21.15.137:56018) 2023-11-15T01:54:10.535 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:10.531+0000 7fbc62ffd700 1 -- 172.21.15.137:0/1587215411 learned_addr learned my addr 172.21.15.137:0/1587215411 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:54:10.537 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:10.531+0000 7fbc62ffd700 1 -- 172.21.15.137:0/1587215411 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc54005040 con 0x7fbc640f6510 2023-11-15T01:54:10.537 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:10.532+0000 7fbc62ffd700 1 --2- 172.21.15.137:0/1587215411 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbc640f6510 0x7fbc64106730 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7fbc54013510 tx=0x7fbc5400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:54:10.537 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:10.532+0000 7fbc5b7fe700 1 -- 172.21.15.137:0/1587215411 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbc54009320 con 0x7fbc640f6510 2023-11-15T01:54:10.538 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:10.532+0000 7fbc693ef700 1 -- 172.21.15.137:0/1587215411 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc64106e20 con 0x7fbc640f6510 2023-11-15T01:54:10.538 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:10.532+0000 7fbc5b7fe700 1 -- 172.21.15.137:0/1587215411 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbc54013f00 con 0x7fbc640f6510 2023-11-15T01:54:10.538 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:10.532+0000 7fbc5b7fe700 1 -- 172.21.15.137:0/1587215411 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbc54014d90 con 0x7fbc640f6510 2023-11-15T01:54:10.538 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:10.532+0000 7fbc693ef700 1 -- 172.21.15.137:0/1587215411 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc641072b0 con 0x7fbc640f6510 2023-11-15T01:54:10.539 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:10.532+0000 7fbc5b7fe700 1 -- 172.21.15.137:0/1587215411 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbc540095f0 con 0x7fbc640f6510 2023-11-15T01:54:10.539 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:10.532+0000 7fbc693ef700 1 -- 172.21.15.137:0/1587215411 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc64045ad0 con 0x7fbc640f6510 2023-11-15T01:54:10.539 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:10.533+0000 7fbc5b7fe700 1 --2- 172.21.15.137:0/1587215411 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fbc5002ff60 0x7fbc50032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:54:10.539 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:10.533+0000 7fbc5b7fe700 1 -- 172.21.15.137:0/1587215411 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbc54040fe0 con 0x7fbc640f6510 2023-11-15T01:54:10.539 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:10.533+0000 7fbc627fc700 1 --2- 172.21.15.137:0/1587215411 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fbc5002ff60 0x7fbc50032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:54:10.540 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:10.533+0000 7fbc627fc700 1 --2- 172.21.15.137:0/1587215411 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fbc5002ff60 0x7fbc50032420 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7fbc4c0096a0 tx=0x7fbc4c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:54:10.540 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:10.538+0000 7fbc5b7fe700 1 -- 172.21.15.137:0/1587215411 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbc5401d7f0 con 0x7fbc640f6510 2023-11-15T01:54:10.746 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:10.745+0000 7fbc693ef700 1 -- 172.21.15.137:0/1587215411 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbc640fbf60 con 0x7fbc640f6510 2023-11-15T01:54:10.747 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:10.745+0000 7fbc5b7fe700 1 -- 172.21.15.137:0/1587215411 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbc54010230 con 0x7fbc640f6510 2023-11-15T01:54:10.748 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:54:10.749 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:54:10.750 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:10.747+0000 7fbc693ef700 1 -- 172.21.15.137:0/1587215411 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fbc5002ff60 msgr2=0x7fbc50032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:54:10.750 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:10.747+0000 7fbc693ef700 1 --2- 172.21.15.137:0/1587215411 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fbc5002ff60 0x7fbc50032420 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7fbc4c0096a0 tx=0x7fbc4c006b40).stop 2023-11-15T01:54:10.751 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:10.747+0000 7fbc693ef700 1 -- 172.21.15.137:0/1587215411 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbc640f6510 msgr2=0x7fbc64106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:54:10.751 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:10.747+0000 7fbc693ef700 1 --2- 172.21.15.137:0/1587215411 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbc640f6510 0x7fbc64106730 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7fbc54013510 tx=0x7fbc5400b040).stop 2023-11-15T01:54:10.752 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:10.747+0000 7fbc693ef700 1 -- 172.21.15.137:0/1587215411 shutdown_connections 2023-11-15T01:54:10.752 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:10.747+0000 7fbc693ef700 1 --2- 172.21.15.137:0/1587215411 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fbc5002ff60 0x7fbc50032420 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:54:10.752 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:10.747+0000 7fbc693ef700 1 --2- 172.21.15.137:0/1587215411 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbc640f6510 0x7fbc64106730 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:54:10.753 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:10.747+0000 7fbc693ef700 1 -- 172.21.15.137:0/1587215411 >> 172.21.15.137:0/1587215411 conn(0x7fbc640f1c70 msgr2=0x7fbc640f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:54:10.753 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:10.747+0000 7fbc693ef700 1 -- 172.21.15.137:0/1587215411 shutdown_connections 2023-11-15T01:54:10.753 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:10.747+0000 7fbc693ef700 1 -- 172.21.15.137:0/1587215411 wait complete. 2023-11-15T01:54:10.754 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:54:11.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:54:11 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:54:10.732802+0000 mgr.smithi055.szwngw (mgr.14164) 135 : cluster [DBG] pgmap v128: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:54:11.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:54:11 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:54:10.747526+0000 mon.smithi055 (mon.0) 210 : audit [DBG] from='client.? 172.21.15.137:0/1587215411' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:54:12.139 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:54:12.140 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:54:13.285 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:13.283+0000 7f96bd296700 1 -- 172.21.15.137:0/1410471980 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f96b80f5410 msgr2=0x7f96b80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:54:13.286 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:13.283+0000 7f96bd296700 1 --2- 172.21.15.137:0/1410471980 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f96b80f5410 0x7f96b80f5830 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7f96a8009fb0 tx=0x7f96a8009b30).stop 2023-11-15T01:54:13.286 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:13.283+0000 7f96bd296700 1 -- 172.21.15.137:0/1410471980 shutdown_connections 2023-11-15T01:54:13.286 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:13.283+0000 7f96bd296700 1 --2- 172.21.15.137:0/1410471980 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f96b80f5410 0x7f96b80f5830 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:54:13.286 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:13.283+0000 7f96bd296700 1 -- 172.21.15.137:0/1410471980 >> 172.21.15.137:0/1410471980 conn(0x7f96b80f0b70 msgr2=0x7f96b80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:54:13.287 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:13.283+0000 7f96bd296700 1 -- 172.21.15.137:0/1410471980 shutdown_connections 2023-11-15T01:54:13.287 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:13.283+0000 7f96bd296700 1 -- 172.21.15.137:0/1410471980 wait complete. 2023-11-15T01:54:13.289 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:13.284+0000 7f96bd296700 1 Processor -- start 2023-11-15T01:54:13.289 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:13.284+0000 7f96bd296700 1 -- start start 2023-11-15T01:54:13.289 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:13.284+0000 7f96bd296700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f96b80f5410 0x7f96b81066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:54:13.289 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:13.284+0000 7f96bd296700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96b8106bd0 con 0x7f96b80f5410 2023-11-15T01:54:13.290 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:13.284+0000 7f96b6ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f96b80f5410 0x7f96b81066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:54:13.290 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:13.284+0000 7f96b6ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f96b80f5410 0x7f96b81066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:56036/0 (socket says 172.21.15.137:56036) 2023-11-15T01:54:13.290 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:13.284+0000 7f96b6ffd700 1 -- 172.21.15.137:0/202977076 learned_addr learned my addr 172.21.15.137:0/202977076 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:54:13.290 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:13.285+0000 7f96b6ffd700 1 -- 172.21.15.137:0/202977076 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96a8005040 con 0x7f96b80f5410 2023-11-15T01:54:13.291 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:13.285+0000 7f96b6ffd700 1 --2- 172.21.15.137:0/202977076 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f96b80f5410 0x7f96b81066e0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f96a8013510 tx=0x7f96a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:54:13.291 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:13.285+0000 7f96af7fe700 1 -- 172.21.15.137:0/202977076 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f96a8009320 con 0x7f96b80f5410 2023-11-15T01:54:13.291 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:13.285+0000 7f96bd296700 1 -- 172.21.15.137:0/202977076 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96b8106dd0 con 0x7f96b80f5410 2023-11-15T01:54:13.291 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:13.285+0000 7f96af7fe700 1 -- 172.21.15.137:0/202977076 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f96a8013f00 con 0x7f96b80f5410 2023-11-15T01:54:13.291 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:13.285+0000 7f96af7fe700 1 -- 172.21.15.137:0/202977076 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f96a8014d90 con 0x7f96b80f5410 2023-11-15T01:54:13.292 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:13.285+0000 7f96bd296700 1 -- 172.21.15.137:0/202977076 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96b8107260 con 0x7f96b80f5410 2023-11-15T01:54:13.292 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:13.286+0000 7f96bd296700 1 -- 172.21.15.137:0/202977076 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96b8045ad0 con 0x7f96b80f5410 2023-11-15T01:54:13.292 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:13.286+0000 7f96af7fe700 1 -- 172.21.15.137:0/202977076 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f96a80095f0 con 0x7f96b80f5410 2023-11-15T01:54:13.292 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:13.286+0000 7f96af7fe700 1 --2- 172.21.15.137:0/202977076 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f96a402ff60 0x7f96a4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:54:13.292 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:13.286+0000 7f96af7fe700 1 -- 172.21.15.137:0/202977076 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f96a8041000 con 0x7f96b80f5410 2023-11-15T01:54:13.293 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:13.286+0000 7f96b67fc700 1 --2- 172.21.15.137:0/202977076 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f96a402ff60 0x7f96a4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:54:13.293 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:13.287+0000 7f96b67fc700 1 --2- 172.21.15.137:0/202977076 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f96a402ff60 0x7f96a4032420 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f96a00096a0 tx=0x7f96a0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:54:13.293 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:13.292+0000 7f96af7fe700 1 -- 172.21.15.137:0/202977076 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f96a8014420 con 0x7f96b80f5410 2023-11-15T01:54:13.496 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:13.495+0000 7f96bd296700 1 -- 172.21.15.137:0/202977076 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f96b80fae60 con 0x7f96b80f5410 2023-11-15T01:54:13.497 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:13.495+0000 7f96af7fe700 1 -- 172.21.15.137:0/202977076 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f96a801d7f0 con 0x7f96b80f5410 2023-11-15T01:54:13.497 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:54:13.498 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:54:13.499 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:13.497+0000 7f96bd296700 1 -- 172.21.15.137:0/202977076 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f96a402ff60 msgr2=0x7f96a4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:54:13.499 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:13.497+0000 7f96bd296700 1 --2- 172.21.15.137:0/202977076 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f96a402ff60 0x7f96a4032420 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f96a00096a0 tx=0x7f96a0006b40).stop 2023-11-15T01:54:13.499 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:13.497+0000 7f96bd296700 1 -- 172.21.15.137:0/202977076 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f96b80f5410 msgr2=0x7f96b81066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:54:13.500 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:13.497+0000 7f96bd296700 1 --2- 172.21.15.137:0/202977076 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f96b80f5410 0x7f96b81066e0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f96a8013510 tx=0x7f96a800b040).stop 2023-11-15T01:54:13.500 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:13.497+0000 7f96bd296700 1 -- 172.21.15.137:0/202977076 shutdown_connections 2023-11-15T01:54:13.500 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:13.497+0000 7f96bd296700 1 --2- 172.21.15.137:0/202977076 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f96a402ff60 0x7f96a4032420 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:54:13.500 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:13.497+0000 7f96bd296700 1 --2- 172.21.15.137:0/202977076 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f96b80f5410 0x7f96b81066e0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:54:13.500 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:13.497+0000 7f96bd296700 1 -- 172.21.15.137:0/202977076 >> 172.21.15.137:0/202977076 conn(0x7f96b80f0b70 msgr2=0x7f96b805fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:54:13.501 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:13.497+0000 7f96bd296700 1 -- 172.21.15.137:0/202977076 shutdown_connections 2023-11-15T01:54:13.501 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:13.497+0000 7f96bd296700 1 -- 172.21.15.137:0/202977076 wait complete. 2023-11-15T01:54:13.501 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:54:14.079 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:54:13 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:54:12.733178+0000 mgr.smithi055.szwngw (mgr.14164) 136 : cluster [DBG] pgmap v129: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:54:14.079 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:54:13 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:54:13.497458+0000 mon.smithi055 (mon.0) 211 : audit [DBG] from='client.? 172.21.15.137:0/202977076' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:54:14.875 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:54:14.876 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:54:16.048 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:16.045+0000 7f90bd7a1700 1 -- 172.21.15.137:0/2312780027 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f90b80f62a0 msgr2=0x7f90b80f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:54:16.048 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:16.045+0000 7f90bd7a1700 1 --2- 172.21.15.137:0/2312780027 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f90b80f62a0 0x7f90b80f66c0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7f90a8009fb0 tx=0x7f90a8009b30).stop 2023-11-15T01:54:16.049 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:16.045+0000 7f90bd7a1700 1 -- 172.21.15.137:0/2312780027 shutdown_connections 2023-11-15T01:54:16.049 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:16.045+0000 7f90bd7a1700 1 --2- 172.21.15.137:0/2312780027 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f90b80f62a0 0x7f90b80f66c0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:54:16.049 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:16.045+0000 7f90bd7a1700 1 -- 172.21.15.137:0/2312780027 >> 172.21.15.137:0/2312780027 conn(0x7f90b80f1aa0 msgr2=0x7f90b80f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:54:16.049 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:16.046+0000 7f90bd7a1700 1 -- 172.21.15.137:0/2312780027 shutdown_connections 2023-11-15T01:54:16.050 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:16.046+0000 7f90bd7a1700 1 -- 172.21.15.137:0/2312780027 wait complete. 2023-11-15T01:54:16.050 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:16.046+0000 7f90bd7a1700 1 Processor -- start 2023-11-15T01:54:16.050 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:16.046+0000 7f90bd7a1700 1 -- start start 2023-11-15T01:54:16.050 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:16.047+0000 7f90bd7a1700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f90b80f62a0 0x7f90b81064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:54:16.050 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:16.047+0000 7f90bd7a1700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90b81069a0 con 0x7f90b80f62a0 2023-11-15T01:54:16.051 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:16.047+0000 7f90b7fff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f90b80f62a0 0x7f90b81064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:54:16.051 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:16.047+0000 7f90b7fff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f90b80f62a0 0x7f90b81064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:56054/0 (socket says 172.21.15.137:56054) 2023-11-15T01:54:16.051 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:16.047+0000 7f90b7fff700 1 -- 172.21.15.137:0/2434157291 learned_addr learned my addr 172.21.15.137:0/2434157291 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:54:16.051 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:16.047+0000 7f90b7fff700 1 -- 172.21.15.137:0/2434157291 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90a8005040 con 0x7f90b80f62a0 2023-11-15T01:54:16.051 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:16.047+0000 7f90b7fff700 1 --2- 172.21.15.137:0/2434157291 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f90b80f62a0 0x7f90b81064b0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7f90a8009830 tx=0x7f90a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:54:16.052 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:16.047+0000 7f90b4ff9700 1 -- 172.21.15.137:0/2434157291 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f90a8009320 con 0x7f90b80f62a0 2023-11-15T01:54:16.052 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:16.048+0000 7f90bd7a1700 1 -- 172.21.15.137:0/2434157291 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90b8106ba0 con 0x7f90b80f62a0 2023-11-15T01:54:16.052 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:16.048+0000 7f90b4ff9700 1 -- 172.21.15.137:0/2434157291 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f90a80136c0 con 0x7f90b80f62a0 2023-11-15T01:54:16.052 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:16.048+0000 7f90b4ff9700 1 -- 172.21.15.137:0/2434157291 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f90a801ee90 con 0x7f90b80f62a0 2023-11-15T01:54:16.052 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:16.048+0000 7f90bd7a1700 1 -- 172.21.15.137:0/2434157291 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90b8107030 con 0x7f90b80f62a0 2023-11-15T01:54:16.053 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:16.048+0000 7f90b4ff9700 1 -- 172.21.15.137:0/2434157291 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f90a801e420 con 0x7f90b80f62a0 2023-11-15T01:54:16.054 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:16.048+0000 7f90bd7a1700 1 -- 172.21.15.137:0/2434157291 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90b8045ad0 con 0x7f90b80f62a0 2023-11-15T01:54:16.054 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:16.048+0000 7f90b4ff9700 1 --2- 172.21.15.137:0/2434157291 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f90a002ff10 0x7f90a00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:54:16.054 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:16.049+0000 7f90b4ff9700 1 -- 172.21.15.137:0/2434157291 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f90a8041010 con 0x7f90b80f62a0 2023-11-15T01:54:16.054 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:16.049+0000 7f90b77fe700 1 --2- 172.21.15.137:0/2434157291 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f90a002ff10 0x7f90a00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:54:16.055 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:16.049+0000 7f90b77fe700 1 --2- 172.21.15.137:0/2434157291 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f90a002ff10 0x7f90a00323d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f90ac0096a0 tx=0x7f90ac006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:54:16.055 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:16.054+0000 7f90b4ff9700 1 -- 172.21.15.137:0/2434157291 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f90a80095f0 con 0x7f90b80f62a0 2023-11-15T01:54:16.077 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:54:15 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:54:14.733494+0000 mgr.smithi055.szwngw (mgr.14164) 137 : cluster [DBG] pgmap v130: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:54:16.261 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:16.261+0000 7f90bd7a1700 1 -- 172.21.15.137:0/2434157291 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f90b8107ee0 con 0x7f90b80f62a0 2023-11-15T01:54:16.262 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:16.262+0000 7f90b4ff9700 1 -- 172.21.15.137:0/2434157291 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f90a801cb10 con 0x7f90b80f62a0 2023-11-15T01:54:16.263 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:54:16.263 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:54:16.264 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:16.263+0000 7f90bd7a1700 1 -- 172.21.15.137:0/2434157291 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f90a002ff10 msgr2=0x7f90a00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:54:16.264 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:16.263+0000 7f90bd7a1700 1 --2- 172.21.15.137:0/2434157291 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f90a002ff10 0x7f90a00323d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f90ac0096a0 tx=0x7f90ac006b40).stop 2023-11-15T01:54:16.264 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:16.263+0000 7f90bd7a1700 1 -- 172.21.15.137:0/2434157291 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f90b80f62a0 msgr2=0x7f90b81064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:54:16.265 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:16.263+0000 7f90bd7a1700 1 --2- 172.21.15.137:0/2434157291 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f90b80f62a0 0x7f90b81064b0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7f90a8009830 tx=0x7f90a800b040).stop 2023-11-15T01:54:16.265 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:16.263+0000 7f90bd7a1700 1 -- 172.21.15.137:0/2434157291 shutdown_connections 2023-11-15T01:54:16.265 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:16.263+0000 7f90bd7a1700 1 --2- 172.21.15.137:0/2434157291 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f90a002ff10 0x7f90a00323d0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:54:16.265 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:16.263+0000 7f90bd7a1700 1 --2- 172.21.15.137:0/2434157291 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f90b80f62a0 0x7f90b81064b0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:54:16.265 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:16.263+0000 7f90bd7a1700 1 -- 172.21.15.137:0/2434157291 >> 172.21.15.137:0/2434157291 conn(0x7f90b80f1aa0 msgr2=0x7f90b80f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:54:16.266 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:16.263+0000 7f90bd7a1700 1 -- 172.21.15.137:0/2434157291 shutdown_connections 2023-11-15T01:54:16.266 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:16.263+0000 7f90bd7a1700 1 -- 172.21.15.137:0/2434157291 wait complete. 2023-11-15T01:54:16.266 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:54:17.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:54:16 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:54:16.262650+0000 mon.smithi055 (mon.0) 212 : audit [DBG] from='client.? 172.21.15.137:0/2434157291' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:54:17.663 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:54:17.664 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:54:18.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:54:17 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:54:16.733852+0000 mgr.smithi055.szwngw (mgr.14164) 138 : cluster [DBG] pgmap v131: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:54:18.767 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:18.766+0000 7f7a4e831700 1 -- 172.21.15.137:0/2859947785 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7a480f62a0 msgr2=0x7f7a480f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:54:18.768 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:18.766+0000 7f7a4e831700 1 --2- 172.21.15.137:0/2859947785 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7a480f62a0 0x7f7a480f66c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7f7a44009fb0 tx=0x7f7a44009b30).stop 2023-11-15T01:54:18.768 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:18.766+0000 7f7a4e831700 1 -- 172.21.15.137:0/2859947785 shutdown_connections 2023-11-15T01:54:18.768 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:18.766+0000 7f7a4e831700 1 --2- 172.21.15.137:0/2859947785 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7a480f62a0 0x7f7a480f66c0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:54:18.768 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:18.766+0000 7f7a4e831700 1 -- 172.21.15.137:0/2859947785 >> 172.21.15.137:0/2859947785 conn(0x7f7a480f1aa0 msgr2=0x7f7a480f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:54:18.768 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:18.766+0000 7f7a4e831700 1 -- 172.21.15.137:0/2859947785 shutdown_connections 2023-11-15T01:54:18.769 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:18.766+0000 7f7a4e831700 1 -- 172.21.15.137:0/2859947785 wait complete. 2023-11-15T01:54:18.769 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:18.766+0000 7f7a4e831700 1 Processor -- start 2023-11-15T01:54:18.769 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:18.767+0000 7f7a4e831700 1 -- start start 2023-11-15T01:54:18.769 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:18.767+0000 7f7a4e831700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7a480f62a0 0x7f7a481064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:54:18.769 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:18.767+0000 7f7a4e831700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a481069a0 con 0x7f7a480f62a0 2023-11-15T01:54:18.770 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:18.769+0000 7f7a4d82f700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7a480f62a0 0x7f7a481064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:54:18.770 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:18.769+0000 7f7a4d82f700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7a480f62a0 0x7f7a481064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:55956/0 (socket says 172.21.15.137:55956) 2023-11-15T01:54:18.771 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:18.769+0000 7f7a4d82f700 1 -- 172.21.15.137:0/979706657 learned_addr learned my addr 172.21.15.137:0/979706657 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:54:18.771 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:18.771+0000 7f7a4d82f700 1 -- 172.21.15.137:0/979706657 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a44005040 con 0x7f7a480f62a0 2023-11-15T01:54:18.772 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:18.771+0000 7f7a4d82f700 1 --2- 172.21.15.137:0/979706657 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7a480f62a0 0x7f7a481064b0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f7a44013510 tx=0x7f7a4400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:54:18.773 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:18.772+0000 7f7a3e7fc700 1 -- 172.21.15.137:0/979706657 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a44009320 con 0x7f7a480f62a0 2023-11-15T01:54:18.773 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:18.772+0000 7f7a4e831700 1 -- 172.21.15.137:0/979706657 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a48106ba0 con 0x7f7a480f62a0 2023-11-15T01:54:18.773 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:18.772+0000 7f7a3e7fc700 1 -- 172.21.15.137:0/979706657 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7a44013f00 con 0x7f7a480f62a0 2023-11-15T01:54:18.773 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:18.772+0000 7f7a3e7fc700 1 -- 172.21.15.137:0/979706657 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a44014dc0 con 0x7f7a480f62a0 2023-11-15T01:54:18.774 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:18.772+0000 7f7a4e831700 1 -- 172.21.15.137:0/979706657 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a48107030 con 0x7f7a480f62a0 2023-11-15T01:54:18.775 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:18.773+0000 7f7a4e831700 1 -- 172.21.15.137:0/979706657 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a48045ad0 con 0x7f7a480f62a0 2023-11-15T01:54:18.775 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:18.773+0000 7f7a3e7fc700 1 -- 172.21.15.137:0/979706657 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7a440095f0 con 0x7f7a480f62a0 2023-11-15T01:54:18.775 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:18.773+0000 7f7a3e7fc700 1 --2- 172.21.15.137:0/979706657 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f7a3402ff60 0x7f7a34032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:54:18.775 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:18.773+0000 7f7a3e7fc700 1 -- 172.21.15.137:0/979706657 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7a44041100 con 0x7f7a480f62a0 2023-11-15T01:54:18.775 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:18.774+0000 7f7a4d02e700 1 --2- 172.21.15.137:0/979706657 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f7a3402ff60 0x7f7a34032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:54:18.776 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:18.774+0000 7f7a4d02e700 1 --2- 172.21.15.137:0/979706657 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f7a3402ff60 0x7f7a34032420 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f7a380096a0 tx=0x7f7a38006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:54:18.779 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:18.778+0000 7f7a3e7fc700 1 -- 172.21.15.137:0/979706657 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7a44014420 con 0x7f7a480f62a0 2023-11-15T01:54:18.988 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:18.988+0000 7f7a4e831700 1 -- 172.21.15.137:0/979706657 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7a48107ee0 con 0x7f7a480f62a0 2023-11-15T01:54:18.989 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:18.988+0000 7f7a3e7fc700 1 -- 172.21.15.137:0/979706657 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7a44010230 con 0x7f7a480f62a0 2023-11-15T01:54:18.990 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:54:18.990 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:54:18.991 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:18.990+0000 7f7a4e831700 1 -- 172.21.15.137:0/979706657 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f7a3402ff60 msgr2=0x7f7a34032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:54:18.991 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:18.990+0000 7f7a4e831700 1 --2- 172.21.15.137:0/979706657 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f7a3402ff60 0x7f7a34032420 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f7a380096a0 tx=0x7f7a38006b40).stop 2023-11-15T01:54:18.992 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:18.990+0000 7f7a4e831700 1 -- 172.21.15.137:0/979706657 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7a480f62a0 msgr2=0x7f7a481064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:54:18.992 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:18.990+0000 7f7a4e831700 1 --2- 172.21.15.137:0/979706657 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7a480f62a0 0x7f7a481064b0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f7a44013510 tx=0x7f7a4400b040).stop 2023-11-15T01:54:18.992 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:18.990+0000 7f7a4e831700 1 -- 172.21.15.137:0/979706657 shutdown_connections 2023-11-15T01:54:18.992 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:18.990+0000 7f7a4e831700 1 --2- 172.21.15.137:0/979706657 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f7a3402ff60 0x7f7a34032420 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:54:18.993 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:18.990+0000 7f7a4e831700 1 --2- 172.21.15.137:0/979706657 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7a480f62a0 0x7f7a481064b0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:54:18.993 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:18.990+0000 7f7a4e831700 1 -- 172.21.15.137:0/979706657 >> 172.21.15.137:0/979706657 conn(0x7f7a480f1aa0 msgr2=0x7f7a480f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:54:18.993 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:18.990+0000 7f7a4e831700 1 -- 172.21.15.137:0/979706657 shutdown_connections 2023-11-15T01:54:18.993 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:18.990+0000 7f7a4e831700 1 -- 172.21.15.137:0/979706657 wait complete. 2023-11-15T01:54:18.994 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:54:20.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:54:19 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:54:18.734133+0000 mgr.smithi055.szwngw (mgr.14164) 139 : cluster [DBG] pgmap v132: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:54:20.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:54:19 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:54:18.989600+0000 mon.smithi055 (mon.0) 213 : audit [DBG] from='client.? 172.21.15.137:0/979706657' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:54:20.371 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:54:20.372 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:54:21.526 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:21.524+0000 7f7e952ca700 1 -- 172.21.15.137:0/3941952019 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7e900f51a0 msgr2=0x7f7e900f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:54:21.526 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:21.524+0000 7f7e952ca700 1 --2- 172.21.15.137:0/3941952019 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7e900f51a0 0x7f7e900f55c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7f7e80009fb0 tx=0x7f7e80009b30).stop 2023-11-15T01:54:21.526 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:21.525+0000 7f7e952ca700 1 -- 172.21.15.137:0/3941952019 shutdown_connections 2023-11-15T01:54:21.526 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:21.525+0000 7f7e952ca700 1 --2- 172.21.15.137:0/3941952019 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7e900f51a0 0x7f7e900f55c0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:54:21.527 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:21.525+0000 7f7e952ca700 1 -- 172.21.15.137:0/3941952019 >> 172.21.15.137:0/3941952019 conn(0x7f7e900f09a0 msgr2=0x7f7e900f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:54:21.527 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:21.525+0000 7f7e952ca700 1 -- 172.21.15.137:0/3941952019 shutdown_connections 2023-11-15T01:54:21.527 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:21.525+0000 7f7e952ca700 1 -- 172.21.15.137:0/3941952019 wait complete. 2023-11-15T01:54:21.527 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:21.525+0000 7f7e952ca700 1 Processor -- start 2023-11-15T01:54:21.528 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:21.525+0000 7f7e952ca700 1 -- start start 2023-11-15T01:54:21.530 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:21.525+0000 7f7e952ca700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7e900f51a0 0x7f7e901091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:54:21.530 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:21.525+0000 7f7e952ca700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e90060240 con 0x7f7e900f51a0 2023-11-15T01:54:21.530 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:21.526+0000 7f7e8ffff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7e900f51a0 0x7f7e901091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:54:21.530 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:21.526+0000 7f7e8ffff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7e900f51a0 0x7f7e901091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:55974/0 (socket says 172.21.15.137:55974) 2023-11-15T01:54:21.531 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:21.526+0000 7f7e8ffff700 1 -- 172.21.15.137:0/653203523 learned_addr learned my addr 172.21.15.137:0/653203523 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:54:21.531 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:21.526+0000 7f7e8ffff700 1 -- 172.21.15.137:0/653203523 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e80005040 con 0x7f7e900f51a0 2023-11-15T01:54:21.531 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:21.526+0000 7f7e8ffff700 1 --2- 172.21.15.137:0/653203523 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7e900f51a0 0x7f7e901091b0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f7e80009830 tx=0x7f7e8000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:54:21.531 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:21.526+0000 7f7e8cff9700 1 -- 172.21.15.137:0/653203523 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e80009320 con 0x7f7e900f51a0 2023-11-15T01:54:21.531 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:21.527+0000 7f7e952ca700 1 -- 172.21.15.137:0/653203523 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e90060440 con 0x7f7e900f51a0 2023-11-15T01:54:21.532 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:21.527+0000 7f7e8cff9700 1 -- 172.21.15.137:0/653203523 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7e800136c0 con 0x7f7e900f51a0 2023-11-15T01:54:21.532 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:21.527+0000 7f7e8cff9700 1 -- 172.21.15.137:0/653203523 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e8001ee90 con 0x7f7e900f51a0 2023-11-15T01:54:21.532 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:21.527+0000 7f7e952ca700 1 -- 172.21.15.137:0/653203523 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e90109860 con 0x7f7e900f51a0 2023-11-15T01:54:21.532 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:21.527+0000 7f7e8cff9700 1 -- 172.21.15.137:0/653203523 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7e8001e420 con 0x7f7e900f51a0 2023-11-15T01:54:21.532 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:21.527+0000 7f7e952ca700 1 -- 172.21.15.137:0/653203523 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e90045ad0 con 0x7f7e900f51a0 2023-11-15T01:54:21.533 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:21.528+0000 7f7e8cff9700 1 --2- 172.21.15.137:0/653203523 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f7e7802ff10 0x7f7e780323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:54:21.533 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:21.528+0000 7f7e8cff9700 1 -- 172.21.15.137:0/653203523 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7e80040f30 con 0x7f7e900f51a0 2023-11-15T01:54:21.533 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:21.528+0000 7f7e8f7fe700 1 --2- 172.21.15.137:0/653203523 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f7e7802ff10 0x7f7e780323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:54:21.533 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:21.528+0000 7f7e8f7fe700 1 --2- 172.21.15.137:0/653203523 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f7e7802ff10 0x7f7e780323d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f7e840096a0 tx=0x7f7e84006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:54:21.534 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:21.533+0000 7f7e8cff9700 1 -- 172.21.15.137:0/653203523 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7e8006d2b0 con 0x7f7e900f51a0 2023-11-15T01:54:21.740 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:21.740+0000 7f7e952ca700 1 -- 172.21.15.137:0/653203523 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7e9010a080 con 0x7f7e900f51a0 2023-11-15T01:54:21.741 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:21.741+0000 7f7e8cff9700 1 -- 172.21.15.137:0/653203523 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7e8001c790 con 0x7f7e900f51a0 2023-11-15T01:54:21.741 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:54:21.742 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:54:21.743 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:21.742+0000 7f7e952ca700 1 -- 172.21.15.137:0/653203523 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f7e7802ff10 msgr2=0x7f7e780323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:54:21.743 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:21.742+0000 7f7e952ca700 1 --2- 172.21.15.137:0/653203523 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f7e7802ff10 0x7f7e780323d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f7e840096a0 tx=0x7f7e84006b40).stop 2023-11-15T01:54:21.743 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:21.742+0000 7f7e952ca700 1 -- 172.21.15.137:0/653203523 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7e900f51a0 msgr2=0x7f7e901091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:54:21.743 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:21.742+0000 7f7e952ca700 1 --2- 172.21.15.137:0/653203523 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7e900f51a0 0x7f7e901091b0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f7e80009830 tx=0x7f7e8000b040).stop 2023-11-15T01:54:21.744 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:21.742+0000 7f7e952ca700 1 -- 172.21.15.137:0/653203523 shutdown_connections 2023-11-15T01:54:21.744 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:21.742+0000 7f7e952ca700 1 --2- 172.21.15.137:0/653203523 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f7e7802ff10 0x7f7e780323d0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:54:21.744 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:21.742+0000 7f7e952ca700 1 --2- 172.21.15.137:0/653203523 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7e900f51a0 0x7f7e901091b0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:54:21.744 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:21.742+0000 7f7e952ca700 1 -- 172.21.15.137:0/653203523 >> 172.21.15.137:0/653203523 conn(0x7f7e900f09a0 msgr2=0x7f7e900f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:54:21.745 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:21.742+0000 7f7e952ca700 1 -- 172.21.15.137:0/653203523 shutdown_connections 2023-11-15T01:54:21.745 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:21.742+0000 7f7e952ca700 1 -- 172.21.15.137:0/653203523 wait complete. 2023-11-15T01:54:21.745 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:54:21.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:54:21 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:54:20.734410+0000 mgr.smithi055.szwngw (mgr.14164) 140 : cluster [DBG] pgmap v133: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:54:22.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:54:22 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:54:21.741645+0000 mon.smithi055 (mon.0) 214 : audit [DBG] from='client.? 172.21.15.137:0/653203523' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:54:23.136 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:54:23.136 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:54:23.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:54:23 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:54:22.734724+0000 mgr.smithi055.szwngw (mgr.14164) 141 : cluster [DBG] pgmap v134: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:54:24.296 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:24.295+0000 7f5cc1806700 1 -- 172.21.15.137:0/831391450 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5cbc0f5410 msgr2=0x7f5cbc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:54:24.297 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:24.295+0000 7f5cc1806700 1 --2- 172.21.15.137:0/831391450 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5cbc0f5410 0x7f5cbc0f5830 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7f5cb0009fb0 tx=0x7f5cb0009b30).stop 2023-11-15T01:54:24.297 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:24.295+0000 7f5cc1806700 1 -- 172.21.15.137:0/831391450 shutdown_connections 2023-11-15T01:54:24.297 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:24.295+0000 7f5cc1806700 1 --2- 172.21.15.137:0/831391450 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5cbc0f5410 0x7f5cbc0f5830 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:54:24.298 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:24.295+0000 7f5cc1806700 1 -- 172.21.15.137:0/831391450 >> 172.21.15.137:0/831391450 conn(0x7f5cbc0f0b70 msgr2=0x7f5cbc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:54:24.298 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:24.295+0000 7f5cc1806700 1 -- 172.21.15.137:0/831391450 shutdown_connections 2023-11-15T01:54:24.298 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:24.295+0000 7f5cc1806700 1 -- 172.21.15.137:0/831391450 wait complete. 2023-11-15T01:54:24.298 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:24.296+0000 7f5cc1806700 1 Processor -- start 2023-11-15T01:54:24.299 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:24.296+0000 7f5cc1806700 1 -- start start 2023-11-15T01:54:24.299 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:24.296+0000 7f5cc1806700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5cbc0f5410 0x7f5cbc10b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:54:24.299 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:24.296+0000 7f5cc1806700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cbc107210 con 0x7f5cbc0f5410 2023-11-15T01:54:24.301 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:24.296+0000 7f5cbaffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5cbc0f5410 0x7f5cbc10b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:54:24.301 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:24.296+0000 7f5cbaffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5cbc0f5410 0x7f5cbc10b820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:55986/0 (socket says 172.21.15.137:55986) 2023-11-15T01:54:24.301 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:24.296+0000 7f5cbaffd700 1 -- 172.21.15.137:0/211792426 learned_addr learned my addr 172.21.15.137:0/211792426 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:54:24.302 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:24.297+0000 7f5cbaffd700 1 -- 172.21.15.137:0/211792426 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cb0005040 con 0x7f5cbc0f5410 2023-11-15T01:54:24.302 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:24.297+0000 7f5cbaffd700 1 --2- 172.21.15.137:0/211792426 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5cbc0f5410 0x7f5cbc10b820 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f5cb0014000 tx=0x7f5cb000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:54:24.302 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:24.297+0000 7f5ca37fe700 1 -- 172.21.15.137:0/211792426 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5cb00095f0 con 0x7f5cbc0f5410 2023-11-15T01:54:24.302 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:24.297+0000 7f5cc1806700 1 -- 172.21.15.137:0/211792426 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cbc060220 con 0x7f5cbc0f5410 2023-11-15T01:54:24.303 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:24.297+0000 7f5ca37fe700 1 -- 172.21.15.137:0/211792426 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5cb0007e70 con 0x7f5cbc0f5410 2023-11-15T01:54:24.303 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:24.297+0000 7f5ca37fe700 1 -- 172.21.15.137:0/211792426 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5cb0014ea0 con 0x7f5cbc0f5410 2023-11-15T01:54:24.303 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:24.297+0000 7f5cc1806700 1 -- 172.21.15.137:0/211792426 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cbc10bd10 con 0x7f5cbc0f5410 2023-11-15T01:54:24.303 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:24.298+0000 7f5ca37fe700 1 -- 172.21.15.137:0/211792426 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5cb0025070 con 0x7f5cbc0f5410 2023-11-15T01:54:24.304 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:24.298+0000 7f5cc1806700 1 -- 172.21.15.137:0/211792426 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cbc045ad0 con 0x7f5cbc0f5410 2023-11-15T01:54:24.304 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:24.298+0000 7f5ca37fe700 1 --2- 172.21.15.137:0/211792426 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f5ca402ff60 0x7f5ca4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:54:24.304 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:24.298+0000 7f5ca37fe700 1 -- 172.21.15.137:0/211792426 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5cb0040a30 con 0x7f5cbc0f5410 2023-11-15T01:54:24.304 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:24.298+0000 7f5cba7fc700 1 --2- 172.21.15.137:0/211792426 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f5ca402ff60 0x7f5ca4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:54:24.305 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:24.299+0000 7f5cba7fc700 1 --2- 172.21.15.137:0/211792426 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f5ca402ff60 0x7f5ca4032420 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f5cac0096a0 tx=0x7f5cac006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:54:24.305 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:24.303+0000 7f5ca37fe700 1 -- 172.21.15.137:0/211792426 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5cb00079c0 con 0x7f5cbc0f5410 2023-11-15T01:54:24.509 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:24.509+0000 7f5cc1806700 1 -- 172.21.15.137:0/211792426 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5cbc0fae60 con 0x7f5cbc0f5410 2023-11-15T01:54:24.510 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:24.509+0000 7f5ca37fe700 1 -- 172.21.15.137:0/211792426 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5cb000f300 con 0x7f5cbc0f5410 2023-11-15T01:54:24.510 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:54:24.511 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:54:24.512 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:24.511+0000 7f5cc1806700 1 -- 172.21.15.137:0/211792426 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f5ca402ff60 msgr2=0x7f5ca4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:54:24.512 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:24.511+0000 7f5cc1806700 1 --2- 172.21.15.137:0/211792426 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f5ca402ff60 0x7f5ca4032420 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f5cac0096a0 tx=0x7f5cac006b40).stop 2023-11-15T01:54:24.512 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:24.511+0000 7f5cc1806700 1 -- 172.21.15.137:0/211792426 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5cbc0f5410 msgr2=0x7f5cbc10b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:54:24.513 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:24.511+0000 7f5cc1806700 1 --2- 172.21.15.137:0/211792426 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5cbc0f5410 0x7f5cbc10b820 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f5cb0014000 tx=0x7f5cb000b040).stop 2023-11-15T01:54:24.513 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:24.511+0000 7f5cc1806700 1 -- 172.21.15.137:0/211792426 shutdown_connections 2023-11-15T01:54:24.513 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:24.511+0000 7f5cc1806700 1 --2- 172.21.15.137:0/211792426 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f5ca402ff60 0x7f5ca4032420 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:54:24.513 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:24.511+0000 7f5cc1806700 1 --2- 172.21.15.137:0/211792426 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5cbc0f5410 0x7f5cbc10b820 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:54:24.514 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:24.511+0000 7f5cc1806700 1 -- 172.21.15.137:0/211792426 >> 172.21.15.137:0/211792426 conn(0x7f5cbc0f0b70 msgr2=0x7f5cbc0f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:54:24.514 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:24.511+0000 7f5cc1806700 1 -- 172.21.15.137:0/211792426 shutdown_connections 2023-11-15T01:54:24.514 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:24.511+0000 7f5cc1806700 1 -- 172.21.15.137:0/211792426 wait complete. 2023-11-15T01:54:24.515 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:54:24.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:54:24 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:54:24.510561+0000 mon.smithi055 (mon.0) 215 : audit [DBG] from='client.? 172.21.15.137:0/211792426' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:54:25.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:54:25 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:54:24.735013+0000 mgr.smithi055.szwngw (mgr.14164) 142 : cluster [DBG] pgmap v135: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:54:25.893 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:54:25.894 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:54:27.038 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:27.036+0000 7f007badc700 1 -- 172.21.15.137:0/1921456493 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f00740f6510 msgr2=0x7f00740f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:54:27.038 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:27.036+0000 7f007badc700 1 --2- 172.21.15.137:0/1921456493 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f00740f6510 0x7f00740f6930 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7f0070009fb0 tx=0x7f0070009b30).stop 2023-11-15T01:54:27.038 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:27.036+0000 7f007badc700 1 -- 172.21.15.137:0/1921456493 shutdown_connections 2023-11-15T01:54:27.039 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:27.036+0000 7f007badc700 1 --2- 172.21.15.137:0/1921456493 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f00740f6510 0x7f00740f6930 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:54:27.039 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:27.036+0000 7f007badc700 1 -- 172.21.15.137:0/1921456493 >> 172.21.15.137:0/1921456493 conn(0x7f00740f1c70 msgr2=0x7f00740f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:54:27.039 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:27.036+0000 7f007badc700 1 -- 172.21.15.137:0/1921456493 shutdown_connections 2023-11-15T01:54:27.039 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:27.036+0000 7f007badc700 1 -- 172.21.15.137:0/1921456493 wait complete. 2023-11-15T01:54:27.039 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:27.037+0000 7f007badc700 1 Processor -- start 2023-11-15T01:54:27.039 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:27.037+0000 7f007badc700 1 -- start start 2023-11-15T01:54:27.040 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:27.037+0000 7f007badc700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f00740f6510 0x7f0074106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:54:27.040 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:27.037+0000 7f007badc700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0074106c20 con 0x7f00740f6510 2023-11-15T01:54:27.040 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:27.037+0000 7f0079878700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f00740f6510 0x7f0074106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:54:27.040 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:27.037+0000 7f0079878700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f00740f6510 0x7f0074106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:56000/0 (socket says 172.21.15.137:56000) 2023-11-15T01:54:27.040 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:27.037+0000 7f0079878700 1 -- 172.21.15.137:0/4173358905 learned_addr learned my addr 172.21.15.137:0/4173358905 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:54:27.042 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:27.038+0000 7f0079878700 1 -- 172.21.15.137:0/4173358905 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0070005040 con 0x7f00740f6510 2023-11-15T01:54:27.042 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:27.038+0000 7f0079878700 1 --2- 172.21.15.137:0/4173358905 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f00740f6510 0x7f0074106730 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f0070014000 tx=0x7f007000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:54:27.042 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:27.038+0000 7f006a7fc700 1 -- 172.21.15.137:0/4173358905 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f00700095f0 con 0x7f00740f6510 2023-11-15T01:54:27.043 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:27.038+0000 7f007badc700 1 -- 172.21.15.137:0/4173358905 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0074106e20 con 0x7f00740f6510 2023-11-15T01:54:27.043 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:27.038+0000 7f006a7fc700 1 -- 172.21.15.137:0/4173358905 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0070007e70 con 0x7f00740f6510 2023-11-15T01:54:27.043 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:27.038+0000 7f006a7fc700 1 -- 172.21.15.137:0/4173358905 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0070014ea0 con 0x7f00740f6510 2023-11-15T01:54:27.043 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:27.038+0000 7f007badc700 1 -- 172.21.15.137:0/4173358905 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00741072b0 con 0x7f00740f6510 2023-11-15T01:54:27.043 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:27.039+0000 7f006a7fc700 1 -- 172.21.15.137:0/4173358905 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0070025070 con 0x7f00740f6510 2023-11-15T01:54:27.044 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:27.039+0000 7f007badc700 1 -- 172.21.15.137:0/4173358905 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0074045ad0 con 0x7f00740f6510 2023-11-15T01:54:27.044 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:27.039+0000 7f006a7fc700 1 --2- 172.21.15.137:0/4173358905 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f006002ff60 0x7f0060032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:54:27.044 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:27.039+0000 7f006a7fc700 1 -- 172.21.15.137:0/4173358905 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f00700409f0 con 0x7f00740f6510 2023-11-15T01:54:27.044 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:27.039+0000 7f0079077700 1 --2- 172.21.15.137:0/4173358905 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f006002ff60 0x7f0060032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:54:27.044 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:27.039+0000 7f0079077700 1 --2- 172.21.15.137:0/4173358905 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f006002ff60 0x7f0060032420 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f00640096a0 tx=0x7f0064006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:54:27.045 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:27.044+0000 7f006a7fc700 1 -- 172.21.15.137:0/4173358905 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f007000f4e0 con 0x7f00740f6510 2023-11-15T01:54:27.250 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:27.250+0000 7f007badc700 1 -- 172.21.15.137:0/4173358905 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f00740fbf60 con 0x7f00740f6510 2023-11-15T01:54:27.251 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:27.250+0000 7f006a7fc700 1 -- 172.21.15.137:0/4173358905 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f007000f300 con 0x7f00740f6510 2023-11-15T01:54:27.251 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:54:27.252 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:54:27.252 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:27.252+0000 7f007badc700 1 -- 172.21.15.137:0/4173358905 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f006002ff60 msgr2=0x7f0060032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:54:27.253 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:27.252+0000 7f007badc700 1 --2- 172.21.15.137:0/4173358905 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f006002ff60 0x7f0060032420 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f00640096a0 tx=0x7f0064006b40).stop 2023-11-15T01:54:27.253 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:27.252+0000 7f007badc700 1 -- 172.21.15.137:0/4173358905 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f00740f6510 msgr2=0x7f0074106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:54:27.253 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:27.252+0000 7f007badc700 1 --2- 172.21.15.137:0/4173358905 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f00740f6510 0x7f0074106730 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f0070014000 tx=0x7f007000b040).stop 2023-11-15T01:54:27.253 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:27.252+0000 7f007badc700 1 -- 172.21.15.137:0/4173358905 shutdown_connections 2023-11-15T01:54:27.253 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:27.252+0000 7f007badc700 1 --2- 172.21.15.137:0/4173358905 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f006002ff60 0x7f0060032420 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:54:27.254 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:27.252+0000 7f007badc700 1 --2- 172.21.15.137:0/4173358905 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f00740f6510 0x7f0074106730 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:54:27.254 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:27.252+0000 7f007badc700 1 -- 172.21.15.137:0/4173358905 >> 172.21.15.137:0/4173358905 conn(0x7f00740f1c70 msgr2=0x7f00740f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:54:27.254 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:27.252+0000 7f007badc700 1 -- 172.21.15.137:0/4173358905 shutdown_connections 2023-11-15T01:54:27.254 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:27.252+0000 7f007badc700 1 -- 172.21.15.137:0/4173358905 wait complete. 2023-11-15T01:54:27.255 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:54:28.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:54:27 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:54:26.735245+0000 mgr.smithi055.szwngw (mgr.14164) 143 : cluster [DBG] pgmap v136: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:54:28.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:54:27 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:54:27.251499+0000 mon.smithi055 (mon.0) 216 : audit [DBG] from='client.? 172.21.15.137:0/4173358905' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:54:28.636 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:54:28.637 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:54:29.785 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:29.785+0000 7f7df7a75700 1 -- 172.21.15.137:0/2681909956 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7df00f6510 msgr2=0x7f7df00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:54:29.786 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:29.785+0000 7f7df7a75700 1 --2- 172.21.15.137:0/2681909956 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7df00f6510 0x7f7df00f6930 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7f7dec009fb0 tx=0x7f7dec009b30).stop 2023-11-15T01:54:29.786 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:29.785+0000 7f7df7a75700 1 -- 172.21.15.137:0/2681909956 shutdown_connections 2023-11-15T01:54:29.786 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:29.785+0000 7f7df7a75700 1 --2- 172.21.15.137:0/2681909956 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7df00f6510 0x7f7df00f6930 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:54:29.786 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:29.785+0000 7f7df7a75700 1 -- 172.21.15.137:0/2681909956 >> 172.21.15.137:0/2681909956 conn(0x7f7df00f1c70 msgr2=0x7f7df00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:54:29.787 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:29.785+0000 7f7df7a75700 1 -- 172.21.15.137:0/2681909956 shutdown_connections 2023-11-15T01:54:29.787 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:29.785+0000 7f7df7a75700 1 -- 172.21.15.137:0/2681909956 wait complete. 2023-11-15T01:54:29.789 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:29.785+0000 7f7df7a75700 1 Processor -- start 2023-11-15T01:54:29.789 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:29.785+0000 7f7df7a75700 1 -- start start 2023-11-15T01:54:29.789 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:29.785+0000 7f7df7a75700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7df00f6510 0x7f7df0106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:54:29.790 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:29.786+0000 7f7df7a75700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7df0106c20 con 0x7f7df00f6510 2023-11-15T01:54:29.790 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:29.786+0000 7f7df5811700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7df00f6510 0x7f7df0106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:54:29.790 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:29.786+0000 7f7df5811700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7df00f6510 0x7f7df0106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:55714/0 (socket says 172.21.15.137:55714) 2023-11-15T01:54:29.790 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:29.786+0000 7f7df5811700 1 -- 172.21.15.137:0/1863619494 learned_addr learned my addr 172.21.15.137:0/1863619494 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:54:29.790 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:29.786+0000 7f7df5811700 1 -- 172.21.15.137:0/1863619494 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dec005040 con 0x7f7df00f6510 2023-11-15T01:54:29.791 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:29.786+0000 7f7df5811700 1 --2- 172.21.15.137:0/1863619494 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7df00f6510 0x7f7df0106730 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f7dec014000 tx=0x7f7dec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:54:29.791 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:29.786+0000 7f7de67fc700 1 -- 172.21.15.137:0/1863619494 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7dec0095f0 con 0x7f7df00f6510 2023-11-15T01:54:29.791 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:29.786+0000 7f7df7a75700 1 -- 172.21.15.137:0/1863619494 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7df0106e20 con 0x7f7df00f6510 2023-11-15T01:54:29.791 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:29.786+0000 7f7de67fc700 1 -- 172.21.15.137:0/1863619494 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7dec007e70 con 0x7f7df00f6510 2023-11-15T01:54:29.791 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:29.786+0000 7f7de67fc700 1 -- 172.21.15.137:0/1863619494 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7dec014ea0 con 0x7f7df00f6510 2023-11-15T01:54:29.792 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:29.787+0000 7f7df7a75700 1 -- 172.21.15.137:0/1863619494 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7df01072b0 con 0x7f7df00f6510 2023-11-15T01:54:29.792 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:29.787+0000 7f7de67fc700 1 -- 172.21.15.137:0/1863619494 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7dec01d4e0 con 0x7f7df00f6510 2023-11-15T01:54:29.792 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:29.787+0000 7f7de67fc700 1 --2- 172.21.15.137:0/1863619494 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f7ddc02ff60 0x7f7ddc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:54:29.792 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:29.787+0000 7f7de67fc700 1 -- 172.21.15.137:0/1863619494 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7dec040980 con 0x7f7df00f6510 2023-11-15T01:54:29.792 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:29.787+0000 7f7df7a75700 1 -- 172.21.15.137:0/1863619494 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7df0045ad0 con 0x7f7df00f6510 2023-11-15T01:54:29.793 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:29.788+0000 7f7df5010700 1 --2- 172.21.15.137:0/1863619494 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f7ddc02ff60 0x7f7ddc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:54:29.793 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:29.788+0000 7f7df5010700 1 --2- 172.21.15.137:0/1863619494 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f7ddc02ff60 0x7f7ddc032420 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f7de00096a0 tx=0x7f7de0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:54:29.793 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:29.793+0000 7f7de67fc700 1 -- 172.21.15.137:0/1863619494 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7dec00f8f0 con 0x7f7df00f6510 2023-11-15T01:54:30.000 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:29.999+0000 7f7df7a75700 1 -- 172.21.15.137:0/1863619494 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7df00fc050 con 0x7f7df00f6510 2023-11-15T01:54:30.000 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:30.000+0000 7f7de67fc700 1 -- 172.21.15.137:0/1863619494 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7dec02c090 con 0x7f7df00f6510 2023-11-15T01:54:30.001 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:54:30.001 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:54:30.002 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:30.001+0000 7f7df7a75700 1 -- 172.21.15.137:0/1863619494 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f7ddc02ff60 msgr2=0x7f7ddc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:54:30.002 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:30.001+0000 7f7df7a75700 1 --2- 172.21.15.137:0/1863619494 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f7ddc02ff60 0x7f7ddc032420 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f7de00096a0 tx=0x7f7de0006b40).stop 2023-11-15T01:54:30.003 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:30.001+0000 7f7df7a75700 1 -- 172.21.15.137:0/1863619494 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7df00f6510 msgr2=0x7f7df0106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:54:30.003 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:30.001+0000 7f7df7a75700 1 --2- 172.21.15.137:0/1863619494 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7df00f6510 0x7f7df0106730 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f7dec014000 tx=0x7f7dec00b040).stop 2023-11-15T01:54:30.003 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:30.001+0000 7f7df7a75700 1 -- 172.21.15.137:0/1863619494 shutdown_connections 2023-11-15T01:54:30.003 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:30.001+0000 7f7df7a75700 1 --2- 172.21.15.137:0/1863619494 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f7ddc02ff60 0x7f7ddc032420 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:54:30.004 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:30.001+0000 7f7df7a75700 1 --2- 172.21.15.137:0/1863619494 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7df00f6510 0x7f7df0106730 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:54:30.004 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:30.001+0000 7f7df7a75700 1 -- 172.21.15.137:0/1863619494 >> 172.21.15.137:0/1863619494 conn(0x7f7df00f1c70 msgr2=0x7f7df00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:54:30.004 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:30.001+0000 7f7df7a75700 1 -- 172.21.15.137:0/1863619494 shutdown_connections 2023-11-15T01:54:30.005 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:30.001+0000 7f7df7a75700 1 -- 172.21.15.137:0/1863619494 wait complete. 2023-11-15T01:54:30.006 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:54:30.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:54:29 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:54:28.735561+0000 mgr.smithi055.szwngw (mgr.14164) 144 : cluster [DBG] pgmap v137: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:54:31.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:54:30 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:54:30.000833+0000 mon.smithi055 (mon.0) 217 : audit [DBG] from='client.? 172.21.15.137:0/1863619494' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:54:31.403 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:54:31.403 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:54:32.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:54:31 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:54:30.735898+0000 mgr.smithi055.szwngw (mgr.14164) 145 : cluster [DBG] pgmap v138: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:54:32.557 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:32.555+0000 7ff7db150700 1 -- 172.21.15.137:0/1225199672 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff7d40f51a0 msgr2=0x7ff7d40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:54:32.558 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:32.555+0000 7ff7db150700 1 --2- 172.21.15.137:0/1225199672 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff7d40f51a0 0x7ff7d40f55c0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7ff7c4009fb0 tx=0x7ff7c4009b30).stop 2023-11-15T01:54:32.558 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:32.556+0000 7ff7db150700 1 -- 172.21.15.137:0/1225199672 shutdown_connections 2023-11-15T01:54:32.558 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:32.556+0000 7ff7db150700 1 --2- 172.21.15.137:0/1225199672 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff7d40f51a0 0x7ff7d40f55c0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:54:32.558 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:32.556+0000 7ff7db150700 1 -- 172.21.15.137:0/1225199672 >> 172.21.15.137:0/1225199672 conn(0x7ff7d40f09a0 msgr2=0x7ff7d40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:54:32.559 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:32.556+0000 7ff7db150700 1 -- 172.21.15.137:0/1225199672 shutdown_connections 2023-11-15T01:54:32.559 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:32.556+0000 7ff7db150700 1 -- 172.21.15.137:0/1225199672 wait complete. 2023-11-15T01:54:32.559 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:32.556+0000 7ff7db150700 1 Processor -- start 2023-11-15T01:54:32.559 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:32.556+0000 7ff7db150700 1 -- start start 2023-11-15T01:54:32.559 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:32.556+0000 7ff7db150700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff7d40f51a0 0x7ff7d40fd120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:54:32.560 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:32.556+0000 7ff7db150700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d40fb5a0 con 0x7ff7d40f51a0 2023-11-15T01:54:32.560 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:32.557+0000 7ff7da14e700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff7d40f51a0 0x7ff7d40fd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:54:32.560 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:32.557+0000 7ff7da14e700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff7d40f51a0 0x7ff7d40fd120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:55734/0 (socket says 172.21.15.137:55734) 2023-11-15T01:54:32.560 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:32.557+0000 7ff7da14e700 1 -- 172.21.15.137:0/3144348534 learned_addr learned my addr 172.21.15.137:0/3144348534 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:54:32.560 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:32.557+0000 7ff7da14e700 1 -- 172.21.15.137:0/3144348534 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7c4005040 con 0x7ff7d40f51a0 2023-11-15T01:54:32.561 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:32.557+0000 7ff7da14e700 1 --2- 172.21.15.137:0/3144348534 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff7d40f51a0 0x7ff7d40fd120 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7ff7c4013510 tx=0x7ff7c400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:54:32.562 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:32.557+0000 7ff7caffd700 1 -- 172.21.15.137:0/3144348534 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff7c4009320 con 0x7ff7d40f51a0 2023-11-15T01:54:32.562 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:32.557+0000 7ff7db150700 1 -- 172.21.15.137:0/3144348534 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7d40fb740 con 0x7ff7d40f51a0 2023-11-15T01:54:32.562 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:32.557+0000 7ff7caffd700 1 -- 172.21.15.137:0/3144348534 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff7c4013f00 con 0x7ff7d40f51a0 2023-11-15T01:54:32.562 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:32.557+0000 7ff7caffd700 1 -- 172.21.15.137:0/3144348534 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff7c4014dc0 con 0x7ff7d40f51a0 2023-11-15T01:54:32.563 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:32.557+0000 7ff7db150700 1 -- 172.21.15.137:0/3144348534 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7d40fbbd0 con 0x7ff7d40f51a0 2023-11-15T01:54:32.563 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:32.558+0000 7ff7caffd700 1 -- 172.21.15.137:0/3144348534 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff7c401d3d0 con 0x7ff7d40f51a0 2023-11-15T01:54:32.563 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:32.558+0000 7ff7caffd700 1 --2- 172.21.15.137:0/3144348534 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7ff7c002ffb0 0x7ff7c0032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:54:32.563 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:32.558+0000 7ff7db150700 1 -- 172.21.15.137:0/3144348534 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7d4059100 con 0x7ff7d40f51a0 2023-11-15T01:54:32.563 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:32.558+0000 7ff7caffd700 1 -- 172.21.15.137:0/3144348534 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff7c40413e0 con 0x7ff7d40f51a0 2023-11-15T01:54:32.564 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:32.558+0000 7ff7d994d700 1 --2- 172.21.15.137:0/3144348534 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7ff7c002ffb0 0x7ff7c0032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:54:32.564 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:32.559+0000 7ff7d994d700 1 --2- 172.21.15.137:0/3144348534 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7ff7c002ffb0 0x7ff7c0032470 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7ff7d00096a0 tx=0x7ff7d0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:54:32.564 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:32.563+0000 7ff7caffd700 1 -- 172.21.15.137:0/3144348534 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff7c4012510 con 0x7ff7d40f51a0 2023-11-15T01:54:32.770 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:32.770+0000 7ff7db150700 1 -- 172.21.15.137:0/3144348534 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff7d40fabe0 con 0x7ff7d40f51a0 2023-11-15T01:54:32.771 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:32.770+0000 7ff7caffd700 1 -- 172.21.15.137:0/3144348534 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff7c4012330 con 0x7ff7d40f51a0 2023-11-15T01:54:32.772 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:54:32.772 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:54:32.773 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:32.772+0000 7ff7db150700 1 -- 172.21.15.137:0/3144348534 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7ff7c002ffb0 msgr2=0x7ff7c0032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:54:32.773 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:32.772+0000 7ff7db150700 1 --2- 172.21.15.137:0/3144348534 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7ff7c002ffb0 0x7ff7c0032470 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7ff7d00096a0 tx=0x7ff7d0006b40).stop 2023-11-15T01:54:32.773 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:32.772+0000 7ff7db150700 1 -- 172.21.15.137:0/3144348534 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff7d40f51a0 msgr2=0x7ff7d40fd120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:54:32.774 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:32.772+0000 7ff7db150700 1 --2- 172.21.15.137:0/3144348534 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff7d40f51a0 0x7ff7d40fd120 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7ff7c4013510 tx=0x7ff7c400b040).stop 2023-11-15T01:54:32.774 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:32.772+0000 7ff7db150700 1 -- 172.21.15.137:0/3144348534 shutdown_connections 2023-11-15T01:54:32.774 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:32.772+0000 7ff7db150700 1 --2- 172.21.15.137:0/3144348534 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7ff7c002ffb0 0x7ff7c0032470 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:54:32.774 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:32.772+0000 7ff7db150700 1 --2- 172.21.15.137:0/3144348534 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff7d40f51a0 0x7ff7d40fd120 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:54:32.775 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:32.772+0000 7ff7db150700 1 -- 172.21.15.137:0/3144348534 >> 172.21.15.137:0/3144348534 conn(0x7ff7d40f09a0 msgr2=0x7ff7d405fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:54:32.775 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:32.772+0000 7ff7db150700 1 -- 172.21.15.137:0/3144348534 shutdown_connections 2023-11-15T01:54:32.775 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:32.772+0000 7ff7db150700 1 -- 172.21.15.137:0/3144348534 wait complete. 2023-11-15T01:54:32.775 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:54:33.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:54:32 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:54:32.771402+0000 mon.smithi055 (mon.0) 218 : audit [DBG] from='client.? 172.21.15.137:0/3144348534' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:54:34.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:54:33 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:54:32.736241+0000 mgr.smithi055.szwngw (mgr.14164) 146 : cluster [DBG] pgmap v139: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:54:34.177 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:54:34.177 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:54:35.370 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:35.368+0000 7f6f68d0c700 1 -- 172.21.15.137:0/3317615451 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6f640f6510 msgr2=0x7f6f640f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:54:35.371 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:35.368+0000 7f6f68d0c700 1 --2- 172.21.15.137:0/3317615451 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6f640f6510 0x7f6f640f6930 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7f6f54009fb0 tx=0x7f6f54009b30).stop 2023-11-15T01:54:35.371 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:35.368+0000 7f6f68d0c700 1 -- 172.21.15.137:0/3317615451 shutdown_connections 2023-11-15T01:54:35.371 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:35.368+0000 7f6f68d0c700 1 --2- 172.21.15.137:0/3317615451 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6f640f6510 0x7f6f640f6930 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:54:35.371 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:35.368+0000 7f6f68d0c700 1 -- 172.21.15.137:0/3317615451 >> 172.21.15.137:0/3317615451 conn(0x7f6f640f1c70 msgr2=0x7f6f640f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:54:35.372 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:35.368+0000 7f6f68d0c700 1 -- 172.21.15.137:0/3317615451 shutdown_connections 2023-11-15T01:54:35.372 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:35.368+0000 7f6f68d0c700 1 -- 172.21.15.137:0/3317615451 wait complete. 2023-11-15T01:54:35.372 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:35.368+0000 7f6f68d0c700 1 Processor -- start 2023-11-15T01:54:35.372 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:35.368+0000 7f6f68d0c700 1 -- start start 2023-11-15T01:54:35.372 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:35.368+0000 7f6f68d0c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6f640f6510 0x7f6f6406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:54:35.373 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:35.368+0000 7f6f68d0c700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f64069e20 con 0x7f6f640f6510 2023-11-15T01:54:35.373 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:35.369+0000 7f6f6259c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6f640f6510 0x7f6f6406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:54:35.373 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:35.369+0000 7f6f6259c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6f640f6510 0x7f6f6406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:55758/0 (socket says 172.21.15.137:55758) 2023-11-15T01:54:35.373 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:35.369+0000 7f6f6259c700 1 -- 172.21.15.137:0/1465422056 learned_addr learned my addr 172.21.15.137:0/1465422056 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:54:35.374 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:35.369+0000 7f6f6259c700 1 -- 172.21.15.137:0/1465422056 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f54005040 con 0x7f6f640f6510 2023-11-15T01:54:35.374 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:35.369+0000 7f6f6259c700 1 --2- 172.21.15.137:0/1465422056 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6f640f6510 0x7f6f6406b9a0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f6f54014000 tx=0x7f6f5400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:54:35.374 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:35.369+0000 7f6f5affd700 1 -- 172.21.15.137:0/1465422056 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f540095f0 con 0x7f6f640f6510 2023-11-15T01:54:35.374 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:35.370+0000 7f6f68d0c700 1 -- 172.21.15.137:0/1465422056 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f64069fc0 con 0x7f6f640f6510 2023-11-15T01:54:35.374 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:35.370+0000 7f6f5affd700 1 -- 172.21.15.137:0/1465422056 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6f54007e70 con 0x7f6f640f6510 2023-11-15T01:54:35.375 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:35.370+0000 7f6f5affd700 1 -- 172.21.15.137:0/1465422056 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f54014ea0 con 0x7f6f640f6510 2023-11-15T01:54:35.375 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:35.370+0000 7f6f68d0c700 1 -- 172.21.15.137:0/1465422056 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f6406a450 con 0x7f6f640f6510 2023-11-15T01:54:35.376 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:35.370+0000 7f6f68d0c700 1 -- 172.21.15.137:0/1465422056 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f64059100 con 0x7f6f640f6510 2023-11-15T01:54:35.376 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:35.370+0000 7f6f5affd700 1 -- 172.21.15.137:0/1465422056 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6f54025070 con 0x7f6f640f6510 2023-11-15T01:54:35.376 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:35.371+0000 7f6f5affd700 1 --2- 172.21.15.137:0/1465422056 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f6f5002ff60 0x7f6f50032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:54:35.377 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:35.371+0000 7f6f5affd700 1 -- 172.21.15.137:0/1465422056 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6f54040940 con 0x7f6f640f6510 2023-11-15T01:54:35.377 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:35.371+0000 7f6f61d9b700 1 --2- 172.21.15.137:0/1465422056 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f6f5002ff60 0x7f6f50032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:54:35.377 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:35.372+0000 7f6f61d9b700 1 --2- 172.21.15.137:0/1465422056 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f6f5002ff60 0x7f6f50032420 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f6f4c0096a0 tx=0x7f6f4c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:54:35.378 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:35.376+0000 7f6f5affd700 1 -- 172.21.15.137:0/1465422056 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6f540079c0 con 0x7f6f640f6510 2023-11-15T01:54:35.583 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:35.582+0000 7f6f68d0c700 1 -- 172.21.15.137:0/1465422056 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6f64045ad0 con 0x7f6f640f6510 2023-11-15T01:54:35.584 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:35.583+0000 7f6f5affd700 1 -- 172.21.15.137:0/1465422056 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6f5400f300 con 0x7f6f640f6510 2023-11-15T01:54:35.585 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:54:35.585 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:54:35.587 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:35.585+0000 7f6f68d0c700 1 -- 172.21.15.137:0/1465422056 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f6f5002ff60 msgr2=0x7f6f50032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:54:35.587 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:35.585+0000 7f6f68d0c700 1 --2- 172.21.15.137:0/1465422056 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f6f5002ff60 0x7f6f50032420 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f6f4c0096a0 tx=0x7f6f4c006b40).stop 2023-11-15T01:54:35.588 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:35.585+0000 7f6f68d0c700 1 -- 172.21.15.137:0/1465422056 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6f640f6510 msgr2=0x7f6f6406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:54:35.588 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:35.585+0000 7f6f68d0c700 1 --2- 172.21.15.137:0/1465422056 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6f640f6510 0x7f6f6406b9a0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f6f54014000 tx=0x7f6f5400b040).stop 2023-11-15T01:54:35.589 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:35.585+0000 7f6f68d0c700 1 -- 172.21.15.137:0/1465422056 shutdown_connections 2023-11-15T01:54:35.589 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:35.585+0000 7f6f68d0c700 1 --2- 172.21.15.137:0/1465422056 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f6f5002ff60 0x7f6f50032420 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:54:35.589 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:35.585+0000 7f6f68d0c700 1 --2- 172.21.15.137:0/1465422056 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6f640f6510 0x7f6f6406b9a0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:54:35.590 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:35.585+0000 7f6f68d0c700 1 -- 172.21.15.137:0/1465422056 >> 172.21.15.137:0/1465422056 conn(0x7f6f640f1c70 msgr2=0x7f6f640f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:54:35.590 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:35.585+0000 7f6f68d0c700 1 -- 172.21.15.137:0/1465422056 shutdown_connections 2023-11-15T01:54:35.591 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:35.585+0000 7f6f68d0c700 1 -- 172.21.15.137:0/1465422056 wait complete. 2023-11-15T01:54:35.591 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:54:36.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:54:35 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:54:34.736539+0000 mgr.smithi055.szwngw (mgr.14164) 147 : cluster [DBG] pgmap v140: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:54:36.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:54:35 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:54:35.584421+0000 mon.smithi055 (mon.0) 219 : audit [DBG] from='client.? 172.21.15.137:0/1465422056' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:54:36.983 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:54:36.983 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:54:38.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:54:37 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:54:36.736802+0000 mgr.smithi055.szwngw (mgr.14164) 148 : cluster [DBG] pgmap v141: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:54:38.121 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:38.120+0000 7fd5c984c700 1 -- 172.21.15.137:0/3607370287 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd5c40f5410 msgr2=0x7fd5c40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:54:38.122 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:38.121+0000 7fd5c984c700 1 --2- 172.21.15.137:0/3607370287 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd5c40f5410 0x7fd5c40f5830 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7fd5b4009fb0 tx=0x7fd5b4009b30).stop 2023-11-15T01:54:38.122 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:38.121+0000 7fd5c984c700 1 -- 172.21.15.137:0/3607370287 shutdown_connections 2023-11-15T01:54:38.122 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:38.121+0000 7fd5c984c700 1 --2- 172.21.15.137:0/3607370287 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd5c40f5410 0x7fd5c40f5830 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:54:38.122 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:38.121+0000 7fd5c984c700 1 -- 172.21.15.137:0/3607370287 >> 172.21.15.137:0/3607370287 conn(0x7fd5c40f0b70 msgr2=0x7fd5c40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:54:38.123 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:38.121+0000 7fd5c984c700 1 -- 172.21.15.137:0/3607370287 shutdown_connections 2023-11-15T01:54:38.123 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:38.121+0000 7fd5c984c700 1 -- 172.21.15.137:0/3607370287 wait complete. 2023-11-15T01:54:38.125 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:38.121+0000 7fd5c984c700 1 Processor -- start 2023-11-15T01:54:38.125 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:38.121+0000 7fd5c984c700 1 -- start start 2023-11-15T01:54:38.125 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:38.121+0000 7fd5c984c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd5c40f5410 0x7fd5c4105600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:54:38.125 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:38.121+0000 7fd5c984c700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5c4105af0 con 0x7fd5c40f5410 2023-11-15T01:54:38.126 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:38.122+0000 7fd5c2ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd5c40f5410 0x7fd5c4105600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:54:38.126 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:38.122+0000 7fd5c2ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd5c40f5410 0x7fd5c4105600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:55780/0 (socket says 172.21.15.137:55780) 2023-11-15T01:54:38.126 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:38.122+0000 7fd5c2ffd700 1 -- 172.21.15.137:0/2721494022 learned_addr learned my addr 172.21.15.137:0/2721494022 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:54:38.126 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:38.122+0000 7fd5c2ffd700 1 -- 172.21.15.137:0/2721494022 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5b4005040 con 0x7fd5c40f5410 2023-11-15T01:54:38.126 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:38.122+0000 7fd5c2ffd700 1 --2- 172.21.15.137:0/2721494022 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd5c40f5410 0x7fd5c4105600 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7fd5b4013510 tx=0x7fd5b400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:54:38.127 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:38.122+0000 7fd5ab7fe700 1 -- 172.21.15.137:0/2721494022 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd5b4009320 con 0x7fd5c40f5410 2023-11-15T01:54:38.127 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:38.122+0000 7fd5c984c700 1 -- 172.21.15.137:0/2721494022 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5c4105cf0 con 0x7fd5c40f5410 2023-11-15T01:54:38.127 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:38.122+0000 7fd5ab7fe700 1 -- 172.21.15.137:0/2721494022 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd5b4013f00 con 0x7fd5c40f5410 2023-11-15T01:54:38.127 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:38.122+0000 7fd5ab7fe700 1 -- 172.21.15.137:0/2721494022 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd5b4014d90 con 0x7fd5c40f5410 2023-11-15T01:54:38.128 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:38.122+0000 7fd5c984c700 1 -- 172.21.15.137:0/2721494022 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5c4106180 con 0x7fd5c40f5410 2023-11-15T01:54:38.128 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:38.123+0000 7fd5ab7fe700 1 -- 172.21.15.137:0/2721494022 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd5b4014420 con 0x7fd5c40f5410 2023-11-15T01:54:38.128 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:38.123+0000 7fd5c984c700 1 -- 172.21.15.137:0/2721494022 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5c4045ad0 con 0x7fd5c40f5410 2023-11-15T01:54:38.128 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:38.123+0000 7fd5ab7fe700 1 --2- 172.21.15.137:0/2721494022 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fd5ac02ff60 0x7fd5ac032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:54:38.128 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:38.123+0000 7fd5ab7fe700 1 -- 172.21.15.137:0/2721494022 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd5b4041070 con 0x7fd5c40f5410 2023-11-15T01:54:38.128 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:38.124+0000 7fd5c27fc700 1 --2- 172.21.15.137:0/2721494022 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fd5ac02ff60 0x7fd5ac032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:54:38.129 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:38.124+0000 7fd5c27fc700 1 --2- 172.21.15.137:0/2721494022 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fd5ac02ff60 0x7fd5ac032420 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7fd5b80096a0 tx=0x7fd5b8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:54:38.129 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:38.128+0000 7fd5ab7fe700 1 -- 172.21.15.137:0/2721494022 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd5b40095f0 con 0x7fd5c40f5410 2023-11-15T01:54:38.338 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:38.337+0000 7fd5c984c700 1 -- 172.21.15.137:0/2721494022 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd5c40fae60 con 0x7fd5c40f5410 2023-11-15T01:54:38.339 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:38.338+0000 7fd5ab7fe700 1 -- 172.21.15.137:0/2721494022 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd5b401db70 con 0x7fd5c40f5410 2023-11-15T01:54:38.339 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:54:38.339 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:54:38.340 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:38.340+0000 7fd5c984c700 1 -- 172.21.15.137:0/2721494022 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fd5ac02ff60 msgr2=0x7fd5ac032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:54:38.341 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:38.340+0000 7fd5c984c700 1 --2- 172.21.15.137:0/2721494022 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fd5ac02ff60 0x7fd5ac032420 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7fd5b80096a0 tx=0x7fd5b8006b40).stop 2023-11-15T01:54:38.341 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:38.340+0000 7fd5c984c700 1 -- 172.21.15.137:0/2721494022 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd5c40f5410 msgr2=0x7fd5c4105600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:54:38.341 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:38.340+0000 7fd5c984c700 1 --2- 172.21.15.137:0/2721494022 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd5c40f5410 0x7fd5c4105600 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7fd5b4013510 tx=0x7fd5b400b040).stop 2023-11-15T01:54:38.341 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:38.340+0000 7fd5c984c700 1 -- 172.21.15.137:0/2721494022 shutdown_connections 2023-11-15T01:54:38.341 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:38.340+0000 7fd5c984c700 1 --2- 172.21.15.137:0/2721494022 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fd5ac02ff60 0x7fd5ac032420 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:54:38.341 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:38.340+0000 7fd5c984c700 1 --2- 172.21.15.137:0/2721494022 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd5c40f5410 0x7fd5c4105600 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:54:38.342 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:38.340+0000 7fd5c984c700 1 -- 172.21.15.137:0/2721494022 >> 172.21.15.137:0/2721494022 conn(0x7fd5c40f0b70 msgr2=0x7fd5c40f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:54:38.342 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:38.340+0000 7fd5c984c700 1 -- 172.21.15.137:0/2721494022 shutdown_connections 2023-11-15T01:54:38.342 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:38.340+0000 7fd5c984c700 1 -- 172.21.15.137:0/2721494022 wait complete. 2023-11-15T01:54:38.342 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:54:39.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:54:38 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:54:38.339044+0000 mon.smithi055 (mon.0) 220 : audit [DBG] from='client.? 172.21.15.137:0/2721494022' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:54:39.738 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:54:39.739 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:54:40.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:54:39 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:54:38.737076+0000 mgr.smithi055.szwngw (mgr.14164) 149 : cluster [DBG] pgmap v142: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:54:40.876 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:40.874+0000 7fb02e2df700 1 -- 172.21.15.137:0/3339641862 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb028060280 msgr2=0x7fb0280f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:54:40.876 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:40.874+0000 7fb02e2df700 1 --2- 172.21.15.137:0/3339641862 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb028060280 0x7fb0280f90f0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7fb010009fb0 tx=0x7fb010009b30).stop 2023-11-15T01:54:40.876 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:40.874+0000 7fb02e2df700 1 -- 172.21.15.137:0/3339641862 shutdown_connections 2023-11-15T01:54:40.877 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:40.874+0000 7fb02e2df700 1 --2- 172.21.15.137:0/3339641862 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb028060280 0x7fb0280f90f0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:54:40.877 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:40.874+0000 7fb02e2df700 1 -- 172.21.15.137:0/3339641862 >> 172.21.15.137:0/3339641862 conn(0x7fb0280f0b70 msgr2=0x7fb0280f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:54:40.877 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:40.874+0000 7fb02e2df700 1 -- 172.21.15.137:0/3339641862 shutdown_connections 2023-11-15T01:54:40.877 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:40.874+0000 7fb02e2df700 1 -- 172.21.15.137:0/3339641862 wait complete. 2023-11-15T01:54:40.877 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:40.875+0000 7fb02e2df700 1 Processor -- start 2023-11-15T01:54:40.878 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:40.875+0000 7fb02e2df700 1 -- start start 2023-11-15T01:54:40.878 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:40.875+0000 7fb02e2df700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb028060280 0x7fb0281066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:54:40.878 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:40.875+0000 7fb02e2df700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb028106bd0 con 0x7fb028060280 2023-11-15T01:54:40.878 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:40.875+0000 7fb027fff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb028060280 0x7fb0281066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:54:40.878 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:40.875+0000 7fb027fff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb028060280 0x7fb0281066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:40580/0 (socket says 172.21.15.137:40580) 2023-11-15T01:54:40.879 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:40.875+0000 7fb027fff700 1 -- 172.21.15.137:0/4001392204 learned_addr learned my addr 172.21.15.137:0/4001392204 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:54:40.880 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:40.876+0000 7fb027fff700 1 -- 172.21.15.137:0/4001392204 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb010005040 con 0x7fb028060280 2023-11-15T01:54:40.880 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:40.876+0000 7fb027fff700 1 --2- 172.21.15.137:0/4001392204 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb028060280 0x7fb0281066e0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7fb010013510 tx=0x7fb01000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:54:40.881 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:40.876+0000 7fb024ff9700 1 -- 172.21.15.137:0/4001392204 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb010009320 con 0x7fb028060280 2023-11-15T01:54:40.881 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:40.876+0000 7fb024ff9700 1 -- 172.21.15.137:0/4001392204 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb010013f00 con 0x7fb028060280 2023-11-15T01:54:40.883 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:40.876+0000 7fb02e2df700 1 -- 172.21.15.137:0/4001392204 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb028106dd0 con 0x7fb028060280 2023-11-15T01:54:40.883 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:40.876+0000 7fb024ff9700 1 -- 172.21.15.137:0/4001392204 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb010014dc0 con 0x7fb028060280 2023-11-15T01:54:40.883 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:40.876+0000 7fb02e2df700 1 -- 172.21.15.137:0/4001392204 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb028107260 con 0x7fb028060280 2023-11-15T01:54:40.883 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:40.877+0000 7fb024ff9700 1 -- 172.21.15.137:0/4001392204 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb0100095f0 con 0x7fb028060280 2023-11-15T01:54:40.884 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:40.877+0000 7fb02e2df700 1 -- 172.21.15.137:0/4001392204 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb028045ad0 con 0x7fb028060280 2023-11-15T01:54:40.884 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:40.877+0000 7fb024ff9700 1 --2- 172.21.15.137:0/4001392204 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fb01402ffb0 0x7fb014032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:54:40.885 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:40.877+0000 7fb024ff9700 1 -- 172.21.15.137:0/4001392204 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb010040fb0 con 0x7fb028060280 2023-11-15T01:54:40.886 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:40.878+0000 7fb0277fe700 1 --2- 172.21.15.137:0/4001392204 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fb01402ffb0 0x7fb014032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:54:40.886 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:40.878+0000 7fb0277fe700 1 --2- 172.21.15.137:0/4001392204 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fb01402ffb0 0x7fb014032470 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7fb0180096a0 tx=0x7fb018006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:54:40.886 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:40.882+0000 7fb024ff9700 1 -- 172.21.15.137:0/4001392204 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb01006d2b0 con 0x7fb028060280 2023-11-15T01:54:41.091 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:41.089+0000 7fb02e2df700 1 -- 172.21.15.137:0/4001392204 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb028108110 con 0x7fb028060280 2023-11-15T01:54:41.092 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:41.091+0000 7fb024ff9700 1 -- 172.21.15.137:0/4001392204 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb01001d7f0 con 0x7fb028060280 2023-11-15T01:54:41.092 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:54:41.093 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:54:41.094 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:41.092+0000 7fb02e2df700 1 -- 172.21.15.137:0/4001392204 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fb01402ffb0 msgr2=0x7fb014032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:54:41.094 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:41.092+0000 7fb02e2df700 1 --2- 172.21.15.137:0/4001392204 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fb01402ffb0 0x7fb014032470 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7fb0180096a0 tx=0x7fb018006b40).stop 2023-11-15T01:54:41.094 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:41.092+0000 7fb02e2df700 1 -- 172.21.15.137:0/4001392204 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb028060280 msgr2=0x7fb0281066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:54:41.094 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:41.092+0000 7fb02e2df700 1 --2- 172.21.15.137:0/4001392204 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb028060280 0x7fb0281066e0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7fb010013510 tx=0x7fb01000b040).stop 2023-11-15T01:54:41.095 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:41.093+0000 7fb02e2df700 1 -- 172.21.15.137:0/4001392204 shutdown_connections 2023-11-15T01:54:41.095 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:41.093+0000 7fb02e2df700 1 --2- 172.21.15.137:0/4001392204 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fb01402ffb0 0x7fb014032470 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:54:41.095 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:41.093+0000 7fb02e2df700 1 --2- 172.21.15.137:0/4001392204 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb028060280 0x7fb0281066e0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:54:41.095 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:41.093+0000 7fb02e2df700 1 -- 172.21.15.137:0/4001392204 >> 172.21.15.137:0/4001392204 conn(0x7fb0280f0b70 msgr2=0x7fb0280f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:54:41.095 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:41.093+0000 7fb02e2df700 1 -- 172.21.15.137:0/4001392204 shutdown_connections 2023-11-15T01:54:41.096 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:41.093+0000 7fb02e2df700 1 -- 172.21.15.137:0/4001392204 wait complete. 2023-11-15T01:54:41.096 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:54:41.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:54:41 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:54:40.737416+0000 mgr.smithi055.szwngw (mgr.14164) 150 : cluster [DBG] pgmap v143: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:54:41.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:54:41 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:54:41.091830+0000 mon.smithi055 (mon.0) 221 : audit [DBG] from='client.? 172.21.15.137:0/4001392204' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:54:42.471 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:54:42.472 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:54:43.617 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:43.616+0000 7f9c42ba8700 1 -- 172.21.15.137:0/928980971 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9c3c0f5410 msgr2=0x7f9c3c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:54:43.618 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:43.616+0000 7f9c42ba8700 1 --2- 172.21.15.137:0/928980971 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9c3c0f5410 0x7f9c3c0f5830 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7f9c30009fb0 tx=0x7f9c30009b30).stop 2023-11-15T01:54:43.618 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:43.616+0000 7f9c42ba8700 1 -- 172.21.15.137:0/928980971 shutdown_connections 2023-11-15T01:54:43.618 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:43.616+0000 7f9c42ba8700 1 --2- 172.21.15.137:0/928980971 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9c3c0f5410 0x7f9c3c0f5830 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:54:43.618 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:43.616+0000 7f9c42ba8700 1 -- 172.21.15.137:0/928980971 >> 172.21.15.137:0/928980971 conn(0x7f9c3c0f0b70 msgr2=0x7f9c3c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:54:43.619 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:43.616+0000 7f9c42ba8700 1 -- 172.21.15.137:0/928980971 shutdown_connections 2023-11-15T01:54:43.619 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:43.616+0000 7f9c42ba8700 1 -- 172.21.15.137:0/928980971 wait complete. 2023-11-15T01:54:43.619 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:43.617+0000 7f9c42ba8700 1 Processor -- start 2023-11-15T01:54:43.619 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:43.617+0000 7f9c42ba8700 1 -- start start 2023-11-15T01:54:43.621 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:43.617+0000 7f9c42ba8700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9c3c0f5410 0x7f9c3c105600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:54:43.621 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:43.617+0000 7f9c42ba8700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c3c105af0 con 0x7f9c3c0f5410 2023-11-15T01:54:43.622 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:43.618+0000 7f9c40944700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9c3c0f5410 0x7f9c3c105600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:54:43.622 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:43.618+0000 7f9c40944700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9c3c0f5410 0x7f9c3c105600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:40602/0 (socket says 172.21.15.137:40602) 2023-11-15T01:54:43.622 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:43.618+0000 7f9c40944700 1 -- 172.21.15.137:0/2849786630 learned_addr learned my addr 172.21.15.137:0/2849786630 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:54:43.622 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:43.619+0000 7f9c40944700 1 -- 172.21.15.137:0/2849786630 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c30005040 con 0x7f9c3c0f5410 2023-11-15T01:54:43.622 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:43.619+0000 7f9c40944700 1 --2- 172.21.15.137:0/2849786630 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9c3c0f5410 0x7f9c3c105600 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f9c30014000 tx=0x7f9c3000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:54:43.623 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:43.619+0000 7f9c397fa700 1 -- 172.21.15.137:0/2849786630 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c300095f0 con 0x7f9c3c0f5410 2023-11-15T01:54:43.623 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:43.619+0000 7f9c397fa700 1 -- 172.21.15.137:0/2849786630 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9c30007e70 con 0x7f9c3c0f5410 2023-11-15T01:54:43.623 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:43.619+0000 7f9c397fa700 1 -- 172.21.15.137:0/2849786630 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c30014ea0 con 0x7f9c3c0f5410 2023-11-15T01:54:43.623 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:43.619+0000 7f9c42ba8700 1 -- 172.21.15.137:0/2849786630 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c3c105cf0 con 0x7f9c3c0f5410 2023-11-15T01:54:43.623 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:43.619+0000 7f9c42ba8700 1 -- 172.21.15.137:0/2849786630 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c3c106180 con 0x7f9c3c0f5410 2023-11-15T01:54:43.624 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:43.620+0000 7f9c397fa700 1 -- 172.21.15.137:0/2849786630 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9c30025070 con 0x7f9c3c0f5410 2023-11-15T01:54:43.624 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:43.620+0000 7f9c42ba8700 1 -- 172.21.15.137:0/2849786630 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c3c045ad0 con 0x7f9c3c0f5410 2023-11-15T01:54:43.624 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:43.620+0000 7f9c397fa700 1 --2- 172.21.15.137:0/2849786630 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f9c2c02ffb0 0x7f9c2c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:54:43.624 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:43.620+0000 7f9c397fa700 1 -- 172.21.15.137:0/2849786630 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9c300409e0 con 0x7f9c3c0f5410 2023-11-15T01:54:43.624 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:43.620+0000 7f9c3bfff700 1 --2- 172.21.15.137:0/2849786630 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f9c2c02ffb0 0x7f9c2c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:54:43.624 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:43.621+0000 7f9c3bfff700 1 --2- 172.21.15.137:0/2849786630 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f9c2c02ffb0 0x7f9c2c032470 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f9c280096a0 tx=0x7f9c28006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:54:43.625 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:43.625+0000 7f9c397fa700 1 -- 172.21.15.137:0/2849786630 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9c300079c0 con 0x7f9c3c0f5410 2023-11-15T01:54:43.832 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:43.831+0000 7f9c42ba8700 1 -- 172.21.15.137:0/2849786630 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9c3c0fae60 con 0x7f9c3c0f5410 2023-11-15T01:54:43.832 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:43.831+0000 7f9c397fa700 1 -- 172.21.15.137:0/2849786630 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9c3000f300 con 0x7f9c3c0f5410 2023-11-15T01:54:43.833 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:54:43.833 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:54:43.834 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:43.833+0000 7f9c42ba8700 1 -- 172.21.15.137:0/2849786630 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f9c2c02ffb0 msgr2=0x7f9c2c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:54:43.835 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:43.833+0000 7f9c42ba8700 1 --2- 172.21.15.137:0/2849786630 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f9c2c02ffb0 0x7f9c2c032470 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f9c280096a0 tx=0x7f9c28006b40).stop 2023-11-15T01:54:43.835 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:43.833+0000 7f9c42ba8700 1 -- 172.21.15.137:0/2849786630 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9c3c0f5410 msgr2=0x7f9c3c105600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:54:43.835 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:43.833+0000 7f9c42ba8700 1 --2- 172.21.15.137:0/2849786630 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9c3c0f5410 0x7f9c3c105600 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f9c30014000 tx=0x7f9c3000b040).stop 2023-11-15T01:54:43.835 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:43.833+0000 7f9c42ba8700 1 -- 172.21.15.137:0/2849786630 shutdown_connections 2023-11-15T01:54:43.835 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:43.833+0000 7f9c42ba8700 1 --2- 172.21.15.137:0/2849786630 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f9c2c02ffb0 0x7f9c2c032470 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:54:43.836 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:43.833+0000 7f9c42ba8700 1 --2- 172.21.15.137:0/2849786630 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9c3c0f5410 0x7f9c3c105600 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:54:43.836 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:43.833+0000 7f9c42ba8700 1 -- 172.21.15.137:0/2849786630 >> 172.21.15.137:0/2849786630 conn(0x7f9c3c0f0b70 msgr2=0x7f9c3c0f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:54:43.836 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:43.833+0000 7f9c42ba8700 1 -- 172.21.15.137:0/2849786630 shutdown_connections 2023-11-15T01:54:43.836 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:43.833+0000 7f9c42ba8700 1 -- 172.21.15.137:0/2849786630 wait complete. 2023-11-15T01:54:43.837 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:54:44.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:54:43 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:54:42.737814+0000 mgr.smithi055.szwngw (mgr.14164) 151 : cluster [DBG] pgmap v144: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:54:44.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:54:43 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:54:42.852381+0000 mon.smithi055 (mon.0) 222 : audit [INF] from='mgr.14164 172.21.15.55:0/59463371' entity='mgr.smithi055.szwngw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi055.szwngw/mirror_snapshot_schedule"}]: dispatch 2023-11-15T01:54:44.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:54:43 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:54:42.854958+0000 mon.smithi055 (mon.0) 223 : audit [INF] from='mgr.14164 172.21.15.55:0/59463371' entity='mgr.smithi055.szwngw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi055.szwngw/trash_purge_schedule"}]: dispatch 2023-11-15T01:54:45.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:54:44 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:54:43.832522+0000 mon.smithi055 (mon.0) 224 : audit [DBG] from='client.? 172.21.15.137:0/2849786630' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:54:45.225 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:54:45.225 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:54:46.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:54:45 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:54:44.738131+0000 mgr.smithi055.szwngw (mgr.14164) 152 : cluster [DBG] pgmap v145: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:54:46.401 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:46.399+0000 7f0cf8f0b700 1 -- 172.21.15.137:0/3476043108 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0cf40f6950 msgr2=0x7f0cf40f6d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:54:46.402 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:46.399+0000 7f0cf8f0b700 1 --2- 172.21.15.137:0/3476043108 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0cf40f6950 0x7f0cf40f6d70 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7f0ce8009fb0 tx=0x7f0ce8009b30).stop 2023-11-15T01:54:46.402 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:46.399+0000 7f0cf8f0b700 1 -- 172.21.15.137:0/3476043108 shutdown_connections 2023-11-15T01:54:46.402 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:46.399+0000 7f0cf8f0b700 1 --2- 172.21.15.137:0/3476043108 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0cf40f6950 0x7f0cf40f6d70 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:54:46.402 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:46.399+0000 7f0cf8f0b700 1 -- 172.21.15.137:0/3476043108 >> 172.21.15.137:0/3476043108 conn(0x7f0cf40f0260 msgr2=0x7f0cf40f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:54:46.403 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:46.399+0000 7f0cf8f0b700 1 -- 172.21.15.137:0/3476043108 shutdown_connections 2023-11-15T01:54:46.403 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:46.399+0000 7f0cf8f0b700 1 -- 172.21.15.137:0/3476043108 wait complete. 2023-11-15T01:54:46.403 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:46.400+0000 7f0cf8f0b700 1 Processor -- start 2023-11-15T01:54:46.403 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:46.400+0000 7f0cf8f0b700 1 -- start start 2023-11-15T01:54:46.403 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:46.400+0000 7f0cf8f0b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0cf40f6950 0x7f0cf410b800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:54:46.404 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:46.400+0000 7f0cf8f0b700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf41071f0 con 0x7f0cf40f6950 2023-11-15T01:54:46.404 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:46.400+0000 7f0cf259c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0cf40f6950 0x7f0cf410b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:54:46.404 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:46.400+0000 7f0cf259c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0cf40f6950 0x7f0cf410b800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:40624/0 (socket says 172.21.15.137:40624) 2023-11-15T01:54:46.404 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:46.400+0000 7f0cf259c700 1 -- 172.21.15.137:0/3829526015 learned_addr learned my addr 172.21.15.137:0/3829526015 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:54:46.404 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:46.401+0000 7f0cf259c700 1 -- 172.21.15.137:0/3829526015 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ce8005040 con 0x7f0cf40f6950 2023-11-15T01:54:46.404 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:46.401+0000 7f0cf259c700 1 --2- 172.21.15.137:0/3829526015 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0cf40f6950 0x7f0cf410b800 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7f0ce8013510 tx=0x7f0ce800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:54:46.405 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:46.401+0000 7f0ce2ffd700 1 -- 172.21.15.137:0/3829526015 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0ce8009320 con 0x7f0cf40f6950 2023-11-15T01:54:46.405 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:46.401+0000 7f0cf8f0b700 1 -- 172.21.15.137:0/3829526015 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cf40f51d0 con 0x7f0cf40f6950 2023-11-15T01:54:46.405 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:46.401+0000 7f0ce2ffd700 1 -- 172.21.15.137:0/3829526015 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0ce8013f00 con 0x7f0cf40f6950 2023-11-15T01:54:46.405 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:46.401+0000 7f0ce2ffd700 1 -- 172.21.15.137:0/3829526015 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0ce8014d90 con 0x7f0cf40f6950 2023-11-15T01:54:46.405 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:46.401+0000 7f0cf8f0b700 1 -- 172.21.15.137:0/3829526015 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cf410bcf0 con 0x7f0cf40f6950 2023-11-15T01:54:46.406 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:46.402+0000 7f0cf8f0b700 1 -- 172.21.15.137:0/3829526015 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cf4045ad0 con 0x7f0cf40f6950 2023-11-15T01:54:46.407 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:46.402+0000 7f0ce2ffd700 1 -- 172.21.15.137:0/3829526015 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0ce80095f0 con 0x7f0cf40f6950 2023-11-15T01:54:46.407 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:46.402+0000 7f0ce2ffd700 1 --2- 172.21.15.137:0/3829526015 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f0cdc02ff60 0x7f0cdc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:54:46.407 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:46.402+0000 7f0ce2ffd700 1 -- 172.21.15.137:0/3829526015 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0ce8040fc0 con 0x7f0cf40f6950 2023-11-15T01:54:46.407 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:46.402+0000 7f0cf1d9b700 1 --2- 172.21.15.137:0/3829526015 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f0cdc02ff60 0x7f0cdc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:54:46.408 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:46.403+0000 7f0cf1d9b700 1 --2- 172.21.15.137:0/3829526015 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f0cdc02ff60 0x7f0cdc032420 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f0ce40096a0 tx=0x7f0ce4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:54:46.408 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:46.407+0000 7f0ce2ffd700 1 -- 172.21.15.137:0/3829526015 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0ce8014420 con 0x7f0cf40f6950 2023-11-15T01:54:46.616 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:46.616+0000 7f0cf8f0b700 1 -- 172.21.15.137:0/3829526015 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0cf410c4e0 con 0x7f0cf40f6950 2023-11-15T01:54:46.617 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:46.617+0000 7f0ce2ffd700 1 -- 172.21.15.137:0/3829526015 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0ce8010230 con 0x7f0cf40f6950 2023-11-15T01:54:46.618 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:54:46.618 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:54:46.619 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:46.618+0000 7f0cf8f0b700 1 -- 172.21.15.137:0/3829526015 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f0cdc02ff60 msgr2=0x7f0cdc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:54:46.619 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:46.618+0000 7f0cf8f0b700 1 --2- 172.21.15.137:0/3829526015 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f0cdc02ff60 0x7f0cdc032420 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f0ce40096a0 tx=0x7f0ce4006b40).stop 2023-11-15T01:54:46.619 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:46.618+0000 7f0cf8f0b700 1 -- 172.21.15.137:0/3829526015 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0cf40f6950 msgr2=0x7f0cf410b800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:54:46.620 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:46.618+0000 7f0cf8f0b700 1 --2- 172.21.15.137:0/3829526015 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0cf40f6950 0x7f0cf410b800 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7f0ce8013510 tx=0x7f0ce800b040).stop 2023-11-15T01:54:46.620 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:46.618+0000 7f0cf8f0b700 1 -- 172.21.15.137:0/3829526015 shutdown_connections 2023-11-15T01:54:46.620 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:46.618+0000 7f0cf8f0b700 1 --2- 172.21.15.137:0/3829526015 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f0cdc02ff60 0x7f0cdc032420 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:54:46.620 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:46.618+0000 7f0cf8f0b700 1 --2- 172.21.15.137:0/3829526015 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0cf40f6950 0x7f0cf410b800 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:54:46.620 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:46.618+0000 7f0cf8f0b700 1 -- 172.21.15.137:0/3829526015 >> 172.21.15.137:0/3829526015 conn(0x7f0cf40f0260 msgr2=0x7f0cf40f0dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:54:46.621 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:46.618+0000 7f0cf8f0b700 1 -- 172.21.15.137:0/3829526015 shutdown_connections 2023-11-15T01:54:46.621 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:46.618+0000 7f0cf8f0b700 1 -- 172.21.15.137:0/3829526015 wait complete. 2023-11-15T01:54:46.621 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:54:47.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:54:46 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:54:46.617745+0000 mon.smithi055 (mon.0) 225 : audit [DBG] from='client.? 172.21.15.137:0/3829526015' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:54:48.000 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:54:48.000 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:54:48.077 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:54:47 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:54:46.738448+0000 mgr.smithi055.szwngw (mgr.14164) 153 : cluster [DBG] pgmap v146: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:54:49.105 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:49.104+0000 7fe41b59e700 1 -- 172.21.15.137:0/3813614285 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe41c0f51a0 msgr2=0x7fe41c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:54:49.106 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:49.104+0000 7fe41b59e700 1 --2- 172.21.15.137:0/3813614285 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe41c0f51a0 0x7fe41c0f55c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7fe40c009fb0 tx=0x7fe40c009b30).stop 2023-11-15T01:54:49.106 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:49.104+0000 7fe41b59e700 1 -- 172.21.15.137:0/3813614285 shutdown_connections 2023-11-15T01:54:49.106 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:49.104+0000 7fe41b59e700 1 --2- 172.21.15.137:0/3813614285 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe41c0f51a0 0x7fe41c0f55c0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:54:49.106 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:49.104+0000 7fe41b59e700 1 -- 172.21.15.137:0/3813614285 >> 172.21.15.137:0/3813614285 conn(0x7fe41c0f09a0 msgr2=0x7fe41c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:54:49.107 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:49.104+0000 7fe41b59e700 1 -- 172.21.15.137:0/3813614285 shutdown_connections 2023-11-15T01:54:49.107 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:49.104+0000 7fe41b59e700 1 -- 172.21.15.137:0/3813614285 wait complete. 2023-11-15T01:54:49.107 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:49.105+0000 7fe41b59e700 1 Processor -- start 2023-11-15T01:54:49.107 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:49.105+0000 7fe41b59e700 1 -- start start 2023-11-15T01:54:49.107 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:49.105+0000 7fe41b59e700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe41c0f51a0 0x7fe41c10b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:54:49.108 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:49.105+0000 7fe41b59e700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe41c106fc0 con 0x7fe41c0f51a0 2023-11-15T01:54:49.109 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:49.105+0000 7fe41a59c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe41c0f51a0 0x7fe41c10b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:54:49.110 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:49.105+0000 7fe41a59c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe41c0f51a0 0x7fe41c10b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:58788/0 (socket says 172.21.15.137:58788) 2023-11-15T01:54:49.110 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:49.105+0000 7fe41a59c700 1 -- 172.21.15.137:0/1074093359 learned_addr learned my addr 172.21.15.137:0/1074093359 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:54:49.110 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:49.106+0000 7fe41a59c700 1 -- 172.21.15.137:0/1074093359 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe40c005040 con 0x7fe41c0f51a0 2023-11-15T01:54:49.110 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:49.106+0000 7fe41a59c700 1 --2- 172.21.15.137:0/1074093359 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe41c0f51a0 0x7fe41c10b5d0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7fe40c009830 tx=0x7fe40c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:54:49.110 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:49.106+0000 7fe40affd700 1 -- 172.21.15.137:0/1074093359 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe40c009320 con 0x7fe41c0f51a0 2023-11-15T01:54:49.111 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:49.106+0000 7fe41b59e700 1 -- 172.21.15.137:0/1074093359 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe41c060220 con 0x7fe41c0f51a0 2023-11-15T01:54:49.111 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:49.106+0000 7fe40affd700 1 -- 172.21.15.137:0/1074093359 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe40c0136c0 con 0x7fe41c0f51a0 2023-11-15T01:54:49.111 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:49.106+0000 7fe40affd700 1 -- 172.21.15.137:0/1074093359 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe40c01ee90 con 0x7fe41c0f51a0 2023-11-15T01:54:49.111 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:49.106+0000 7fe41b59e700 1 -- 172.21.15.137:0/1074093359 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe41c10bac0 con 0x7fe41c0f51a0 2023-11-15T01:54:49.111 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:49.107+0000 7fe40affd700 1 -- 172.21.15.137:0/1074093359 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe40c01e420 con 0x7fe41c0f51a0 2023-11-15T01:54:49.112 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:49.107+0000 7fe41b59e700 1 -- 172.21.15.137:0/1074093359 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe41c045ad0 con 0x7fe41c0f51a0 2023-11-15T01:54:49.112 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:49.107+0000 7fe40affd700 1 --2- 172.21.15.137:0/1074093359 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fe40402ff10 0x7fe4040323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:54:49.112 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:49.107+0000 7fe40affd700 1 -- 172.21.15.137:0/1074093359 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe40c040f40 con 0x7fe41c0f51a0 2023-11-15T01:54:49.112 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:49.107+0000 7fe419d9b700 1 --2- 172.21.15.137:0/1074093359 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fe40402ff10 0x7fe4040323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:54:49.112 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:49.108+0000 7fe419d9b700 1 --2- 172.21.15.137:0/1074093359 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fe40402ff10 0x7fe4040323d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fe4100096a0 tx=0x7fe410006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:54:49.113 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:49.112+0000 7fe40affd700 1 -- 172.21.15.137:0/1074093359 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe40c06d2b0 con 0x7fe41c0f51a0 2023-11-15T01:54:49.316 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:49.316+0000 7fe41b59e700 1 -- 172.21.15.137:0/1074093359 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe41c10c280 con 0x7fe41c0f51a0 2023-11-15T01:54:49.317 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:49.317+0000 7fe40affd700 1 -- 172.21.15.137:0/1074093359 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe40c0095f0 con 0x7fe41c0f51a0 2023-11-15T01:54:49.318 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:54:49.318 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:54:49.319 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:49.318+0000 7fe41b59e700 1 -- 172.21.15.137:0/1074093359 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fe40402ff10 msgr2=0x7fe4040323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:54:49.319 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:49.318+0000 7fe41b59e700 1 --2- 172.21.15.137:0/1074093359 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fe40402ff10 0x7fe4040323d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fe4100096a0 tx=0x7fe410006b40).stop 2023-11-15T01:54:49.319 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:49.318+0000 7fe41b59e700 1 -- 172.21.15.137:0/1074093359 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe41c0f51a0 msgr2=0x7fe41c10b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:54:49.319 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:49.318+0000 7fe41b59e700 1 --2- 172.21.15.137:0/1074093359 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe41c0f51a0 0x7fe41c10b5d0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7fe40c009830 tx=0x7fe40c00b040).stop 2023-11-15T01:54:49.320 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:49.318+0000 7fe41b59e700 1 -- 172.21.15.137:0/1074093359 shutdown_connections 2023-11-15T01:54:49.320 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:49.318+0000 7fe41b59e700 1 --2- 172.21.15.137:0/1074093359 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fe40402ff10 0x7fe4040323d0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:54:49.320 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:49.318+0000 7fe41b59e700 1 --2- 172.21.15.137:0/1074093359 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe41c0f51a0 0x7fe41c10b5d0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:54:49.320 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:49.318+0000 7fe41b59e700 1 -- 172.21.15.137:0/1074093359 >> 172.21.15.137:0/1074093359 conn(0x7fe41c0f09a0 msgr2=0x7fe41c0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:54:49.320 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:49.318+0000 7fe41b59e700 1 -- 172.21.15.137:0/1074093359 shutdown_connections 2023-11-15T01:54:49.320 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:49.318+0000 7fe41b59e700 1 -- 172.21.15.137:0/1074093359 wait complete. 2023-11-15T01:54:49.321 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:54:50.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:54:49 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:54:48.738775+0000 mgr.smithi055.szwngw (mgr.14164) 154 : cluster [DBG] pgmap v147: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:54:50.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:54:49 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:54:49.317757+0000 mon.smithi055 (mon.0) 226 : audit [DBG] from='client.? 172.21.15.137:0/1074093359' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:54:50.703 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:54:50.704 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:54:51.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:54:51 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:54:50.739113+0000 mgr.smithi055.szwngw (mgr.14164) 155 : cluster [DBG] pgmap v148: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:54:51.863 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:51.861+0000 7ff78464c700 1 -- 172.21.15.137:0/2377435374 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff77c0f5410 msgr2=0x7ff77c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:54:51.863 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:51.861+0000 7ff78464c700 1 --2- 172.21.15.137:0/2377435374 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff77c0f5410 0x7ff77c0f5830 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7ff76c009fb0 tx=0x7ff76c009b30).stop 2023-11-15T01:54:51.864 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:51.861+0000 7ff78464c700 1 -- 172.21.15.137:0/2377435374 shutdown_connections 2023-11-15T01:54:51.864 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:51.861+0000 7ff78464c700 1 --2- 172.21.15.137:0/2377435374 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff77c0f5410 0x7ff77c0f5830 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:54:51.864 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:51.861+0000 7ff78464c700 1 -- 172.21.15.137:0/2377435374 >> 172.21.15.137:0/2377435374 conn(0x7ff77c0f0b70 msgr2=0x7ff77c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:54:51.864 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:51.861+0000 7ff78464c700 1 -- 172.21.15.137:0/2377435374 shutdown_connections 2023-11-15T01:54:51.864 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:51.861+0000 7ff78464c700 1 -- 172.21.15.137:0/2377435374 wait complete. 2023-11-15T01:54:51.865 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:51.862+0000 7ff78464c700 1 Processor -- start 2023-11-15T01:54:51.865 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:51.862+0000 7ff78464c700 1 -- start start 2023-11-15T01:54:51.865 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:51.862+0000 7ff78464c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff77c0f5410 0x7ff77c105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:54:51.865 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:51.862+0000 7ff78464c700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff77c105b20 con 0x7ff77c0f5410 2023-11-15T01:54:51.865 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:51.863+0000 7ff7823e8700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff77c0f5410 0x7ff77c105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:54:51.866 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:51.863+0000 7ff7823e8700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff77c0f5410 0x7ff77c105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:58798/0 (socket says 172.21.15.137:58798) 2023-11-15T01:54:51.866 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:51.863+0000 7ff7823e8700 1 -- 172.21.15.137:0/934283147 learned_addr learned my addr 172.21.15.137:0/934283147 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:54:51.867 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:51.863+0000 7ff7823e8700 1 -- 172.21.15.137:0/934283147 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff76c005040 con 0x7ff77c0f5410 2023-11-15T01:54:51.867 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:51.863+0000 7ff7823e8700 1 --2- 172.21.15.137:0/934283147 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff77c0f5410 0x7ff77c105630 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7ff76c013510 tx=0x7ff76c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:54:51.868 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:51.863+0000 7ff772ffd700 1 -- 172.21.15.137:0/934283147 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff76c009320 con 0x7ff77c0f5410 2023-11-15T01:54:51.868 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:51.863+0000 7ff78464c700 1 -- 172.21.15.137:0/934283147 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff77c105d20 con 0x7ff77c0f5410 2023-11-15T01:54:51.868 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:51.863+0000 7ff772ffd700 1 -- 172.21.15.137:0/934283147 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff76c013f00 con 0x7ff77c0f5410 2023-11-15T01:54:51.868 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:51.863+0000 7ff772ffd700 1 -- 172.21.15.137:0/934283147 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff76c014d90 con 0x7ff77c0f5410 2023-11-15T01:54:51.868 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:51.863+0000 7ff78464c700 1 -- 172.21.15.137:0/934283147 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff77c1061b0 con 0x7ff77c0f5410 2023-11-15T01:54:51.869 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:51.864+0000 7ff772ffd700 1 -- 172.21.15.137:0/934283147 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff76c0095f0 con 0x7ff77c0f5410 2023-11-15T01:54:51.869 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:51.864+0000 7ff78464c700 1 -- 172.21.15.137:0/934283147 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff77c045ad0 con 0x7ff77c0f5410 2023-11-15T01:54:51.869 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:51.864+0000 7ff772ffd700 1 --2- 172.21.15.137:0/934283147 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7ff76802ff60 0x7ff768032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:54:51.869 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:51.864+0000 7ff772ffd700 1 -- 172.21.15.137:0/934283147 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff76c041090 con 0x7ff77c0f5410 2023-11-15T01:54:51.869 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:51.864+0000 7ff781be7700 1 --2- 172.21.15.137:0/934283147 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7ff76802ff60 0x7ff768032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:54:51.870 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:51.865+0000 7ff781be7700 1 --2- 172.21.15.137:0/934283147 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7ff76802ff60 0x7ff768032420 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7ff7780096a0 tx=0x7ff778006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:54:51.870 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:51.869+0000 7ff772ffd700 1 -- 172.21.15.137:0/934283147 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff76c014420 con 0x7ff77c0f5410 2023-11-15T01:54:52.076 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:52.076+0000 7ff78464c700 1 -- 172.21.15.137:0/934283147 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff77c0fae60 con 0x7ff77c0f5410 2023-11-15T01:54:52.077 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:52.076+0000 7ff772ffd700 1 -- 172.21.15.137:0/934283147 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff76c01d7f0 con 0x7ff77c0f5410 2023-11-15T01:54:52.078 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:54:52.078 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:54:52.079 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:52.078+0000 7ff78464c700 1 -- 172.21.15.137:0/934283147 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7ff76802ff60 msgr2=0x7ff768032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:54:52.080 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:52.078+0000 7ff78464c700 1 --2- 172.21.15.137:0/934283147 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7ff76802ff60 0x7ff768032420 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7ff7780096a0 tx=0x7ff778006b40).stop 2023-11-15T01:54:52.080 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:52.078+0000 7ff78464c700 1 -- 172.21.15.137:0/934283147 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff77c0f5410 msgr2=0x7ff77c105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:54:52.080 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:52.078+0000 7ff78464c700 1 --2- 172.21.15.137:0/934283147 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff77c0f5410 0x7ff77c105630 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7ff76c013510 tx=0x7ff76c00b040).stop 2023-11-15T01:54:52.080 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:52.078+0000 7ff78464c700 1 -- 172.21.15.137:0/934283147 shutdown_connections 2023-11-15T01:54:52.081 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:52.078+0000 7ff78464c700 1 --2- 172.21.15.137:0/934283147 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7ff76802ff60 0x7ff768032420 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:54:52.081 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:52.078+0000 7ff78464c700 1 --2- 172.21.15.137:0/934283147 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff77c0f5410 0x7ff77c105630 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:54:52.081 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:52.078+0000 7ff78464c700 1 -- 172.21.15.137:0/934283147 >> 172.21.15.137:0/934283147 conn(0x7ff77c0f0b70 msgr2=0x7ff77c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:54:52.081 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:52.078+0000 7ff78464c700 1 -- 172.21.15.137:0/934283147 shutdown_connections 2023-11-15T01:54:52.081 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:52.078+0000 7ff78464c700 1 -- 172.21.15.137:0/934283147 wait complete. 2023-11-15T01:54:52.082 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:54:52.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:54:52 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:54:52.077607+0000 mon.smithi055 (mon.0) 227 : audit [DBG] from='client.? 172.21.15.137:0/934283147' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:54:53.465 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:54:53.466 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:54:53.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:54:53 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:54:52.739486+0000 mgr.smithi055.szwngw (mgr.14164) 156 : cluster [DBG] pgmap v149: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:54:54.607 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:54.605+0000 7fb433c3f700 1 -- 172.21.15.137:0/2834792745 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb42c0f5410 msgr2=0x7fb42c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:54:54.608 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:54.605+0000 7fb433c3f700 1 --2- 172.21.15.137:0/2834792745 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb42c0f5410 0x7fb42c0f5830 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7fb420009fb0 tx=0x7fb420009b30).stop 2023-11-15T01:54:54.608 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:54.605+0000 7fb433c3f700 1 -- 172.21.15.137:0/2834792745 shutdown_connections 2023-11-15T01:54:54.608 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:54.605+0000 7fb433c3f700 1 --2- 172.21.15.137:0/2834792745 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb42c0f5410 0x7fb42c0f5830 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:54:54.609 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:54.605+0000 7fb433c3f700 1 -- 172.21.15.137:0/2834792745 >> 172.21.15.137:0/2834792745 conn(0x7fb42c0f0b70 msgr2=0x7fb42c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:54:54.609 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:54.605+0000 7fb433c3f700 1 -- 172.21.15.137:0/2834792745 shutdown_connections 2023-11-15T01:54:54.609 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:54.605+0000 7fb433c3f700 1 -- 172.21.15.137:0/2834792745 wait complete. 2023-11-15T01:54:54.609 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:54.606+0000 7fb433c3f700 1 Processor -- start 2023-11-15T01:54:54.609 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:54.606+0000 7fb433c3f700 1 -- start start 2023-11-15T01:54:54.610 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:54.606+0000 7fb433c3f700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb42c0f5410 0x7fb42c106710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:54:54.610 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:54.606+0000 7fb433c3f700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb42c106c00 con 0x7fb42c0f5410 2023-11-15T01:54:54.610 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:54.606+0000 7fb4319db700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb42c0f5410 0x7fb42c106710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:54:54.611 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:54.606+0000 7fb4319db700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb42c0f5410 0x7fb42c106710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:58828/0 (socket says 172.21.15.137:58828) 2023-11-15T01:54:54.611 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:54.606+0000 7fb4319db700 1 -- 172.21.15.137:0/2378956265 learned_addr learned my addr 172.21.15.137:0/2378956265 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:54:54.611 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:54.607+0000 7fb4319db700 1 -- 172.21.15.137:0/2378956265 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb420005040 con 0x7fb42c0f5410 2023-11-15T01:54:54.611 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:54.607+0000 7fb4319db700 1 --2- 172.21.15.137:0/2378956265 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb42c0f5410 0x7fb42c106710 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7fb420014000 tx=0x7fb42000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:54:54.612 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:54.607+0000 7fb41e7fc700 1 -- 172.21.15.137:0/2378956265 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb4200095f0 con 0x7fb42c0f5410 2023-11-15T01:54:54.612 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:54.607+0000 7fb433c3f700 1 -- 172.21.15.137:0/2378956265 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb42c106e00 con 0x7fb42c0f5410 2023-11-15T01:54:54.612 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:54.607+0000 7fb41e7fc700 1 -- 172.21.15.137:0/2378956265 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb420007e70 con 0x7fb42c0f5410 2023-11-15T01:54:54.612 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:54.607+0000 7fb41e7fc700 1 -- 172.21.15.137:0/2378956265 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb420014ea0 con 0x7fb42c0f5410 2023-11-15T01:54:54.613 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:54.607+0000 7fb433c3f700 1 -- 172.21.15.137:0/2378956265 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb42c107290 con 0x7fb42c0f5410 2023-11-15T01:54:54.614 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:54.608+0000 7fb41e7fc700 1 -- 172.21.15.137:0/2378956265 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb420025070 con 0x7fb42c0f5410 2023-11-15T01:54:54.614 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:54.608+0000 7fb433c3f700 1 -- 172.21.15.137:0/2378956265 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb42c045ad0 con 0x7fb42c0f5410 2023-11-15T01:54:54.614 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:54.608+0000 7fb41e7fc700 1 --2- 172.21.15.137:0/2378956265 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fb41802ff60 0x7fb418032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:54:54.614 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:54.608+0000 7fb41e7fc700 1 -- 172.21.15.137:0/2378956265 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb420040a30 con 0x7fb42c0f5410 2023-11-15T01:54:54.615 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:54.608+0000 7fb4311da700 1 --2- 172.21.15.137:0/2378956265 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fb41802ff60 0x7fb418032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:54:54.615 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:54.608+0000 7fb4311da700 1 --2- 172.21.15.137:0/2378956265 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fb41802ff60 0x7fb418032420 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fb4280096a0 tx=0x7fb428006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:54:54.615 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:54.613+0000 7fb41e7fc700 1 -- 172.21.15.137:0/2378956265 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb4200079c0 con 0x7fb42c0f5410 2023-11-15T01:54:54.818 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:54.818+0000 7fb433c3f700 1 -- 172.21.15.137:0/2378956265 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb42c108140 con 0x7fb42c0f5410 2023-11-15T01:54:54.819 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:54.818+0000 7fb41e7fc700 1 -- 172.21.15.137:0/2378956265 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb42000f300 con 0x7fb42c0f5410 2023-11-15T01:54:54.819 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:54:54.820 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:54:54.820 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:54.820+0000 7fb433c3f700 1 -- 172.21.15.137:0/2378956265 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fb41802ff60 msgr2=0x7fb418032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:54:54.821 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:54.820+0000 7fb433c3f700 1 --2- 172.21.15.137:0/2378956265 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fb41802ff60 0x7fb418032420 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fb4280096a0 tx=0x7fb428006b40).stop 2023-11-15T01:54:54.821 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:54.820+0000 7fb433c3f700 1 -- 172.21.15.137:0/2378956265 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb42c0f5410 msgr2=0x7fb42c106710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:54:54.821 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:54.820+0000 7fb433c3f700 1 --2- 172.21.15.137:0/2378956265 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb42c0f5410 0x7fb42c106710 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7fb420014000 tx=0x7fb42000b040).stop 2023-11-15T01:54:54.821 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:54.820+0000 7fb433c3f700 1 -- 172.21.15.137:0/2378956265 shutdown_connections 2023-11-15T01:54:54.822 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:54.820+0000 7fb433c3f700 1 --2- 172.21.15.137:0/2378956265 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fb41802ff60 0x7fb418032420 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:54:54.822 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:54.820+0000 7fb433c3f700 1 --2- 172.21.15.137:0/2378956265 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb42c0f5410 0x7fb42c106710 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:54:54.822 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:54.820+0000 7fb433c3f700 1 -- 172.21.15.137:0/2378956265 >> 172.21.15.137:0/2378956265 conn(0x7fb42c0f0b70 msgr2=0x7fb42c05fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:54:54.822 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:54.820+0000 7fb433c3f700 1 -- 172.21.15.137:0/2378956265 shutdown_connections 2023-11-15T01:54:54.822 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:54.820+0000 7fb433c3f700 1 -- 172.21.15.137:0/2378956265 wait complete. 2023-11-15T01:54:54.823 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:54:56.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:54:55 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:54:54.739766+0000 mgr.smithi055.szwngw (mgr.14164) 157 : cluster [DBG] pgmap v150: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:54:56.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:54:55 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:54:54.819405+0000 mon.smithi055 (mon.0) 228 : audit [DBG] from='client.? 172.21.15.137:0/2378956265' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:54:56.214 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:54:56.215 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:54:57.380 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:57.378+0000 7fa38759e700 1 -- 172.21.15.137:0/3547791976 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa3880f62a0 msgr2=0x7fa3880f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:54:57.380 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:57.378+0000 7fa38759e700 1 --2- 172.21.15.137:0/3547791976 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa3880f62a0 0x7fa3880f66c0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7fa37c009fb0 tx=0x7fa37c009b30).stop 2023-11-15T01:54:57.380 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:57.378+0000 7fa38759e700 1 -- 172.21.15.137:0/3547791976 shutdown_connections 2023-11-15T01:54:57.381 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:57.378+0000 7fa38759e700 1 --2- 172.21.15.137:0/3547791976 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa3880f62a0 0x7fa3880f66c0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:54:57.381 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:57.378+0000 7fa38759e700 1 -- 172.21.15.137:0/3547791976 >> 172.21.15.137:0/3547791976 conn(0x7fa3880f1aa0 msgr2=0x7fa3880f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:54:57.381 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:57.379+0000 7fa38759e700 1 -- 172.21.15.137:0/3547791976 shutdown_connections 2023-11-15T01:54:57.381 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:57.379+0000 7fa38759e700 1 -- 172.21.15.137:0/3547791976 wait complete. 2023-11-15T01:54:57.381 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:57.379+0000 7fa38759e700 1 Processor -- start 2023-11-15T01:54:57.382 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:57.379+0000 7fa38759e700 1 -- start start 2023-11-15T01:54:57.382 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:57.379+0000 7fa38759e700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa3880f62a0 0x7fa3881064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:54:57.382 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:57.379+0000 7fa38759e700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3881069a0 con 0x7fa3880f62a0 2023-11-15T01:54:57.383 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:57.380+0000 7fa38659c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa3880f62a0 0x7fa3881064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:54:57.384 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:57.380+0000 7fa38659c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa3880f62a0 0x7fa3881064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:58860/0 (socket says 172.21.15.137:58860) 2023-11-15T01:54:57.384 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:57.380+0000 7fa38659c700 1 -- 172.21.15.137:0/3854250065 learned_addr learned my addr 172.21.15.137:0/3854250065 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:54:57.384 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:57.381+0000 7fa38659c700 1 -- 172.21.15.137:0/3854250065 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa37c005040 con 0x7fa3880f62a0 2023-11-15T01:54:57.384 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:57.382+0000 7fa38659c700 1 --2- 172.21.15.137:0/3854250065 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa3880f62a0 0x7fa3881064b0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7fa37c013510 tx=0x7fa37c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:54:57.384 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:57.382+0000 7fa376ffd700 1 -- 172.21.15.137:0/3854250065 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa37c009320 con 0x7fa3880f62a0 2023-11-15T01:54:57.385 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:57.382+0000 7fa376ffd700 1 -- 172.21.15.137:0/3854250065 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa37c013f00 con 0x7fa3880f62a0 2023-11-15T01:54:57.385 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:57.382+0000 7fa38759e700 1 -- 172.21.15.137:0/3854250065 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa388106ba0 con 0x7fa3880f62a0 2023-11-15T01:54:57.385 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:57.382+0000 7fa376ffd700 1 -- 172.21.15.137:0/3854250065 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa37c014dc0 con 0x7fa3880f62a0 2023-11-15T01:54:57.385 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:57.382+0000 7fa38759e700 1 -- 172.21.15.137:0/3854250065 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa388107030 con 0x7fa3880f62a0 2023-11-15T01:54:57.386 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:57.383+0000 7fa38759e700 1 -- 172.21.15.137:0/3854250065 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa388045ad0 con 0x7fa3880f62a0 2023-11-15T01:54:57.387 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:57.383+0000 7fa376ffd700 1 -- 172.21.15.137:0/3854250065 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa37c0095f0 con 0x7fa3880f62a0 2023-11-15T01:54:57.387 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:57.383+0000 7fa376ffd700 1 --2- 172.21.15.137:0/3854250065 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fa370030000 0x7fa3700324c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:54:57.387 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:57.384+0000 7fa376ffd700 1 -- 172.21.15.137:0/3854250065 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa37c040f90 con 0x7fa3880f62a0 2023-11-15T01:54:57.387 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:57.384+0000 7fa385d9b700 1 --2- 172.21.15.137:0/3854250065 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fa370030000 0x7fa3700324c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:54:57.388 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:57.384+0000 7fa385d9b700 1 --2- 172.21.15.137:0/3854250065 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fa370030000 0x7fa3700324c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7fa3780096a0 tx=0x7fa378006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:54:57.389 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:57.388+0000 7fa376ffd700 1 -- 172.21.15.137:0/3854250065 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa37c014420 con 0x7fa3880f62a0 2023-11-15T01:54:57.595 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:57.594+0000 7fa38759e700 1 -- 172.21.15.137:0/3854250065 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa388107ee0 con 0x7fa3880f62a0 2023-11-15T01:54:57.596 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:57.595+0000 7fa376ffd700 1 -- 172.21.15.137:0/3854250065 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa37c010230 con 0x7fa3880f62a0 2023-11-15T01:54:57.596 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:54:57.597 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:54:57.598 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:57.597+0000 7fa38759e700 1 -- 172.21.15.137:0/3854250065 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fa370030000 msgr2=0x7fa3700324c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:54:57.598 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:57.597+0000 7fa38759e700 1 --2- 172.21.15.137:0/3854250065 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fa370030000 0x7fa3700324c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7fa3780096a0 tx=0x7fa378006b40).stop 2023-11-15T01:54:57.598 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:57.597+0000 7fa38759e700 1 -- 172.21.15.137:0/3854250065 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa3880f62a0 msgr2=0x7fa3881064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:54:57.598 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:57.597+0000 7fa38759e700 1 --2- 172.21.15.137:0/3854250065 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa3880f62a0 0x7fa3881064b0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7fa37c013510 tx=0x7fa37c00b040).stop 2023-11-15T01:54:57.598 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:57.597+0000 7fa38759e700 1 -- 172.21.15.137:0/3854250065 shutdown_connections 2023-11-15T01:54:57.599 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:57.597+0000 7fa38759e700 1 --2- 172.21.15.137:0/3854250065 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fa370030000 0x7fa3700324c0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:54:57.599 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:57.597+0000 7fa38759e700 1 --2- 172.21.15.137:0/3854250065 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa3880f62a0 0x7fa3881064b0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:54:57.599 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:57.597+0000 7fa38759e700 1 -- 172.21.15.137:0/3854250065 >> 172.21.15.137:0/3854250065 conn(0x7fa3880f1aa0 msgr2=0x7fa3880f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:54:57.599 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:57.597+0000 7fa38759e700 1 -- 172.21.15.137:0/3854250065 shutdown_connections 2023-11-15T01:54:57.599 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:54:57.597+0000 7fa38759e700 1 -- 172.21.15.137:0/3854250065 wait complete. 2023-11-15T01:54:57.600 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:54:58.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:54:57 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:54:56.740098+0000 mgr.smithi055.szwngw (mgr.14164) 158 : cluster [DBG] pgmap v151: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:54:58.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:54:57 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:54:57.596167+0000 mon.smithi055 (mon.0) 229 : audit [DBG] from='client.? 172.21.15.137:0/3854250065' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:54:58.977 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:54:58.978 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:55:00.072 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:00.069+0000 7f2fc4e94700 1 -- 172.21.15.137:0/3785293512 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2fc00f51a0 msgr2=0x7f2fc00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:55:00.072 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:00.069+0000 7f2fc4e94700 1 --2- 172.21.15.137:0/3785293512 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2fc00f51a0 0x7f2fc00f55c0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7f2fb0009fb0 tx=0x7f2fb0009b30).stop 2023-11-15T01:55:00.072 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:00.069+0000 7f2fc4e94700 1 -- 172.21.15.137:0/3785293512 shutdown_connections 2023-11-15T01:55:00.072 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:00.069+0000 7f2fc4e94700 1 --2- 172.21.15.137:0/3785293512 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2fc00f51a0 0x7f2fc00f55c0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:55:00.073 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:00.069+0000 7f2fc4e94700 1 -- 172.21.15.137:0/3785293512 >> 172.21.15.137:0/3785293512 conn(0x7f2fc00f09a0 msgr2=0x7f2fc00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:55:00.073 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:00.069+0000 7f2fc4e94700 1 -- 172.21.15.137:0/3785293512 shutdown_connections 2023-11-15T01:55:00.073 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:00.069+0000 7f2fc4e94700 1 -- 172.21.15.137:0/3785293512 wait complete. 2023-11-15T01:55:00.073 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:00.069+0000 7f2fc4e94700 1 Processor -- start 2023-11-15T01:55:00.073 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:00.070+0000 7f2fc4e94700 1 -- start start 2023-11-15T01:55:00.074 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:00.070+0000 7f2fc4e94700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2fc00f51a0 0x7f2fc01091e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:55:00.074 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:00.070+0000 7f2fc4e94700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc0060240 con 0x7f2fc00f51a0 2023-11-15T01:55:00.074 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:00.070+0000 7f2fbf7fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2fc00f51a0 0x7f2fc01091e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:55:00.074 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:00.070+0000 7f2fbf7fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2fc00f51a0 0x7f2fc01091e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:54786/0 (socket says 172.21.15.137:54786) 2023-11-15T01:55:00.074 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:00.070+0000 7f2fbf7fe700 1 -- 172.21.15.137:0/486014709 learned_addr learned my addr 172.21.15.137:0/486014709 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:55:00.075 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:00.071+0000 7f2fbf7fe700 1 -- 172.21.15.137:0/486014709 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fb0005040 con 0x7f2fc00f51a0 2023-11-15T01:55:00.075 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:00.071+0000 7f2fbf7fe700 1 --2- 172.21.15.137:0/486014709 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2fc00f51a0 0x7f2fc01091e0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f2fb0009830 tx=0x7f2fb000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:55:00.075 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:00.071+0000 7f2fa7fff700 1 -- 172.21.15.137:0/486014709 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2fb0009320 con 0x7f2fc00f51a0 2023-11-15T01:55:00.075 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:00.071+0000 7f2fc4e94700 1 -- 172.21.15.137:0/486014709 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fc0060440 con 0x7f2fc00f51a0 2023-11-15T01:55:00.076 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:00.071+0000 7f2fa7fff700 1 -- 172.21.15.137:0/486014709 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2fb00136c0 con 0x7f2fc00f51a0 2023-11-15T01:55:00.076 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:00.071+0000 7f2fa7fff700 1 -- 172.21.15.137:0/486014709 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2fb001ee90 con 0x7f2fc00f51a0 2023-11-15T01:55:00.076 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:00.071+0000 7f2fc4e94700 1 -- 172.21.15.137:0/486014709 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fc0109890 con 0x7f2fc00f51a0 2023-11-15T01:55:00.077 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:00.072+0000 7f2fa7fff700 1 -- 172.21.15.137:0/486014709 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2fb001e420 con 0x7f2fc00f51a0 2023-11-15T01:55:00.077 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:00.072+0000 7f2fc4e94700 1 -- 172.21.15.137:0/486014709 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fc0045ad0 con 0x7f2fc00f51a0 2023-11-15T01:55:00.078 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:00.072+0000 7f2fa7fff700 1 --2- 172.21.15.137:0/486014709 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f2fa802ff10 0x7f2fa80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:55:00.078 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:00.072+0000 7f2fa7fff700 1 -- 172.21.15.137:0/486014709 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2fb0040f40 con 0x7f2fc00f51a0 2023-11-15T01:55:00.078 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:00.072+0000 7f2fbeffd700 1 --2- 172.21.15.137:0/486014709 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f2fa802ff10 0x7f2fa80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:55:00.078 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:00.072+0000 7f2fbeffd700 1 --2- 172.21.15.137:0/486014709 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f2fa802ff10 0x7f2fa80323d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f2fb40096a0 tx=0x7f2fb4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:55:00.079 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:00.077+0000 7f2fa7fff700 1 -- 172.21.15.137:0/486014709 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2fb006d2b0 con 0x7f2fc00f51a0 2023-11-15T01:55:00.079 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:54:59 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:54:58.740455+0000 mgr.smithi055.szwngw (mgr.14164) 159 : cluster [DBG] pgmap v152: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:55:00.288 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:00.287+0000 7f2fc4e94700 1 -- 172.21.15.137:0/486014709 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2fc010a0b0 con 0x7f2fc00f51a0 2023-11-15T01:55:00.288 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:00.288+0000 7f2fa7fff700 1 -- 172.21.15.137:0/486014709 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2fb00095f0 con 0x7f2fc00f51a0 2023-11-15T01:55:00.289 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:55:00.289 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:55:00.290 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:00.289+0000 7f2fc4e94700 1 -- 172.21.15.137:0/486014709 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f2fa802ff10 msgr2=0x7f2fa80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:55:00.291 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:00.289+0000 7f2fc4e94700 1 --2- 172.21.15.137:0/486014709 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f2fa802ff10 0x7f2fa80323d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f2fb40096a0 tx=0x7f2fb4006b40).stop 2023-11-15T01:55:00.291 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:00.289+0000 7f2fc4e94700 1 -- 172.21.15.137:0/486014709 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2fc00f51a0 msgr2=0x7f2fc01091e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:55:00.291 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:00.289+0000 7f2fc4e94700 1 --2- 172.21.15.137:0/486014709 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2fc00f51a0 0x7f2fc01091e0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f2fb0009830 tx=0x7f2fb000b040).stop 2023-11-15T01:55:00.291 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:00.290+0000 7f2fc4e94700 1 -- 172.21.15.137:0/486014709 shutdown_connections 2023-11-15T01:55:00.292 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:00.290+0000 7f2fc4e94700 1 --2- 172.21.15.137:0/486014709 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f2fa802ff10 0x7f2fa80323d0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:55:00.292 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:00.290+0000 7f2fc4e94700 1 --2- 172.21.15.137:0/486014709 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2fc00f51a0 0x7f2fc01091e0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:55:00.292 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:00.290+0000 7f2fc4e94700 1 -- 172.21.15.137:0/486014709 >> 172.21.15.137:0/486014709 conn(0x7f2fc00f09a0 msgr2=0x7f2fc00f13e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:55:00.292 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:00.290+0000 7f2fc4e94700 1 -- 172.21.15.137:0/486014709 shutdown_connections 2023-11-15T01:55:00.292 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:00.290+0000 7f2fc4e94700 1 -- 172.21.15.137:0/486014709 wait complete. 2023-11-15T01:55:00.293 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:55:01.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:00 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:55:00.288893+0000 mon.smithi055 (mon.0) 230 : audit [DBG] from='client.? 172.21.15.137:0/486014709' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:55:01.668 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:55:01.668 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:55:02.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:01 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:55:00.740754+0000 mgr.smithi055.szwngw (mgr.14164) 160 : cluster [DBG] pgmap v153: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:55:02.794 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:02.791+0000 7f9d968f9700 1 -- 172.21.15.137:0/1227571957 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9d900f6510 msgr2=0x7f9d900f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:55:02.794 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:02.791+0000 7f9d968f9700 1 --2- 172.21.15.137:0/1227571957 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9d900f6510 0x7f9d900f6930 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7f9d80009fb0 tx=0x7f9d80009b30).stop 2023-11-15T01:55:02.795 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:02.791+0000 7f9d968f9700 1 -- 172.21.15.137:0/1227571957 shutdown_connections 2023-11-15T01:55:02.795 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:02.791+0000 7f9d968f9700 1 --2- 172.21.15.137:0/1227571957 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9d900f6510 0x7f9d900f6930 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:55:02.795 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:02.791+0000 7f9d968f9700 1 -- 172.21.15.137:0/1227571957 >> 172.21.15.137:0/1227571957 conn(0x7f9d900f1c70 msgr2=0x7f9d900f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:55:02.795 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:02.791+0000 7f9d968f9700 1 -- 172.21.15.137:0/1227571957 shutdown_connections 2023-11-15T01:55:02.795 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:02.791+0000 7f9d968f9700 1 -- 172.21.15.137:0/1227571957 wait complete. 2023-11-15T01:55:02.796 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:02.792+0000 7f9d968f9700 1 Processor -- start 2023-11-15T01:55:02.796 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:02.792+0000 7f9d968f9700 1 -- start start 2023-11-15T01:55:02.796 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:02.792+0000 7f9d968f9700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9d900f6510 0x7f9d9006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:55:02.796 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:02.792+0000 7f9d968f9700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d90069e20 con 0x7f9d900f6510 2023-11-15T01:55:02.798 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:02.793+0000 7f9d8ffff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9d900f6510 0x7f9d9006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:55:02.798 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:02.794+0000 7f9d8ffff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9d900f6510 0x7f9d9006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:54800/0 (socket says 172.21.15.137:54800) 2023-11-15T01:55:02.798 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:02.794+0000 7f9d8ffff700 1 -- 172.21.15.137:0/1139281429 learned_addr learned my addr 172.21.15.137:0/1139281429 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:55:02.798 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:02.794+0000 7f9d8ffff700 1 -- 172.21.15.137:0/1139281429 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d80005040 con 0x7f9d900f6510 2023-11-15T01:55:02.798 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:02.795+0000 7f9d8ffff700 1 --2- 172.21.15.137:0/1139281429 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9d900f6510 0x7f9d9006b9a0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f9d80013510 tx=0x7f9d8000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:55:02.799 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:02.795+0000 7f9d8cff9700 1 -- 172.21.15.137:0/1139281429 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d80009320 con 0x7f9d900f6510 2023-11-15T01:55:02.799 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:02.795+0000 7f9d968f9700 1 -- 172.21.15.137:0/1139281429 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d90069fc0 con 0x7f9d900f6510 2023-11-15T01:55:02.799 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:02.795+0000 7f9d8cff9700 1 -- 172.21.15.137:0/1139281429 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9d80013f00 con 0x7f9d900f6510 2023-11-15T01:55:02.799 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:02.795+0000 7f9d8cff9700 1 -- 172.21.15.137:0/1139281429 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d80014d90 con 0x7f9d900f6510 2023-11-15T01:55:02.799 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:02.795+0000 7f9d968f9700 1 -- 172.21.15.137:0/1139281429 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d9006a450 con 0x7f9d900f6510 2023-11-15T01:55:02.800 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:02.796+0000 7f9d968f9700 1 -- 172.21.15.137:0/1139281429 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d90059100 con 0x7f9d900f6510 2023-11-15T01:55:02.800 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:02.796+0000 7f9d8cff9700 1 -- 172.21.15.137:0/1139281429 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9d800095f0 con 0x7f9d900f6510 2023-11-15T01:55:02.801 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:02.796+0000 7f9d8cff9700 1 --2- 172.21.15.137:0/1139281429 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f9d7802ff60 0x7f9d78032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:55:02.801 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:02.797+0000 7f9d8cff9700 1 -- 172.21.15.137:0/1139281429 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9d800418b0 con 0x7f9d900f6510 2023-11-15T01:55:02.801 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:02.797+0000 7f9d8f7fe700 1 --2- 172.21.15.137:0/1139281429 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f9d7802ff60 0x7f9d78032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:55:02.801 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:02.797+0000 7f9d8f7fe700 1 --2- 172.21.15.137:0/1139281429 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f9d7802ff60 0x7f9d78032420 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f9d840096a0 tx=0x7f9d84006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:55:02.803 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:02.801+0000 7f9d8cff9700 1 -- 172.21.15.137:0/1139281429 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9d80014420 con 0x7f9d900f6510 2023-11-15T01:55:03.009 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:03.008+0000 7f9d968f9700 1 -- 172.21.15.137:0/1139281429 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9d900fbf60 con 0x7f9d900f6510 2023-11-15T01:55:03.010 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:03.009+0000 7f9d8cff9700 1 -- 172.21.15.137:0/1139281429 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9d8001c930 con 0x7f9d900f6510 2023-11-15T01:55:03.011 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:55:03.011 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:55:03.012 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:03.010+0000 7f9d968f9700 1 -- 172.21.15.137:0/1139281429 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f9d7802ff60 msgr2=0x7f9d78032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:55:03.012 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:03.010+0000 7f9d968f9700 1 --2- 172.21.15.137:0/1139281429 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f9d7802ff60 0x7f9d78032420 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f9d840096a0 tx=0x7f9d84006b40).stop 2023-11-15T01:55:03.013 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:03.010+0000 7f9d968f9700 1 -- 172.21.15.137:0/1139281429 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9d900f6510 msgr2=0x7f9d9006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:55:03.013 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:03.010+0000 7f9d968f9700 1 --2- 172.21.15.137:0/1139281429 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9d900f6510 0x7f9d9006b9a0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f9d80013510 tx=0x7f9d8000b040).stop 2023-11-15T01:55:03.013 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:03.011+0000 7f9d968f9700 1 -- 172.21.15.137:0/1139281429 shutdown_connections 2023-11-15T01:55:03.013 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:03.011+0000 7f9d968f9700 1 --2- 172.21.15.137:0/1139281429 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f9d7802ff60 0x7f9d78032420 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:55:03.013 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:03.011+0000 7f9d968f9700 1 --2- 172.21.15.137:0/1139281429 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9d900f6510 0x7f9d9006b9a0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:55:03.014 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:03.011+0000 7f9d968f9700 1 -- 172.21.15.137:0/1139281429 >> 172.21.15.137:0/1139281429 conn(0x7f9d900f1c70 msgr2=0x7f9d900f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:55:03.014 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:03.011+0000 7f9d968f9700 1 -- 172.21.15.137:0/1139281429 shutdown_connections 2023-11-15T01:55:03.014 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:03.011+0000 7f9d968f9700 1 -- 172.21.15.137:0/1139281429 wait complete. 2023-11-15T01:55:03.014 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:55:04.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:03 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:55:02.741073+0000 mgr.smithi055.szwngw (mgr.14164) 161 : cluster [DBG] pgmap v154: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:55:04.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:03 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:55:03.010499+0000 mon.smithi055 (mon.0) 231 : audit [DBG] from='client.? 172.21.15.137:0/1139281429' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:55:04.409 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:55:04.409 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:55:05.578 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:05.575+0000 7f9912673700 1 -- 172.21.15.137:0/3956911781 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f990c0f51a0 msgr2=0x7f990c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:55:05.578 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:05.575+0000 7f9912673700 1 --2- 172.21.15.137:0/3956911781 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f990c0f51a0 0x7f990c0f55c0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7f98fc009fb0 tx=0x7f98fc009b30).stop 2023-11-15T01:55:05.578 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:05.575+0000 7f9912673700 1 -- 172.21.15.137:0/3956911781 shutdown_connections 2023-11-15T01:55:05.578 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:05.575+0000 7f9912673700 1 --2- 172.21.15.137:0/3956911781 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f990c0f51a0 0x7f990c0f55c0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:55:05.579 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:05.575+0000 7f9912673700 1 -- 172.21.15.137:0/3956911781 >> 172.21.15.137:0/3956911781 conn(0x7f990c0f09a0 msgr2=0x7f990c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:55:05.579 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:05.575+0000 7f9912673700 1 -- 172.21.15.137:0/3956911781 shutdown_connections 2023-11-15T01:55:05.579 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:05.575+0000 7f9912673700 1 -- 172.21.15.137:0/3956911781 wait complete. 2023-11-15T01:55:05.579 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:05.576+0000 7f9912673700 1 Processor -- start 2023-11-15T01:55:05.579 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:05.576+0000 7f9912673700 1 -- start start 2023-11-15T01:55:05.579 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:05.576+0000 7f9912673700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f990c0f51a0 0x7f990c1053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:55:05.580 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:05.576+0000 7f9912673700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f990c1058a0 con 0x7f990c0f51a0 2023-11-15T01:55:05.580 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:05.576+0000 7f9911671700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f990c0f51a0 0x7f990c1053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:55:05.580 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:05.576+0000 7f9911671700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f990c0f51a0 0x7f990c1053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:54820/0 (socket says 172.21.15.137:54820) 2023-11-15T01:55:05.580 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:05.576+0000 7f9911671700 1 -- 172.21.15.137:0/2389212771 learned_addr learned my addr 172.21.15.137:0/2389212771 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:55:05.582 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:05.576+0000 7f9911671700 1 -- 172.21.15.137:0/2389212771 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98fc005040 con 0x7f990c0f51a0 2023-11-15T01:55:05.582 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:05.577+0000 7f9911671700 1 --2- 172.21.15.137:0/2389212771 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f990c0f51a0 0x7f990c1053b0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f98fc013510 tx=0x7f98fc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:55:05.582 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:05.577+0000 7f9901ffb700 1 -- 172.21.15.137:0/2389212771 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f98fc009320 con 0x7f990c0f51a0 2023-11-15T01:55:05.582 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:05.577+0000 7f9912673700 1 -- 172.21.15.137:0/2389212771 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f990c105aa0 con 0x7f990c0f51a0 2023-11-15T01:55:05.583 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:05.577+0000 7f9901ffb700 1 -- 172.21.15.137:0/2389212771 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f98fc013f00 con 0x7f990c0f51a0 2023-11-15T01:55:05.583 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:05.577+0000 7f9901ffb700 1 -- 172.21.15.137:0/2389212771 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f98fc014dc0 con 0x7f990c0f51a0 2023-11-15T01:55:05.583 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:05.577+0000 7f9912673700 1 -- 172.21.15.137:0/2389212771 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f990c105f30 con 0x7f990c0f51a0 2023-11-15T01:55:05.583 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:05.578+0000 7f9912673700 1 -- 172.21.15.137:0/2389212771 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f990c045ad0 con 0x7f990c0f51a0 2023-11-15T01:55:05.583 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:05.578+0000 7f9901ffb700 1 -- 172.21.15.137:0/2389212771 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f98fc0095f0 con 0x7f990c0f51a0 2023-11-15T01:55:05.584 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:05.578+0000 7f9901ffb700 1 --2- 172.21.15.137:0/2389212771 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f98f802ff60 0x7f98f8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:55:05.584 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:05.578+0000 7f9901ffb700 1 -- 172.21.15.137:0/2389212771 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f98fc041040 con 0x7f990c0f51a0 2023-11-15T01:55:05.584 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:05.578+0000 7f9910e70700 1 --2- 172.21.15.137:0/2389212771 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f98f802ff60 0x7f98f8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:55:05.584 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:05.579+0000 7f9910e70700 1 --2- 172.21.15.137:0/2389212771 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f98f802ff60 0x7f98f8032420 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f99080096a0 tx=0x7f9908006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:55:05.585 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:05.583+0000 7f9901ffb700 1 -- 172.21.15.137:0/2389212771 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f98fc014420 con 0x7f990c0f51a0 2023-11-15T01:55:05.791 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:05.790+0000 7f9912673700 1 -- 172.21.15.137:0/2389212771 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f990c106de0 con 0x7f990c0f51a0 2023-11-15T01:55:05.792 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:05.791+0000 7f9901ffb700 1 -- 172.21.15.137:0/2389212771 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f98fc010230 con 0x7f990c0f51a0 2023-11-15T01:55:05.792 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:55:05.793 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:55:05.794 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:05.792+0000 7f9912673700 1 -- 172.21.15.137:0/2389212771 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f98f802ff60 msgr2=0x7f98f8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:55:05.794 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:05.792+0000 7f9912673700 1 --2- 172.21.15.137:0/2389212771 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f98f802ff60 0x7f98f8032420 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f99080096a0 tx=0x7f9908006b40).stop 2023-11-15T01:55:05.795 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:05.792+0000 7f9912673700 1 -- 172.21.15.137:0/2389212771 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f990c0f51a0 msgr2=0x7f990c1053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:55:05.795 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:05.792+0000 7f9912673700 1 --2- 172.21.15.137:0/2389212771 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f990c0f51a0 0x7f990c1053b0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f98fc013510 tx=0x7f98fc00b040).stop 2023-11-15T01:55:05.795 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:05.792+0000 7f9912673700 1 -- 172.21.15.137:0/2389212771 shutdown_connections 2023-11-15T01:55:05.795 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:05.792+0000 7f9912673700 1 --2- 172.21.15.137:0/2389212771 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f98f802ff60 0x7f98f8032420 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:55:05.796 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:05.792+0000 7f9912673700 1 --2- 172.21.15.137:0/2389212771 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f990c0f51a0 0x7f990c1053b0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:55:05.796 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:05.792+0000 7f9912673700 1 -- 172.21.15.137:0/2389212771 >> 172.21.15.137:0/2389212771 conn(0x7f990c0f09a0 msgr2=0x7f990c0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:55:05.796 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:05.792+0000 7f9912673700 1 -- 172.21.15.137:0/2389212771 shutdown_connections 2023-11-15T01:55:05.796 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:05.792+0000 7f9912673700 1 -- 172.21.15.137:0/2389212771 wait complete. 2023-11-15T01:55:05.796 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:55:06.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:05 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:55:04.741446+0000 mgr.smithi055.szwngw (mgr.14164) 162 : cluster [DBG] pgmap v155: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:55:06.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:05 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:55:05.792738+0000 mon.smithi055 (mon.0) 232 : audit [DBG] from='client.? 172.21.15.137:0/2389212771' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:55:07.179 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:55:07.180 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:55:08.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:07 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:55:06.741825+0000 mgr.smithi055.szwngw (mgr.14164) 163 : cluster [DBG] pgmap v156: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:55:08.335 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:08.333+0000 7f45a6d5c700 1 -- 172.21.15.137:0/9079122 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f45a00f5410 msgr2=0x7f45a00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:55:08.335 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:08.333+0000 7f45a6d5c700 1 --2- 172.21.15.137:0/9079122 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f45a00f5410 0x7f45a00f5830 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7f4594009fb0 tx=0x7f4594009b30).stop 2023-11-15T01:55:08.335 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:08.333+0000 7f45a6d5c700 1 -- 172.21.15.137:0/9079122 shutdown_connections 2023-11-15T01:55:08.335 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:08.333+0000 7f45a6d5c700 1 --2- 172.21.15.137:0/9079122 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f45a00f5410 0x7f45a00f5830 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:55:08.336 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:08.333+0000 7f45a6d5c700 1 -- 172.21.15.137:0/9079122 >> 172.21.15.137:0/9079122 conn(0x7f45a00f0b70 msgr2=0x7f45a00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:55:08.336 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:08.333+0000 7f45a6d5c700 1 -- 172.21.15.137:0/9079122 shutdown_connections 2023-11-15T01:55:08.336 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:08.333+0000 7f45a6d5c700 1 -- 172.21.15.137:0/9079122 wait complete. 2023-11-15T01:55:08.336 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:08.333+0000 7f45a6d5c700 1 Processor -- start 2023-11-15T01:55:08.337 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:08.333+0000 7f45a6d5c700 1 -- start start 2023-11-15T01:55:08.337 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:08.333+0000 7f45a6d5c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f45a00f5410 0x7f45a0106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:55:08.337 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:08.333+0000 7f45a6d5c700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45a0106c20 con 0x7f45a00f5410 2023-11-15T01:55:08.339 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:08.334+0000 7f45a4af8700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f45a00f5410 0x7f45a0106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:55:08.339 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:08.334+0000 7f45a4af8700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f45a00f5410 0x7f45a0106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:47008/0 (socket says 172.21.15.137:47008) 2023-11-15T01:55:08.339 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:08.334+0000 7f45a4af8700 1 -- 172.21.15.137:0/2472676833 learned_addr learned my addr 172.21.15.137:0/2472676833 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:55:08.339 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:08.334+0000 7f45a4af8700 1 -- 172.21.15.137:0/2472676833 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4594005040 con 0x7f45a00f5410 2023-11-15T01:55:08.340 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:08.334+0000 7f45a4af8700 1 --2- 172.21.15.137:0/2472676833 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f45a00f5410 0x7f45a0106730 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f4594013510 tx=0x7f459400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:55:08.340 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:08.334+0000 7f459d7fa700 1 -- 172.21.15.137:0/2472676833 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4594009320 con 0x7f45a00f5410 2023-11-15T01:55:08.340 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:08.334+0000 7f45a6d5c700 1 -- 172.21.15.137:0/2472676833 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45a0106e20 con 0x7f45a00f5410 2023-11-15T01:55:08.340 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:08.334+0000 7f459d7fa700 1 -- 172.21.15.137:0/2472676833 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4594013f00 con 0x7f45a00f5410 2023-11-15T01:55:08.340 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:08.334+0000 7f459d7fa700 1 -- 172.21.15.137:0/2472676833 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4594014d90 con 0x7f45a00f5410 2023-11-15T01:55:08.341 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:08.334+0000 7f45a6d5c700 1 -- 172.21.15.137:0/2472676833 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45a01072b0 con 0x7f45a00f5410 2023-11-15T01:55:08.341 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:08.335+0000 7f459d7fa700 1 -- 172.21.15.137:0/2472676833 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f45940263a0 con 0x7f45a00f5410 2023-11-15T01:55:08.341 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:08.335+0000 7f459d7fa700 1 --2- 172.21.15.137:0/2472676833 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f459002ff60 0x7f4590032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:55:08.341 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:08.335+0000 7f45a6d5c700 1 -- 172.21.15.137:0/2472676833 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45a0045ad0 con 0x7f45a00f5410 2023-11-15T01:55:08.341 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:08.335+0000 7f459d7fa700 1 -- 172.21.15.137:0/2472676833 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4594040ff0 con 0x7f45a00f5410 2023-11-15T01:55:08.342 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:08.336+0000 7f459ffff700 1 --2- 172.21.15.137:0/2472676833 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f459002ff60 0x7f4590032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:55:08.342 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:08.336+0000 7f459ffff700 1 --2- 172.21.15.137:0/2472676833 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f459002ff60 0x7f4590032420 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f458c0096a0 tx=0x7f458c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:55:08.342 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:08.340+0000 7f459d7fa700 1 -- 172.21.15.137:0/2472676833 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4594012920 con 0x7f45a00f5410 2023-11-15T01:55:08.546 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:08.545+0000 7f45a6d5c700 1 -- 172.21.15.137:0/2472676833 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f45a00faf50 con 0x7f45a00f5410 2023-11-15T01:55:08.547 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:08.546+0000 7f459d7fa700 1 -- 172.21.15.137:0/2472676833 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4594012330 con 0x7f45a00f5410 2023-11-15T01:55:08.548 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:55:08.548 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:55:08.549 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:08.547+0000 7f45a6d5c700 1 -- 172.21.15.137:0/2472676833 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f459002ff60 msgr2=0x7f4590032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:55:08.549 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:08.547+0000 7f45a6d5c700 1 --2- 172.21.15.137:0/2472676833 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f459002ff60 0x7f4590032420 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f458c0096a0 tx=0x7f458c006b40).stop 2023-11-15T01:55:08.549 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:08.547+0000 7f45a6d5c700 1 -- 172.21.15.137:0/2472676833 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f45a00f5410 msgr2=0x7f45a0106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:55:08.549 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:08.547+0000 7f45a6d5c700 1 --2- 172.21.15.137:0/2472676833 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f45a00f5410 0x7f45a0106730 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f4594013510 tx=0x7f459400b040).stop 2023-11-15T01:55:08.550 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:08.547+0000 7f45a6d5c700 1 -- 172.21.15.137:0/2472676833 shutdown_connections 2023-11-15T01:55:08.550 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:08.547+0000 7f45a6d5c700 1 --2- 172.21.15.137:0/2472676833 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f459002ff60 0x7f4590032420 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:55:08.551 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:08.547+0000 7f45a6d5c700 1 --2- 172.21.15.137:0/2472676833 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f45a00f5410 0x7f45a0106730 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:55:08.551 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:08.547+0000 7f45a6d5c700 1 -- 172.21.15.137:0/2472676833 >> 172.21.15.137:0/2472676833 conn(0x7f45a00f0b70 msgr2=0x7f45a00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:55:08.551 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:08.547+0000 7f45a6d5c700 1 -- 172.21.15.137:0/2472676833 shutdown_connections 2023-11-15T01:55:08.552 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:08.547+0000 7f45a6d5c700 1 -- 172.21.15.137:0/2472676833 wait complete. 2023-11-15T01:55:08.552 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:55:09.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:08 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:55:08.547721+0000 mon.smithi055 (mon.0) 233 : audit [DBG] from='client.? 172.21.15.137:0/2472676833' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:55:09.934 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:55:09.935 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:55:10.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:09 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:55:08.742166+0000 mgr.smithi055.szwngw (mgr.14164) 164 : cluster [DBG] pgmap v157: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:55:11.085 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:11.084+0000 7f7ce5a6b700 1 -- 172.21.15.137:0/1144747412 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7ce00f5410 msgr2=0x7f7ce00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:55:11.085 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:11.084+0000 7f7ce5a6b700 1 --2- 172.21.15.137:0/1144747412 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7ce00f5410 0x7f7ce00f5830 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7f7cd4009fb0 tx=0x7f7cd4009b30).stop 2023-11-15T01:55:11.086 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:11.084+0000 7f7ce5a6b700 1 -- 172.21.15.137:0/1144747412 shutdown_connections 2023-11-15T01:55:11.086 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:11.084+0000 7f7ce5a6b700 1 --2- 172.21.15.137:0/1144747412 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7ce00f5410 0x7f7ce00f5830 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:55:11.086 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:11.084+0000 7f7ce5a6b700 1 -- 172.21.15.137:0/1144747412 >> 172.21.15.137:0/1144747412 conn(0x7f7ce00f0b70 msgr2=0x7f7ce00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:55:11.086 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:11.084+0000 7f7ce5a6b700 1 -- 172.21.15.137:0/1144747412 shutdown_connections 2023-11-15T01:55:11.087 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:11.084+0000 7f7ce5a6b700 1 -- 172.21.15.137:0/1144747412 wait complete. 2023-11-15T01:55:11.087 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:11.085+0000 7f7ce5a6b700 1 Processor -- start 2023-11-15T01:55:11.087 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:11.085+0000 7f7ce5a6b700 1 -- start start 2023-11-15T01:55:11.089 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:11.085+0000 7f7ce5a6b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7ce00f5410 0x7f7ce00fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:55:11.089 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:11.085+0000 7f7ce5a6b700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ce00fb820 con 0x7f7ce00f5410 2023-11-15T01:55:11.089 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:11.085+0000 7f7cdf7fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7ce00f5410 0x7f7ce00fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:55:11.090 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:11.085+0000 7f7cdf7fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7ce00f5410 0x7f7ce00fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:47016/0 (socket says 172.21.15.137:47016) 2023-11-15T01:55:11.090 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:11.085+0000 7f7cdf7fe700 1 -- 172.21.15.137:0/3837452759 learned_addr learned my addr 172.21.15.137:0/3837452759 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:55:11.090 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:11.086+0000 7f7cdf7fe700 1 -- 172.21.15.137:0/3837452759 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cd4005040 con 0x7f7ce00f5410 2023-11-15T01:55:11.090 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:11.086+0000 7f7cdf7fe700 1 --2- 172.21.15.137:0/3837452759 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7ce00f5410 0x7f7ce00fd3a0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f7cd4013510 tx=0x7f7cd400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:55:11.091 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:11.086+0000 7f7cc7fff700 1 -- 172.21.15.137:0/3837452759 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7cd4009320 con 0x7f7ce00f5410 2023-11-15T01:55:11.091 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:11.086+0000 7f7ce5a6b700 1 -- 172.21.15.137:0/3837452759 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ce00fb9c0 con 0x7f7ce00f5410 2023-11-15T01:55:11.091 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:11.086+0000 7f7cc7fff700 1 -- 172.21.15.137:0/3837452759 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7cd4013f00 con 0x7f7ce00f5410 2023-11-15T01:55:11.091 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:11.086+0000 7f7cc7fff700 1 -- 172.21.15.137:0/3837452759 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7cd4014d90 con 0x7f7ce00f5410 2023-11-15T01:55:11.091 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:11.086+0000 7f7ce5a6b700 1 -- 172.21.15.137:0/3837452759 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ce00fbe50 con 0x7f7ce00f5410 2023-11-15T01:55:11.092 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:11.087+0000 7f7cc7fff700 1 -- 172.21.15.137:0/3837452759 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7cd40095f0 con 0x7f7ce00f5410 2023-11-15T01:55:11.092 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:11.087+0000 7f7ce5a6b700 1 -- 172.21.15.137:0/3837452759 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ce0059100 con 0x7f7ce00f5410 2023-11-15T01:55:11.092 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:11.087+0000 7f7cc7fff700 1 --2- 172.21.15.137:0/3837452759 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f7cc802ff60 0x7f7cc8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:55:11.092 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:11.087+0000 7f7cc7fff700 1 -- 172.21.15.137:0/3837452759 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7cd4041000 con 0x7f7ce00f5410 2023-11-15T01:55:11.092 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:11.087+0000 7f7cdeffd700 1 --2- 172.21.15.137:0/3837452759 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f7cc802ff60 0x7f7cc8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:55:11.093 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:11.088+0000 7f7cdeffd700 1 --2- 172.21.15.137:0/3837452759 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f7cc802ff60 0x7f7cc8032420 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f7cd00096a0 tx=0x7f7cd0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:55:11.093 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:11.092+0000 7f7cc7fff700 1 -- 172.21.15.137:0/3837452759 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7cd4014420 con 0x7f7ce00f5410 2023-11-15T01:55:11.298 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:11.297+0000 7f7ce5a6b700 1 -- 172.21.15.137:0/3837452759 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7ce00fae60 con 0x7f7ce00f5410 2023-11-15T01:55:11.298 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:11.298+0000 7f7cc7fff700 1 -- 172.21.15.137:0/3837452759 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7cd4010230 con 0x7f7ce00f5410 2023-11-15T01:55:11.299 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:55:11.299 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:55:11.300 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:11.299+0000 7f7ce5a6b700 1 -- 172.21.15.137:0/3837452759 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f7cc802ff60 msgr2=0x7f7cc8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:55:11.300 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:11.299+0000 7f7ce5a6b700 1 --2- 172.21.15.137:0/3837452759 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f7cc802ff60 0x7f7cc8032420 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f7cd00096a0 tx=0x7f7cd0006b40).stop 2023-11-15T01:55:11.300 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:11.299+0000 7f7ce5a6b700 1 -- 172.21.15.137:0/3837452759 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7ce00f5410 msgr2=0x7f7ce00fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:55:11.301 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:11.299+0000 7f7ce5a6b700 1 --2- 172.21.15.137:0/3837452759 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7ce00f5410 0x7f7ce00fd3a0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f7cd4013510 tx=0x7f7cd400b040).stop 2023-11-15T01:55:11.301 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:11.299+0000 7f7ce5a6b700 1 -- 172.21.15.137:0/3837452759 shutdown_connections 2023-11-15T01:55:11.301 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:11.299+0000 7f7ce5a6b700 1 --2- 172.21.15.137:0/3837452759 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f7cc802ff60 0x7f7cc8032420 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:55:11.301 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:11.299+0000 7f7ce5a6b700 1 --2- 172.21.15.137:0/3837452759 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7ce00f5410 0x7f7ce00fd3a0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:55:11.301 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:11.299+0000 7f7ce5a6b700 1 -- 172.21.15.137:0/3837452759 >> 172.21.15.137:0/3837452759 conn(0x7f7ce00f0b70 msgr2=0x7f7ce005fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:55:11.302 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:11.299+0000 7f7ce5a6b700 1 -- 172.21.15.137:0/3837452759 shutdown_connections 2023-11-15T01:55:11.302 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:11.299+0000 7f7ce5a6b700 1 -- 172.21.15.137:0/3837452759 wait complete. 2023-11-15T01:55:11.302 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:55:11.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:11 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:55:10.742518+0000 mgr.smithi055.szwngw (mgr.14164) 165 : cluster [DBG] pgmap v158: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:55:11.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:11 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:55:11.298711+0000 mon.smithi055 (mon.0) 234 : audit [DBG] from='client.? 172.21.15.137:0/3837452759' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:55:12.684 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:55:12.684 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:55:13.823 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:13.820+0000 7fe7b6a7c700 1 -- 172.21.15.137:0/2324932495 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe7b00f51a0 msgr2=0x7fe7b00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:55:13.823 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:13.820+0000 7fe7b6a7c700 1 --2- 172.21.15.137:0/2324932495 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe7b00f51a0 0x7fe7b00f55c0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7fe7ac009fb0 tx=0x7fe7ac009b30).stop 2023-11-15T01:55:13.824 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:13.820+0000 7fe7b6a7c700 1 -- 172.21.15.137:0/2324932495 shutdown_connections 2023-11-15T01:55:13.824 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:13.820+0000 7fe7b6a7c700 1 --2- 172.21.15.137:0/2324932495 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe7b00f51a0 0x7fe7b00f55c0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:55:13.824 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:13.820+0000 7fe7b6a7c700 1 -- 172.21.15.137:0/2324932495 >> 172.21.15.137:0/2324932495 conn(0x7fe7b00f09a0 msgr2=0x7fe7b00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:55:13.824 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:13.820+0000 7fe7b6a7c700 1 -- 172.21.15.137:0/2324932495 shutdown_connections 2023-11-15T01:55:13.825 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:13.820+0000 7fe7b6a7c700 1 -- 172.21.15.137:0/2324932495 wait complete. 2023-11-15T01:55:13.825 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:13.821+0000 7fe7b6a7c700 1 Processor -- start 2023-11-15T01:55:13.825 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:13.821+0000 7fe7b6a7c700 1 -- start start 2023-11-15T01:55:13.825 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:13.821+0000 7fe7b6a7c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe7b00f51a0 0x7fe7b01091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:55:13.825 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:13.821+0000 7fe7b6a7c700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7b0060240 con 0x7fe7b00f51a0 2023-11-15T01:55:13.826 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:13.821+0000 7fe7b5a7a700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe7b00f51a0 0x7fe7b01091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:55:13.826 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:13.821+0000 7fe7b5a7a700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe7b00f51a0 0x7fe7b01091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:47044/0 (socket says 172.21.15.137:47044) 2023-11-15T01:55:13.826 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:13.821+0000 7fe7b5a7a700 1 -- 172.21.15.137:0/3146917081 learned_addr learned my addr 172.21.15.137:0/3146917081 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:55:13.826 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:13.822+0000 7fe7b5a7a700 1 -- 172.21.15.137:0/3146917081 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7ac005040 con 0x7fe7b00f51a0 2023-11-15T01:55:13.827 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:13.822+0000 7fe7b5a7a700 1 --2- 172.21.15.137:0/3146917081 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe7b00f51a0 0x7fe7b01091b0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7fe7ac013510 tx=0x7fe7ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:55:13.827 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:13.822+0000 7fe7a67fc700 1 -- 172.21.15.137:0/3146917081 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7ac009320 con 0x7fe7b00f51a0 2023-11-15T01:55:13.828 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:13.822+0000 7fe7b6a7c700 1 -- 172.21.15.137:0/3146917081 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7b0060440 con 0x7fe7b00f51a0 2023-11-15T01:55:13.828 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:13.822+0000 7fe7a67fc700 1 -- 172.21.15.137:0/3146917081 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe7ac013f00 con 0x7fe7b00f51a0 2023-11-15T01:55:13.828 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:13.822+0000 7fe7a67fc700 1 -- 172.21.15.137:0/3146917081 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7ac014dc0 con 0x7fe7b00f51a0 2023-11-15T01:55:13.829 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:13.822+0000 7fe7b6a7c700 1 -- 172.21.15.137:0/3146917081 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7b0109860 con 0x7fe7b00f51a0 2023-11-15T01:55:13.829 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:13.823+0000 7fe7a67fc700 1 -- 172.21.15.137:0/3146917081 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe7ac0095f0 con 0x7fe7b00f51a0 2023-11-15T01:55:13.830 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:13.823+0000 7fe7b6a7c700 1 -- 172.21.15.137:0/3146917081 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7b0045ad0 con 0x7fe7b00f51a0 2023-11-15T01:55:13.830 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:13.823+0000 7fe7a67fc700 1 --2- 172.21.15.137:0/3146917081 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fe79c02ff60 0x7fe79c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:55:13.830 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:13.823+0000 7fe7a67fc700 1 -- 172.21.15.137:0/3146917081 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe7ac041000 con 0x7fe7b00f51a0 2023-11-15T01:55:13.830 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:13.823+0000 7fe7b5279700 1 --2- 172.21.15.137:0/3146917081 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fe79c02ff60 0x7fe79c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:55:13.831 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:13.824+0000 7fe7b5279700 1 --2- 172.21.15.137:0/3146917081 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fe79c02ff60 0x7fe79c032420 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7fe7a00096a0 tx=0x7fe7a0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:55:13.831 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:13.828+0000 7fe7a67fc700 1 -- 172.21.15.137:0/3146917081 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe7ac014420 con 0x7fe7b00f51a0 2023-11-15T01:55:14.034 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:14.033+0000 7fe7b6a7c700 1 -- 172.21.15.137:0/3146917081 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe7b010a080 con 0x7fe7b00f51a0 2023-11-15T01:55:14.035 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:14.034+0000 7fe7a67fc700 1 -- 172.21.15.137:0/3146917081 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe7ac010230 con 0x7fe7b00f51a0 2023-11-15T01:55:14.035 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:55:14.036 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:55:14.037 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:14.036+0000 7fe7b6a7c700 1 -- 172.21.15.137:0/3146917081 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fe79c02ff60 msgr2=0x7fe79c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:55:14.037 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:14.036+0000 7fe7b6a7c700 1 --2- 172.21.15.137:0/3146917081 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fe79c02ff60 0x7fe79c032420 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7fe7a00096a0 tx=0x7fe7a0006b40).stop 2023-11-15T01:55:14.037 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:14.036+0000 7fe7b6a7c700 1 -- 172.21.15.137:0/3146917081 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe7b00f51a0 msgr2=0x7fe7b01091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:55:14.037 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:14.036+0000 7fe7b6a7c700 1 --2- 172.21.15.137:0/3146917081 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe7b00f51a0 0x7fe7b01091b0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7fe7ac013510 tx=0x7fe7ac00b040).stop 2023-11-15T01:55:14.038 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:14.036+0000 7fe7b6a7c700 1 -- 172.21.15.137:0/3146917081 shutdown_connections 2023-11-15T01:55:14.038 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:14.036+0000 7fe7b6a7c700 1 --2- 172.21.15.137:0/3146917081 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fe79c02ff60 0x7fe79c032420 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:55:14.038 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:14.036+0000 7fe7b6a7c700 1 --2- 172.21.15.137:0/3146917081 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe7b00f51a0 0x7fe7b01091b0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:55:14.038 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:14.036+0000 7fe7b6a7c700 1 -- 172.21.15.137:0/3146917081 >> 172.21.15.137:0/3146917081 conn(0x7fe7b00f09a0 msgr2=0x7fe7b00f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:55:14.039 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:14.036+0000 7fe7b6a7c700 1 -- 172.21.15.137:0/3146917081 shutdown_connections 2023-11-15T01:55:14.039 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:14.036+0000 7fe7b6a7c700 1 -- 172.21.15.137:0/3146917081 wait complete. 2023-11-15T01:55:14.039 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:55:14.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:13 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:55:12.742806+0000 mgr.smithi055.szwngw (mgr.14164) 166 : cluster [DBG] pgmap v159: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:55:15.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:14 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:55:14.035265+0000 mon.smithi055 (mon.0) 235 : audit [DBG] from='client.? 172.21.15.137:0/3146917081' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:55:15.425 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:55:15.425 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:55:16.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:15 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:55:14.743120+0000 mgr.smithi055.szwngw (mgr.14164) 167 : cluster [DBG] pgmap v160: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:55:16.578 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:16.578+0000 7f4079588700 1 -- 172.21.15.137:0/4178152769 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4074060280 msgr2=0x7f40740f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:55:16.579 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:16.578+0000 7f4079588700 1 --2- 172.21.15.137:0/4178152769 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4074060280 0x7f40740f90f0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7f4064009fb0 tx=0x7f4064009b30).stop 2023-11-15T01:55:16.579 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:16.578+0000 7f4079588700 1 -- 172.21.15.137:0/4178152769 shutdown_connections 2023-11-15T01:55:16.579 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:16.578+0000 7f4079588700 1 --2- 172.21.15.137:0/4178152769 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4074060280 0x7f40740f90f0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:55:16.579 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:16.578+0000 7f4079588700 1 -- 172.21.15.137:0/4178152769 >> 172.21.15.137:0/4178152769 conn(0x7f40740f0b70 msgr2=0x7f40740f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:55:16.579 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:16.578+0000 7f4079588700 1 -- 172.21.15.137:0/4178152769 shutdown_connections 2023-11-15T01:55:16.580 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:16.578+0000 7f4079588700 1 -- 172.21.15.137:0/4178152769 wait complete. 2023-11-15T01:55:16.582 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:16.578+0000 7f4079588700 1 Processor -- start 2023-11-15T01:55:16.582 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:16.578+0000 7f4079588700 1 -- start start 2023-11-15T01:55:16.582 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:16.578+0000 7f4079588700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4074060280 0x7f40740f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:55:16.582 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:16.578+0000 7f4079588700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40740f4ff0 con 0x7f4074060280 2023-11-15T01:55:16.583 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:16.579+0000 7f4072ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4074060280 0x7f40740f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:55:16.583 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:16.579+0000 7f4072ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4074060280 0x7f40740f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:47054/0 (socket says 172.21.15.137:47054) 2023-11-15T01:55:16.583 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:16.579+0000 7f4072ffd700 1 -- 172.21.15.137:0/4152158079 learned_addr learned my addr 172.21.15.137:0/4152158079 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:55:16.583 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:16.579+0000 7f4072ffd700 1 -- 172.21.15.137:0/4152158079 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4064005040 con 0x7f4074060280 2023-11-15T01:55:16.583 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:16.579+0000 7f4072ffd700 1 --2- 172.21.15.137:0/4152158079 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4074060280 0x7f40740f6b70 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f4064013510 tx=0x7f406400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:55:16.584 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:16.579+0000 7f405b7fe700 1 -- 172.21.15.137:0/4152158079 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4064009320 con 0x7f4074060280 2023-11-15T01:55:16.584 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:16.579+0000 7f4079588700 1 -- 172.21.15.137:0/4152158079 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40740f5190 con 0x7f4074060280 2023-11-15T01:55:16.584 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:16.579+0000 7f405b7fe700 1 -- 172.21.15.137:0/4152158079 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4064013f00 con 0x7f4074060280 2023-11-15T01:55:16.584 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:16.579+0000 7f405b7fe700 1 -- 172.21.15.137:0/4152158079 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4064014dc0 con 0x7f4074060280 2023-11-15T01:55:16.584 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:16.579+0000 7f4079588700 1 -- 172.21.15.137:0/4152158079 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40740f5620 con 0x7f4074060280 2023-11-15T01:55:16.584 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:16.580+0000 7f405b7fe700 1 -- 172.21.15.137:0/4152158079 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f40640263a0 con 0x7f4074060280 2023-11-15T01:55:16.585 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:16.580+0000 7f405b7fe700 1 --2- 172.21.15.137:0/4152158079 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f405c02ffb0 0x7f405c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:55:16.585 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:16.580+0000 7f405b7fe700 1 -- 172.21.15.137:0/4152158079 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4064040ff0 con 0x7f4074060280 2023-11-15T01:55:16.585 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:16.580+0000 7f4079588700 1 -- 172.21.15.137:0/4152158079 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4074059100 con 0x7f4074060280 2023-11-15T01:55:16.585 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:16.581+0000 7f40727fc700 1 --2- 172.21.15.137:0/4152158079 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f405c02ffb0 0x7f405c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:55:16.585 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:16.581+0000 7f40727fc700 1 --2- 172.21.15.137:0/4152158079 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f405c02ffb0 0x7f405c032470 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f40680096a0 tx=0x7f4068006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:55:16.586 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:16.586+0000 7f405b7fe700 1 -- 172.21.15.137:0/4152158079 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4064012920 con 0x7f4074060280 2023-11-15T01:55:16.796 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:16.795+0000 7f4079588700 1 -- 172.21.15.137:0/4152158079 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4074045ad0 con 0x7f4074060280 2023-11-15T01:55:16.796 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:16.795+0000 7f405b7fe700 1 -- 172.21.15.137:0/4152158079 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f406402b020 con 0x7f4074060280 2023-11-15T01:55:16.797 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:55:16.797 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:55:16.798 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:16.797+0000 7f4079588700 1 -- 172.21.15.137:0/4152158079 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f405c02ffb0 msgr2=0x7f405c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:55:16.798 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:16.797+0000 7f4079588700 1 --2- 172.21.15.137:0/4152158079 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f405c02ffb0 0x7f405c032470 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f40680096a0 tx=0x7f4068006b40).stop 2023-11-15T01:55:16.798 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:16.797+0000 7f4079588700 1 -- 172.21.15.137:0/4152158079 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4074060280 msgr2=0x7f40740f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:55:16.798 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:16.797+0000 7f4079588700 1 --2- 172.21.15.137:0/4152158079 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4074060280 0x7f40740f6b70 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f4064013510 tx=0x7f406400b040).stop 2023-11-15T01:55:16.798 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:16.797+0000 7f4079588700 1 -- 172.21.15.137:0/4152158079 shutdown_connections 2023-11-15T01:55:16.799 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:16.797+0000 7f4079588700 1 --2- 172.21.15.137:0/4152158079 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f405c02ffb0 0x7f405c032470 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:55:16.799 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:16.797+0000 7f4079588700 1 --2- 172.21.15.137:0/4152158079 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4074060280 0x7f40740f6b70 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:55:16.799 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:16.797+0000 7f4079588700 1 -- 172.21.15.137:0/4152158079 >> 172.21.15.137:0/4152158079 conn(0x7f40740f0b70 msgr2=0x7f40740f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:55:16.799 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:16.797+0000 7f4079588700 1 -- 172.21.15.137:0/4152158079 shutdown_connections 2023-11-15T01:55:16.799 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:16.797+0000 7f4079588700 1 -- 172.21.15.137:0/4152158079 wait complete. 2023-11-15T01:55:16.800 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:55:17.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:16 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:55:16.796445+0000 mon.smithi055 (mon.0) 236 : audit [DBG] from='client.? 172.21.15.137:0/4152158079' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:55:18.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:17 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:55:16.743389+0000 mgr.smithi055.szwngw (mgr.14164) 168 : cluster [DBG] pgmap v161: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:55:18.195 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:55:18.195 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:55:19.387 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:19.385+0000 7f71fcd83700 1 -- 172.21.15.137:0/1278391761 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f71f80f5410 msgr2=0x7f71f80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:55:19.388 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:19.385+0000 7f71fcd83700 1 --2- 172.21.15.137:0/1278391761 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f71f80f5410 0x7f71f80f5830 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7f71ec009fb0 tx=0x7f71ec009b30).stop 2023-11-15T01:55:19.388 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:19.385+0000 7f71fcd83700 1 -- 172.21.15.137:0/1278391761 shutdown_connections 2023-11-15T01:55:19.388 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:19.385+0000 7f71fcd83700 1 --2- 172.21.15.137:0/1278391761 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f71f80f5410 0x7f71f80f5830 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:55:19.388 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:19.385+0000 7f71fcd83700 1 -- 172.21.15.137:0/1278391761 >> 172.21.15.137:0/1278391761 conn(0x7f71f80f0b70 msgr2=0x7f71f80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:55:19.389 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:19.385+0000 7f71fcd83700 1 -- 172.21.15.137:0/1278391761 shutdown_connections 2023-11-15T01:55:19.389 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:19.385+0000 7f71fcd83700 1 -- 172.21.15.137:0/1278391761 wait complete. 2023-11-15T01:55:19.389 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:19.386+0000 7f71fcd83700 1 Processor -- start 2023-11-15T01:55:19.389 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:19.386+0000 7f71fcd83700 1 -- start start 2023-11-15T01:55:19.390 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:19.386+0000 7f71fcd83700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f71f80f5410 0x7f71f8106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:55:19.390 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:19.386+0000 7f71fcd83700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71f8106c10 con 0x7f71f80f5410 2023-11-15T01:55:19.390 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:19.386+0000 7f71f659c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f71f80f5410 0x7f71f8106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:55:19.391 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:19.386+0000 7f71f659c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f71f80f5410 0x7f71f8106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:54160/0 (socket says 172.21.15.137:54160) 2023-11-15T01:55:19.391 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:19.386+0000 7f71f659c700 1 -- 172.21.15.137:0/3519750882 learned_addr learned my addr 172.21.15.137:0/3519750882 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:55:19.391 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:19.386+0000 7f71f659c700 1 -- 172.21.15.137:0/3519750882 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71ec005040 con 0x7f71f80f5410 2023-11-15T01:55:19.391 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:19.387+0000 7f71f659c700 1 --2- 172.21.15.137:0/3519750882 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f71f80f5410 0x7f71f8106720 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7f71ec013510 tx=0x7f71ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:55:19.392 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:19.387+0000 7f71e6ffd700 1 -- 172.21.15.137:0/3519750882 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f71ec009320 con 0x7f71f80f5410 2023-11-15T01:55:19.392 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:19.387+0000 7f71fcd83700 1 -- 172.21.15.137:0/3519750882 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71f8106e10 con 0x7f71f80f5410 2023-11-15T01:55:19.392 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:19.387+0000 7f71e6ffd700 1 -- 172.21.15.137:0/3519750882 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f71ec013f00 con 0x7f71f80f5410 2023-11-15T01:55:19.392 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:19.387+0000 7f71e6ffd700 1 -- 172.21.15.137:0/3519750882 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f71ec014d90 con 0x7f71f80f5410 2023-11-15T01:55:19.393 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:19.387+0000 7f71fcd83700 1 -- 172.21.15.137:0/3519750882 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71f81072a0 con 0x7f71f80f5410 2023-11-15T01:55:19.394 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:19.388+0000 7f71fcd83700 1 -- 172.21.15.137:0/3519750882 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71f8045ad0 con 0x7f71f80f5410 2023-11-15T01:55:19.394 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:19.388+0000 7f71e6ffd700 1 -- 172.21.15.137:0/3519750882 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f71ec014420 con 0x7f71f80f5410 2023-11-15T01:55:19.395 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:19.388+0000 7f71e6ffd700 1 --2- 172.21.15.137:0/3519750882 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f71e002ff60 0x7f71e0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:55:19.395 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:19.388+0000 7f71e6ffd700 1 -- 172.21.15.137:0/3519750882 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f71ec041000 con 0x7f71f80f5410 2023-11-15T01:55:19.395 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:19.388+0000 7f71f5d9b700 1 --2- 172.21.15.137:0/3519750882 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f71e002ff60 0x7f71e0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:55:19.395 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:19.388+0000 7f71f5d9b700 1 --2- 172.21.15.137:0/3519750882 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f71e002ff60 0x7f71e0032420 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f71e80096a0 tx=0x7f71e8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:55:19.395 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:19.393+0000 7f71e6ffd700 1 -- 172.21.15.137:0/3519750882 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f71ec0095f0 con 0x7f71f80f5410 2023-11-15T01:55:19.599 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:19.598+0000 7f71fcd83700 1 -- 172.21.15.137:0/3519750882 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f71f8108150 con 0x7f71f80f5410 2023-11-15T01:55:19.600 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:19.599+0000 7f71e6ffd700 1 -- 172.21.15.137:0/3519750882 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f71ec01db70 con 0x7f71f80f5410 2023-11-15T01:55:19.601 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:55:19.602 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:55:19.603 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:19.601+0000 7f71fcd83700 1 -- 172.21.15.137:0/3519750882 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f71e002ff60 msgr2=0x7f71e0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:55:19.604 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:19.601+0000 7f71fcd83700 1 --2- 172.21.15.137:0/3519750882 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f71e002ff60 0x7f71e0032420 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f71e80096a0 tx=0x7f71e8006b40).stop 2023-11-15T01:55:19.604 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:19.601+0000 7f71fcd83700 1 -- 172.21.15.137:0/3519750882 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f71f80f5410 msgr2=0x7f71f8106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:55:19.605 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:19.601+0000 7f71fcd83700 1 --2- 172.21.15.137:0/3519750882 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f71f80f5410 0x7f71f8106720 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7f71ec013510 tx=0x7f71ec00b040).stop 2023-11-15T01:55:19.606 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:19.601+0000 7f71fcd83700 1 -- 172.21.15.137:0/3519750882 shutdown_connections 2023-11-15T01:55:19.606 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:19.601+0000 7f71fcd83700 1 --2- 172.21.15.137:0/3519750882 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f71e002ff60 0x7f71e0032420 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:55:19.607 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:19.601+0000 7f71fcd83700 1 --2- 172.21.15.137:0/3519750882 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f71f80f5410 0x7f71f8106720 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:55:19.607 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:19.601+0000 7f71fcd83700 1 -- 172.21.15.137:0/3519750882 >> 172.21.15.137:0/3519750882 conn(0x7f71f80f0b70 msgr2=0x7f71f805fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:55:19.607 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:19.601+0000 7f71fcd83700 1 -- 172.21.15.137:0/3519750882 shutdown_connections 2023-11-15T01:55:19.607 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:19.601+0000 7f71fcd83700 1 -- 172.21.15.137:0/3519750882 wait complete. 2023-11-15T01:55:19.608 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:55:20.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:19 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:55:18.743735+0000 mgr.smithi055.szwngw (mgr.14164) 169 : cluster [DBG] pgmap v162: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:55:20.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:19 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:55:19.600151+0000 mon.smithi055 (mon.0) 237 : audit [DBG] from='client.? 172.21.15.137:0/3519750882' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:55:20.985 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:55:20.986 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:55:21.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:21 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:55:20.743975+0000 mgr.smithi055.szwngw (mgr.14164) 170 : cluster [DBG] pgmap v163: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:55:22.123 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:22.122+0000 7fbb70d33700 1 -- 172.21.15.137:0/2722607788 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbb6c0f5410 msgr2=0x7fbb6c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:55:22.124 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:22.122+0000 7fbb70d33700 1 --2- 172.21.15.137:0/2722607788 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbb6c0f5410 0x7fbb6c0f5830 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7fbb54009fb0 tx=0x7fbb54009b30).stop 2023-11-15T01:55:22.124 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:22.122+0000 7fbb70d33700 1 -- 172.21.15.137:0/2722607788 shutdown_connections 2023-11-15T01:55:22.124 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:22.122+0000 7fbb70d33700 1 --2- 172.21.15.137:0/2722607788 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbb6c0f5410 0x7fbb6c0f5830 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:55:22.124 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:22.122+0000 7fbb70d33700 1 -- 172.21.15.137:0/2722607788 >> 172.21.15.137:0/2722607788 conn(0x7fbb6c0f0b70 msgr2=0x7fbb6c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:55:22.124 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:22.122+0000 7fbb70d33700 1 -- 172.21.15.137:0/2722607788 shutdown_connections 2023-11-15T01:55:22.125 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:22.122+0000 7fbb70d33700 1 -- 172.21.15.137:0/2722607788 wait complete. 2023-11-15T01:55:22.125 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:22.123+0000 7fbb70d33700 1 Processor -- start 2023-11-15T01:55:22.125 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:22.123+0000 7fbb70d33700 1 -- start start 2023-11-15T01:55:22.125 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:22.123+0000 7fbb70d33700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbb6c0f5410 0x7fbb6c0fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:55:22.125 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:22.123+0000 7fbb70d33700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb6c0fb820 con 0x7fbb6c0f5410 2023-11-15T01:55:22.127 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:22.123+0000 7fbb6a59c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbb6c0f5410 0x7fbb6c0fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:55:22.127 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:22.123+0000 7fbb6a59c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbb6c0f5410 0x7fbb6c0fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:54166/0 (socket says 172.21.15.137:54166) 2023-11-15T01:55:22.128 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:22.123+0000 7fbb6a59c700 1 -- 172.21.15.137:0/3330292261 learned_addr learned my addr 172.21.15.137:0/3330292261 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:55:22.128 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:22.123+0000 7fbb6a59c700 1 -- 172.21.15.137:0/3330292261 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb54005040 con 0x7fbb6c0f5410 2023-11-15T01:55:22.128 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:22.123+0000 7fbb6a59c700 1 --2- 172.21.15.137:0/3330292261 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbb6c0f5410 0x7fbb6c0fd3a0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7fbb54013510 tx=0x7fbb5400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:55:22.128 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:22.124+0000 7fbb62ffd700 1 -- 172.21.15.137:0/3330292261 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbb54009320 con 0x7fbb6c0f5410 2023-11-15T01:55:22.128 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:22.124+0000 7fbb70d33700 1 -- 172.21.15.137:0/3330292261 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb6c0fb9c0 con 0x7fbb6c0f5410 2023-11-15T01:55:22.128 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:22.124+0000 7fbb62ffd700 1 -- 172.21.15.137:0/3330292261 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbb54013f00 con 0x7fbb6c0f5410 2023-11-15T01:55:22.129 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:22.124+0000 7fbb62ffd700 1 -- 172.21.15.137:0/3330292261 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbb54014d90 con 0x7fbb6c0f5410 2023-11-15T01:55:22.129 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:22.124+0000 7fbb70d33700 1 -- 172.21.15.137:0/3330292261 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb6c0fbe50 con 0x7fbb6c0f5410 2023-11-15T01:55:22.129 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:22.125+0000 7fbb62ffd700 1 -- 172.21.15.137:0/3330292261 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbb540095f0 con 0x7fbb6c0f5410 2023-11-15T01:55:22.129 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:22.125+0000 7fbb70d33700 1 -- 172.21.15.137:0/3330292261 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb6c059100 con 0x7fbb6c0f5410 2023-11-15T01:55:22.129 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:22.125+0000 7fbb62ffd700 1 --2- 172.21.15.137:0/3330292261 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fbb5802ff60 0x7fbb58032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:55:22.130 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:22.125+0000 7fbb62ffd700 1 -- 172.21.15.137:0/3330292261 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbb54041070 con 0x7fbb6c0f5410 2023-11-15T01:55:22.130 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:22.125+0000 7fbb69d9b700 1 --2- 172.21.15.137:0/3330292261 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fbb5802ff60 0x7fbb58032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:55:22.130 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:22.125+0000 7fbb69d9b700 1 --2- 172.21.15.137:0/3330292261 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fbb5802ff60 0x7fbb58032420 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fbb5c0096a0 tx=0x7fbb5c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:55:22.131 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:22.130+0000 7fbb62ffd700 1 -- 172.21.15.137:0/3330292261 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbb54014420 con 0x7fbb6c0f5410 2023-11-15T01:55:22.337 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:22.337+0000 7fbb70d33700 1 -- 172.21.15.137:0/3330292261 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbb6c045ad0 con 0x7fbb6c0f5410 2023-11-15T01:55:22.338 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:22.337+0000 7fbb62ffd700 1 -- 172.21.15.137:0/3330292261 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbb54010230 con 0x7fbb6c0f5410 2023-11-15T01:55:22.338 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:55:22.339 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:55:22.339 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:22.339+0000 7fbb70d33700 1 -- 172.21.15.137:0/3330292261 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fbb5802ff60 msgr2=0x7fbb58032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:55:22.340 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:22.339+0000 7fbb70d33700 1 --2- 172.21.15.137:0/3330292261 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fbb5802ff60 0x7fbb58032420 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fbb5c0096a0 tx=0x7fbb5c006b40).stop 2023-11-15T01:55:22.340 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:22.339+0000 7fbb70d33700 1 -- 172.21.15.137:0/3330292261 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbb6c0f5410 msgr2=0x7fbb6c0fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:55:22.340 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:22.339+0000 7fbb70d33700 1 --2- 172.21.15.137:0/3330292261 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbb6c0f5410 0x7fbb6c0fd3a0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7fbb54013510 tx=0x7fbb5400b040).stop 2023-11-15T01:55:22.340 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:22.339+0000 7fbb70d33700 1 -- 172.21.15.137:0/3330292261 shutdown_connections 2023-11-15T01:55:22.340 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:22.339+0000 7fbb70d33700 1 --2- 172.21.15.137:0/3330292261 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fbb5802ff60 0x7fbb58032420 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:55:22.341 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:22.339+0000 7fbb70d33700 1 --2- 172.21.15.137:0/3330292261 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbb6c0f5410 0x7fbb6c0fd3a0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:55:22.341 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:22.339+0000 7fbb70d33700 1 -- 172.21.15.137:0/3330292261 >> 172.21.15.137:0/3330292261 conn(0x7fbb6c0f0b70 msgr2=0x7fbb6c05fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:55:22.341 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:22.339+0000 7fbb70d33700 1 -- 172.21.15.137:0/3330292261 shutdown_connections 2023-11-15T01:55:22.341 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:22.339+0000 7fbb70d33700 1 -- 172.21.15.137:0/3330292261 wait complete. 2023-11-15T01:55:22.342 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:55:22.578 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:22 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:55:22.265220+0000 mon.smithi055 (mon.0) 238 : audit [INF] from='mgr.14164 172.21.15.55:0/59463371' entity='mgr.smithi055.szwngw' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi055", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2023-11-15T01:55:22.578 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:22 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:55:22.270517+0000 mon.smithi055 (mon.0) 239 : audit [INF] from='mgr.14164 172.21.15.55:0/59463371' entity='mgr.smithi055.szwngw' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi055", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2023-11-15T01:55:22.579 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:22 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:55:22.271434+0000 mon.smithi055 (mon.0) 240 : audit [DBG] from='mgr.14164 172.21.15.55:0/59463371' entity='mgr.smithi055.szwngw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-15T01:55:22.579 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:22 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:55:22.338374+0000 mon.smithi055 (mon.0) 241 : audit [DBG] from='client.? 172.21.15.137:0/3330292261' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:55:23.719 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:55:23.720 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:55:23.812 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:23 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cephadm 2023-11-15T01:55:22.263806+0000 mgr.smithi055.szwngw (mgr.14164) 171 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon alertmanager.smithi055 ... 2023-11-15T01:55:23.812 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:23 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi055 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2023-11-15T01:55:23.812 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:23 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2023-11-15T01:55:23.812 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:23 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: stat: stderr time="2023-11-15T01:50:31Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T01:55:23.813 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:23 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: stat: stderr time="2023-11-15T01:51:08Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T01:55:23.813 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:23 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: stat: stderr time="2023-11-15T01:51:46Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T01:55:23.813 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:23 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-11-15T01:55:23.813 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:23 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi055 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2023-11-15T01:55:23.814 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:23 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2023-11-15T01:55:23.814 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:23 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: stat: stderr time="2023-11-15T01:52:59Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T01:55:23.814 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:23 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: stat: stderr time="2023-11-15T01:54:07Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: copying system image from manifest list: determining manifest MIME type for docker://prom/alertmanager:v0.20.0: reading manifest sha256:b9323917a2eda265bec69e59a457f001c529facbbc8166df277f4850cdac61a0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T01:55:23.814 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:23 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: stat: stderr time="2023-11-15T01:54:44Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T01:55:23.814 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:23 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-11-15T01:55:23.815 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:23 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: Traceback (most recent call last): 2023-11-15T01:55:23.815 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:23 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: File "/var/lib/ceph/0cb4fc0c-8359-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-11-15T01:55:23.815 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:23 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: main() 2023-11-15T01:55:23.815 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:23 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: File "/var/lib/ceph/0cb4fc0c-8359-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-11-15T01:55:23.815 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:23 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: r = ctx.func(ctx) 2023-11-15T01:55:23.816 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:23 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: File "/var/lib/ceph/0cb4fc0c-8359-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-11-15T01:55:23.816 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:23 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: return func(ctx) 2023-11-15T01:55:23.816 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:23 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: File "/var/lib/ceph/0cb4fc0c-8359-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2023-11-15T01:55:23.816 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:23 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2023-11-15T01:55:23.817 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:23 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: File "/var/lib/ceph/0cb4fc0c-8359-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4191, in extract_uid_gid_monitoring 2023-11-15T01:55:23.817 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:23 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2023-11-15T01:55:23.817 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:23 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: File "/var/lib/ceph/0cb4fc0c-8359-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2023-11-15T01:55:23.817 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:23 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: raise RuntimeError('uid/gid not found') 2023-11-15T01:55:23.817 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:23 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: RuntimeError: uid/gid not found 2023-11-15T01:55:23.818 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:23 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: Traceback (most recent call last): 2023-11-15T01:55:23.818 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:23 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2023-11-15T01:55:23.818 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:23 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: yield (conn, connr) 2023-11-15T01:55:23.818 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:23 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2023-11-15T01:55:23.818 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:23 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: code, '\n'.join(err))) 2023-11-15T01:55:23.819 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:23 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon alertmanager.smithi055 ... 2023-11-15T01:55:23.819 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:23 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi055 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2023-11-15T01:55:23.819 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:23 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2023-11-15T01:55:23.819 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:23 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: stat: stderr time="2023-11-15T01:50:31Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T01:55:23.819 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:23 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: stat: stderr time="2023-11-15T01:51:08Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T01:55:23.820 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:23 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: stat: stderr time="2023-11-15T01:51:46Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T01:55:23.820 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:23 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-11-15T01:55:23.820 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:23 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi055 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2023-11-15T01:55:23.820 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:23 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2023-11-15T01:55:23.821 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:23 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: stat: stderr time="2023-11-15T01:52:59Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T01:55:23.821 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:23 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: stat: stderr time="2023-11-15T01:54:07Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: copying system image from manifest list: determining manifest MIME type for docker://prom/alertmanager:v0.20.0: reading manifest sha256:b9323917a2eda265bec69e59a457f001c529facbbc8166df277f4850cdac61a0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T01:55:23.821 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:23 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: stat: stderr time="2023-11-15T01:54:44Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T01:55:23.821 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:23 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-11-15T01:55:23.822 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:23 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: Traceback (most recent call last): 2023-11-15T01:55:23.822 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:23 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: File "/var/lib/ceph/0cb4fc0c-8359-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-11-15T01:55:23.822 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:23 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: main() 2023-11-15T01:55:23.822 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:23 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: File "/var/lib/ceph/0cb4fc0c-8359-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-11-15T01:55:23.822 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:23 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: r = ctx.func(ctx) 2023-11-15T01:55:23.823 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:23 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: File "/var/lib/ceph/0cb4fc0c-8359-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-11-15T01:55:23.823 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:23 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: return func(ctx) 2023-11-15T01:55:23.823 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:23 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: File "/var/lib/ceph/0cb4fc0c-8359-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2023-11-15T01:55:23.823 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:23 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2023-11-15T01:55:23.824 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:23 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: File "/var/lib/ceph/0cb4fc0c-8359-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4191, in extract_uid_gid_monitoring 2023-11-15T01:55:23.824 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:23 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2023-11-15T01:55:23.824 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:23 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: File "/var/lib/ceph/0cb4fc0c-8359-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2023-11-15T01:55:23.824 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:23 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: raise RuntimeError('uid/gid not found') 2023-11-15T01:55:23.824 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:23 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: RuntimeError: uid/gid not found 2023-11-15T01:55:23.825 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:23 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cephadm 2023-11-15T01:55:22.272141+0000 mgr.smithi055.szwngw (mgr.14164) 172 : cephadm [INF] Deploying daemon crash.smithi055 on smithi055 2023-11-15T01:55:23.825 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:23 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:55:22.744188+0000 mgr.smithi055.szwngw (mgr.14164) 173 : cluster [DBG] pgmap v164: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:55:24.867 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:24.865+0000 7f5d7edb0700 1 -- 172.21.15.137:0/2337215704 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5d780f6510 msgr2=0x7f5d780f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:55:24.867 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:24.865+0000 7f5d7edb0700 1 --2- 172.21.15.137:0/2337215704 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5d780f6510 0x7f5d780f6930 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7f5d6c009fb0 tx=0x7f5d6c009b30).stop 2023-11-15T01:55:24.868 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:24.865+0000 7f5d7edb0700 1 -- 172.21.15.137:0/2337215704 shutdown_connections 2023-11-15T01:55:24.868 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:24.865+0000 7f5d7edb0700 1 --2- 172.21.15.137:0/2337215704 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5d780f6510 0x7f5d780f6930 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:55:24.868 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:24.865+0000 7f5d7edb0700 1 -- 172.21.15.137:0/2337215704 >> 172.21.15.137:0/2337215704 conn(0x7f5d780f1c70 msgr2=0x7f5d780f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:55:24.868 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:24.866+0000 7f5d7edb0700 1 -- 172.21.15.137:0/2337215704 shutdown_connections 2023-11-15T01:55:24.868 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:24.866+0000 7f5d7edb0700 1 -- 172.21.15.137:0/2337215704 wait complete. 2023-11-15T01:55:24.869 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:24.866+0000 7f5d7edb0700 1 Processor -- start 2023-11-15T01:55:24.869 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:24.866+0000 7f5d7edb0700 1 -- start start 2023-11-15T01:55:24.869 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:24.866+0000 7f5d7edb0700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5d780f6510 0x7f5d78106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:55:24.869 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:24.866+0000 7f5d7edb0700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d78106c20 con 0x7f5d780f6510 2023-11-15T01:55:24.869 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:24.867+0000 7f5d7cb4c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5d780f6510 0x7f5d78106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:55:24.870 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:24.867+0000 7f5d7cb4c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5d780f6510 0x7f5d78106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:54186/0 (socket says 172.21.15.137:54186) 2023-11-15T01:55:24.870 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:24.867+0000 7f5d7cb4c700 1 -- 172.21.15.137:0/1130820821 learned_addr learned my addr 172.21.15.137:0/1130820821 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:55:24.874 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:24.867+0000 7f5d7cb4c700 1 -- 172.21.15.137:0/1130820821 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d6c005040 con 0x7f5d780f6510 2023-11-15T01:55:24.874 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:24.867+0000 7f5d7cb4c700 1 --2- 172.21.15.137:0/1130820821 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5d780f6510 0x7f5d78106730 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f5d6c013510 tx=0x7f5d6c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:55:24.875 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:24.867+0000 7f5d757fa700 1 -- 172.21.15.137:0/1130820821 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d6c009320 con 0x7f5d780f6510 2023-11-15T01:55:24.875 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:24.867+0000 7f5d7edb0700 1 -- 172.21.15.137:0/1130820821 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d78106e20 con 0x7f5d780f6510 2023-11-15T01:55:24.875 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:24.867+0000 7f5d757fa700 1 -- 172.21.15.137:0/1130820821 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5d6c013f00 con 0x7f5d780f6510 2023-11-15T01:55:24.875 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:24.867+0000 7f5d757fa700 1 -- 172.21.15.137:0/1130820821 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d6c014d90 con 0x7f5d780f6510 2023-11-15T01:55:24.875 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:24.867+0000 7f5d7edb0700 1 -- 172.21.15.137:0/1130820821 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d781072b0 con 0x7f5d780f6510 2023-11-15T01:55:24.876 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:24.868+0000 7f5d757fa700 1 -- 172.21.15.137:0/1130820821 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5d6c0095f0 con 0x7f5d780f6510 2023-11-15T01:55:24.876 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:24.868+0000 7f5d7edb0700 1 -- 172.21.15.137:0/1130820821 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d78045ad0 con 0x7f5d780f6510 2023-11-15T01:55:24.876 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:24.868+0000 7f5d757fa700 1 --2- 172.21.15.137:0/1130820821 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f5d6802ff60 0x7f5d68032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:55:24.877 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:24.868+0000 7f5d757fa700 1 -- 172.21.15.137:0/1130820821 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5d6c041000 con 0x7f5d780f6510 2023-11-15T01:55:24.877 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:24.869+0000 7f5d77fff700 1 --2- 172.21.15.137:0/1130820821 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f5d6802ff60 0x7f5d68032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:55:24.877 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:24.869+0000 7f5d77fff700 1 --2- 172.21.15.137:0/1130820821 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f5d6802ff60 0x7f5d68032420 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f5d640096a0 tx=0x7f5d64006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:55:24.877 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:24.873+0000 7f5d757fa700 1 -- 172.21.15.137:0/1130820821 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5d6c014420 con 0x7f5d780f6510 2023-11-15T01:55:25.082 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:25.081+0000 7f5d7edb0700 1 -- 172.21.15.137:0/1130820821 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5d780fbf60 con 0x7f5d780f6510 2023-11-15T01:55:25.082 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:25.082+0000 7f5d757fa700 1 -- 172.21.15.137:0/1130820821 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5d6c01d7f0 con 0x7f5d780f6510 2023-11-15T01:55:25.083 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:55:25.083 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:55:25.085 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:25.083+0000 7f5d7edb0700 1 -- 172.21.15.137:0/1130820821 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f5d6802ff60 msgr2=0x7f5d68032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:55:25.085 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:25.083+0000 7f5d7edb0700 1 --2- 172.21.15.137:0/1130820821 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f5d6802ff60 0x7f5d68032420 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f5d640096a0 tx=0x7f5d64006b40).stop 2023-11-15T01:55:25.085 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:25.083+0000 7f5d7edb0700 1 -- 172.21.15.137:0/1130820821 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5d780f6510 msgr2=0x7f5d78106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:55:25.086 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:25.083+0000 7f5d7edb0700 1 --2- 172.21.15.137:0/1130820821 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5d780f6510 0x7f5d78106730 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f5d6c013510 tx=0x7f5d6c00b040).stop 2023-11-15T01:55:25.086 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:25.083+0000 7f5d7edb0700 1 -- 172.21.15.137:0/1130820821 shutdown_connections 2023-11-15T01:55:25.086 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:25.083+0000 7f5d7edb0700 1 --2- 172.21.15.137:0/1130820821 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f5d6802ff60 0x7f5d68032420 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:55:25.086 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:25.083+0000 7f5d7edb0700 1 --2- 172.21.15.137:0/1130820821 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5d780f6510 0x7f5d78106730 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:55:25.086 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:25.083+0000 7f5d7edb0700 1 -- 172.21.15.137:0/1130820821 >> 172.21.15.137:0/1130820821 conn(0x7f5d780f1c70 msgr2=0x7f5d780f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:55:25.087 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:25.083+0000 7f5d7edb0700 1 -- 172.21.15.137:0/1130820821 shutdown_connections 2023-11-15T01:55:25.087 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:25.083+0000 7f5d7edb0700 1 -- 172.21.15.137:0/1130820821 wait complete. 2023-11-15T01:55:25.087 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:55:25.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:25 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:55:23.999212+0000 mon.smithi055 (mon.0) 242 : audit [INF] from='mgr.14164 172.21.15.55:0/59463371' entity='mgr.smithi055.szwngw' 2023-11-15T01:55:25.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:25 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:55:24.161517+0000 mon.smithi055 (mon.0) 243 : audit [INF] from='mgr.14164 172.21.15.55:0/59463371' entity='mgr.smithi055.szwngw' 2023-11-15T01:55:25.329 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:25 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:55:24.166752+0000 mon.smithi055 (mon.0) 244 : audit [INF] from='mgr.14164 172.21.15.55:0/59463371' entity='mgr.smithi055.szwngw' 2023-11-15T01:55:25.329 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:25 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:55:24.167244+0000 mon.smithi055 (mon.0) 245 : audit [INF] from='mgr.14164 172.21.15.55:0/59463371' entity='mgr.smithi055.szwngw' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-11-15T01:55:25.329 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:25 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:55:24.172725+0000 mon.smithi055 (mon.0) 246 : audit [INF] from='mgr.14164 172.21.15.55:0/59463371' entity='mgr.smithi055.szwngw' 2023-11-15T01:55:26.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:26 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:55:24.167748+0000 mgr.smithi055.szwngw (mgr.14164) 174 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-11-15T01:55:26.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:26 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cephadm 2023-11-15T01:55:24.176521+0000 mgr.smithi055.szwngw (mgr.14164) 175 : cephadm [INF] Deploying daemon grafana.smithi055 on smithi055 2023-11-15T01:55:26.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:26 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:55:24.744555+0000 mgr.smithi055.szwngw (mgr.14164) 176 : cluster [DBG] pgmap v165: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:55:26.329 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:26 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:55:25.082862+0000 mon.smithi055 (mon.0) 247 : audit [DBG] from='client.? 172.21.15.137:0/1130820821' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:55:26.464 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:55:26.464 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:55:27.621 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:27.618+0000 7fdebbc6a700 1 -- 172.21.15.137:0/4204540178 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdeb40f5410 msgr2=0x7fdeb40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:55:27.621 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:27.618+0000 7fdebbc6a700 1 --2- 172.21.15.137:0/4204540178 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdeb40f5410 0x7fdeb40f5830 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7fdea4009fb0 tx=0x7fdea4009b30).stop 2023-11-15T01:55:27.621 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:27.618+0000 7fdebbc6a700 1 -- 172.21.15.137:0/4204540178 shutdown_connections 2023-11-15T01:55:27.621 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:27.618+0000 7fdebbc6a700 1 --2- 172.21.15.137:0/4204540178 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdeb40f5410 0x7fdeb40f5830 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:55:27.622 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:27.618+0000 7fdebbc6a700 1 -- 172.21.15.137:0/4204540178 >> 172.21.15.137:0/4204540178 conn(0x7fdeb40f0b70 msgr2=0x7fdeb40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:55:27.622 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:27.618+0000 7fdebbc6a700 1 -- 172.21.15.137:0/4204540178 shutdown_connections 2023-11-15T01:55:27.622 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:27.618+0000 7fdebbc6a700 1 -- 172.21.15.137:0/4204540178 wait complete. 2023-11-15T01:55:27.622 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:27.619+0000 7fdebbc6a700 1 Processor -- start 2023-11-15T01:55:27.622 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:27.619+0000 7fdebbc6a700 1 -- start start 2023-11-15T01:55:27.623 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:27.619+0000 7fdebbc6a700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdeb40f5410 0x7fdeb4109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:55:27.623 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:27.619+0000 7fdebbc6a700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeb4060240 con 0x7fdeb40f5410 2023-11-15T01:55:27.623 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:27.619+0000 7fdeb9a06700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdeb40f5410 0x7fdeb4109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:55:27.623 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:27.619+0000 7fdeb9a06700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdeb40f5410 0x7fdeb4109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:54210/0 (socket says 172.21.15.137:54210) 2023-11-15T01:55:27.623 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:27.619+0000 7fdeb9a06700 1 -- 172.21.15.137:0/1778810937 learned_addr learned my addr 172.21.15.137:0/1778810937 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:55:27.624 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:27.620+0000 7fdeb9a06700 1 -- 172.21.15.137:0/1778810937 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdea4005040 con 0x7fdeb40f5410 2023-11-15T01:55:27.624 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:27.620+0000 7fdeb9a06700 1 --2- 172.21.15.137:0/1778810937 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdeb40f5410 0x7fdeb4109430 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7fdea4013510 tx=0x7fdea400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:55:27.624 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:27.620+0000 7fdeaa7fc700 1 -- 172.21.15.137:0/1778810937 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdea4009320 con 0x7fdeb40f5410 2023-11-15T01:55:27.624 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:27.620+0000 7fdebbc6a700 1 -- 172.21.15.137:0/1778810937 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdeb4060440 con 0x7fdeb40f5410 2023-11-15T01:55:27.624 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:27.620+0000 7fdeaa7fc700 1 -- 172.21.15.137:0/1778810937 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdea4013f00 con 0x7fdeb40f5410 2023-11-15T01:55:27.625 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:27.620+0000 7fdeaa7fc700 1 -- 172.21.15.137:0/1778810937 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdea4014d90 con 0x7fdeb40f5410 2023-11-15T01:55:27.625 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:27.620+0000 7fdebbc6a700 1 -- 172.21.15.137:0/1778810937 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdeb4109ae0 con 0x7fdeb40f5410 2023-11-15T01:55:27.626 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:27.621+0000 7fdebbc6a700 1 -- 172.21.15.137:0/1778810937 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdeb4045ad0 con 0x7fdeb40f5410 2023-11-15T01:55:27.626 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:27.621+0000 7fdeaa7fc700 1 -- 172.21.15.137:0/1778810937 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdea40095f0 con 0x7fdeb40f5410 2023-11-15T01:55:27.626 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:27.621+0000 7fdeaa7fc700 1 --2- 172.21.15.137:0/1778810937 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fdea002ff60 0x7fdea0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:55:27.626 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:27.621+0000 7fdeaa7fc700 1 -- 172.21.15.137:0/1778810937 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdea4041000 con 0x7fdeb40f5410 2023-11-15T01:55:27.627 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:27.621+0000 7fdeb9205700 1 --2- 172.21.15.137:0/1778810937 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fdea002ff60 0x7fdea0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:55:27.627 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:27.622+0000 7fdeb9205700 1 --2- 172.21.15.137:0/1778810937 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fdea002ff60 0x7fdea0032420 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fdeb00096a0 tx=0x7fdeb0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:55:27.628 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:27.626+0000 7fdeaa7fc700 1 -- 172.21.15.137:0/1778810937 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdea4014420 con 0x7fdeb40f5410 2023-11-15T01:55:27.832 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:27.831+0000 7fdebbc6a700 1 -- 172.21.15.137:0/1778810937 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdeb40fae60 con 0x7fdeb40f5410 2023-11-15T01:55:27.832 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:27.832+0000 7fdeaa7fc700 1 -- 172.21.15.137:0/1778810937 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdea4010230 con 0x7fdeb40f5410 2023-11-15T01:55:27.833 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:55:27.833 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:55:27.834 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:27.833+0000 7fdebbc6a700 1 -- 172.21.15.137:0/1778810937 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fdea002ff60 msgr2=0x7fdea0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:55:27.834 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:27.833+0000 7fdebbc6a700 1 --2- 172.21.15.137:0/1778810937 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fdea002ff60 0x7fdea0032420 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fdeb00096a0 tx=0x7fdeb0006b40).stop 2023-11-15T01:55:27.834 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:27.833+0000 7fdebbc6a700 1 -- 172.21.15.137:0/1778810937 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdeb40f5410 msgr2=0x7fdeb4109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:55:27.835 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:27.833+0000 7fdebbc6a700 1 --2- 172.21.15.137:0/1778810937 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdeb40f5410 0x7fdeb4109430 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7fdea4013510 tx=0x7fdea400b040).stop 2023-11-15T01:55:27.835 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:27.833+0000 7fdebbc6a700 1 -- 172.21.15.137:0/1778810937 shutdown_connections 2023-11-15T01:55:27.835 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:27.833+0000 7fdebbc6a700 1 --2- 172.21.15.137:0/1778810937 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fdea002ff60 0x7fdea0032420 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:55:27.835 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:27.833+0000 7fdebbc6a700 1 --2- 172.21.15.137:0/1778810937 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdeb40f5410 0x7fdeb4109430 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:55:27.835 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:27.833+0000 7fdebbc6a700 1 -- 172.21.15.137:0/1778810937 >> 172.21.15.137:0/1778810937 conn(0x7fdeb40f0b70 msgr2=0x7fdeb40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:55:27.835 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:27.833+0000 7fdebbc6a700 1 -- 172.21.15.137:0/1778810937 shutdown_connections 2023-11-15T01:55:27.836 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:27.833+0000 7fdebbc6a700 1 -- 172.21.15.137:0/1778810937 wait complete. 2023-11-15T01:55:27.836 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:55:28.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:28 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:55:26.744906+0000 mgr.smithi055.szwngw (mgr.14164) 177 : cluster [DBG] pgmap v166: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:55:28.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:28 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:55:27.832814+0000 mon.smithi055 (mon.0) 248 : audit [DBG] from='client.? 172.21.15.137:0/1778810937' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:55:29.215 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:55:29.215 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:55:30.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:30 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:55:28.745276+0000 mgr.smithi055.szwngw (mgr.14164) 178 : cluster [DBG] pgmap v167: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:55:30.359 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:30.358+0000 7fa3fe780700 1 -- 172.21.15.137:0/2253853721 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa3f80f6510 msgr2=0x7fa3f80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:55:30.359 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:30.358+0000 7fa3fe780700 1 --2- 172.21.15.137:0/2253853721 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa3f80f6510 0x7fa3f80f6930 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7fa3e8009fb0 tx=0x7fa3e8009b30).stop 2023-11-15T01:55:30.359 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:30.358+0000 7fa3fe780700 1 -- 172.21.15.137:0/2253853721 shutdown_connections 2023-11-15T01:55:30.359 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:30.358+0000 7fa3fe780700 1 --2- 172.21.15.137:0/2253853721 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa3f80f6510 0x7fa3f80f6930 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:55:30.360 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:30.358+0000 7fa3fe780700 1 -- 172.21.15.137:0/2253853721 >> 172.21.15.137:0/2253853721 conn(0x7fa3f80f1c70 msgr2=0x7fa3f80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:55:30.360 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:30.358+0000 7fa3fe780700 1 -- 172.21.15.137:0/2253853721 shutdown_connections 2023-11-15T01:55:30.360 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:30.358+0000 7fa3fe780700 1 -- 172.21.15.137:0/2253853721 wait complete. 2023-11-15T01:55:30.362 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:30.358+0000 7fa3fe780700 1 Processor -- start 2023-11-15T01:55:30.362 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:30.359+0000 7fa3fe780700 1 -- start start 2023-11-15T01:55:30.362 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:30.359+0000 7fa3fe780700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa3f80f6510 0x7fa3f8106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:55:30.364 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:30.359+0000 7fa3fe780700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3f8106c20 con 0x7fa3f80f6510 2023-11-15T01:55:30.364 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:30.359+0000 7fa3f7fff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa3f80f6510 0x7fa3f8106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:55:30.364 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:30.359+0000 7fa3f7fff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa3f80f6510 0x7fa3f8106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:36670/0 (socket says 172.21.15.137:36670) 2023-11-15T01:55:30.364 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:30.359+0000 7fa3f7fff700 1 -- 172.21.15.137:0/1556970706 learned_addr learned my addr 172.21.15.137:0/1556970706 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:55:30.365 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:30.360+0000 7fa3f7fff700 1 -- 172.21.15.137:0/1556970706 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3e8005040 con 0x7fa3f80f6510 2023-11-15T01:55:30.365 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:30.360+0000 7fa3f7fff700 1 --2- 172.21.15.137:0/1556970706 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa3f80f6510 0x7fa3f8106730 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7fa3e8014000 tx=0x7fa3e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:55:30.365 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:30.360+0000 7fa3f4ff9700 1 -- 172.21.15.137:0/1556970706 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa3e80095f0 con 0x7fa3f80f6510 2023-11-15T01:55:30.365 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:30.360+0000 7fa3fe780700 1 -- 172.21.15.137:0/1556970706 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3f8106e20 con 0x7fa3f80f6510 2023-11-15T01:55:30.365 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:30.360+0000 7fa3f4ff9700 1 -- 172.21.15.137:0/1556970706 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa3e8007e70 con 0x7fa3f80f6510 2023-11-15T01:55:30.365 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:30.360+0000 7fa3f4ff9700 1 -- 172.21.15.137:0/1556970706 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa3e8014ea0 con 0x7fa3f80f6510 2023-11-15T01:55:30.366 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:30.360+0000 7fa3fe780700 1 -- 172.21.15.137:0/1556970706 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3f81072b0 con 0x7fa3f80f6510 2023-11-15T01:55:30.366 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:30.361+0000 7fa3fe780700 1 -- 172.21.15.137:0/1556970706 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3f8045ad0 con 0x7fa3f80f6510 2023-11-15T01:55:30.366 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:30.361+0000 7fa3f4ff9700 1 -- 172.21.15.137:0/1556970706 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa3e8025070 con 0x7fa3f80f6510 2023-11-15T01:55:30.366 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:30.361+0000 7fa3f4ff9700 1 --2- 172.21.15.137:0/1556970706 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fa3e002ff60 0x7fa3e0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:55:30.366 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:30.361+0000 7fa3f4ff9700 1 -- 172.21.15.137:0/1556970706 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa3e80409b0 con 0x7fa3f80f6510 2023-11-15T01:55:30.367 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:30.361+0000 7fa3f77fe700 1 --2- 172.21.15.137:0/1556970706 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fa3e002ff60 0x7fa3e0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:55:30.367 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:30.362+0000 7fa3f77fe700 1 --2- 172.21.15.137:0/1556970706 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fa3e002ff60 0x7fa3e0032420 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7fa3ec0096a0 tx=0x7fa3ec006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:55:30.367 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:30.366+0000 7fa3f4ff9700 1 -- 172.21.15.137:0/1556970706 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa3e80079c0 con 0x7fa3f80f6510 2023-11-15T01:55:30.573 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:30.572+0000 7fa3fe780700 1 -- 172.21.15.137:0/1556970706 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa3f80fbf60 con 0x7fa3f80f6510 2023-11-15T01:55:30.573 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:30.573+0000 7fa3f4ff9700 1 -- 172.21.15.137:0/1556970706 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa3e800f300 con 0x7fa3f80f6510 2023-11-15T01:55:30.574 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:55:30.574 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:55:30.575 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:30.574+0000 7fa3fe780700 1 -- 172.21.15.137:0/1556970706 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fa3e002ff60 msgr2=0x7fa3e0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:55:30.575 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:30.574+0000 7fa3fe780700 1 --2- 172.21.15.137:0/1556970706 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fa3e002ff60 0x7fa3e0032420 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7fa3ec0096a0 tx=0x7fa3ec006b40).stop 2023-11-15T01:55:30.576 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:30.574+0000 7fa3fe780700 1 -- 172.21.15.137:0/1556970706 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa3f80f6510 msgr2=0x7fa3f8106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:55:30.576 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:30.574+0000 7fa3fe780700 1 --2- 172.21.15.137:0/1556970706 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa3f80f6510 0x7fa3f8106730 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7fa3e8014000 tx=0x7fa3e800b040).stop 2023-11-15T01:55:30.576 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:30.574+0000 7fa3fe780700 1 -- 172.21.15.137:0/1556970706 shutdown_connections 2023-11-15T01:55:30.576 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:30.574+0000 7fa3fe780700 1 --2- 172.21.15.137:0/1556970706 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fa3e002ff60 0x7fa3e0032420 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:55:30.576 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:30.574+0000 7fa3fe780700 1 --2- 172.21.15.137:0/1556970706 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa3f80f6510 0x7fa3f8106730 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:55:30.577 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:30.574+0000 7fa3fe780700 1 -- 172.21.15.137:0/1556970706 >> 172.21.15.137:0/1556970706 conn(0x7fa3f80f1c70 msgr2=0x7fa3f80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:55:30.577 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:30.574+0000 7fa3fe780700 1 -- 172.21.15.137:0/1556970706 shutdown_connections 2023-11-15T01:55:30.577 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:30.574+0000 7fa3fe780700 1 -- 172.21.15.137:0/1556970706 wait complete. 2023-11-15T01:55:30.577 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:55:31.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:31 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:55:30.573780+0000 mon.smithi055 (mon.0) 249 : audit [DBG] from='client.? 172.21.15.137:0/1556970706' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:55:31.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:31 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:55:30.745634+0000 mgr.smithi055.szwngw (mgr.14164) 179 : cluster [DBG] pgmap v168: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:55:31.960 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:55:31.961 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:55:33.085 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:33.083+0000 7f0c9759e700 1 -- 172.21.15.137:0/486795635 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0c980f51a0 msgr2=0x7f0c980f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:55:33.085 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:33.083+0000 7f0c9759e700 1 --2- 172.21.15.137:0/486795635 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0c980f51a0 0x7f0c980f55c0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7f0c88009fb0 tx=0x7f0c88009b30).stop 2023-11-15T01:55:33.086 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:33.084+0000 7f0c9759e700 1 -- 172.21.15.137:0/486795635 shutdown_connections 2023-11-15T01:55:33.086 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:33.084+0000 7f0c9759e700 1 --2- 172.21.15.137:0/486795635 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0c980f51a0 0x7f0c980f55c0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:55:33.086 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:33.084+0000 7f0c9759e700 1 -- 172.21.15.137:0/486795635 >> 172.21.15.137:0/486795635 conn(0x7f0c980f09a0 msgr2=0x7f0c980f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:55:33.086 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:33.084+0000 7f0c9759e700 1 -- 172.21.15.137:0/486795635 shutdown_connections 2023-11-15T01:55:33.086 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:33.084+0000 7f0c9759e700 1 -- 172.21.15.137:0/486795635 wait complete. 2023-11-15T01:55:33.087 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:33.084+0000 7f0c9759e700 1 Processor -- start 2023-11-15T01:55:33.087 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:33.084+0000 7f0c9759e700 1 -- start start 2023-11-15T01:55:33.087 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:33.084+0000 7f0c9759e700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0c980f51a0 0x7f0c981064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:55:33.087 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:33.084+0000 7f0c9759e700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c981069a0 con 0x7f0c980f51a0 2023-11-15T01:55:33.087 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:33.085+0000 7f0c9659c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0c980f51a0 0x7f0c981064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:55:33.087 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:33.085+0000 7f0c9659c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0c980f51a0 0x7f0c981064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:36696/0 (socket says 172.21.15.137:36696) 2023-11-15T01:55:33.088 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:33.085+0000 7f0c9659c700 1 -- 172.21.15.137:0/3971307545 learned_addr learned my addr 172.21.15.137:0/3971307545 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:55:33.089 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:33.085+0000 7f0c9659c700 1 -- 172.21.15.137:0/3971307545 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c88005040 con 0x7f0c980f51a0 2023-11-15T01:55:33.089 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:33.085+0000 7f0c9659c700 1 --2- 172.21.15.137:0/3971307545 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0c980f51a0 0x7f0c981064b0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f0c88013510 tx=0x7f0c8800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:55:33.089 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:33.085+0000 7f0c86ffd700 1 -- 172.21.15.137:0/3971307545 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c88009320 con 0x7f0c980f51a0 2023-11-15T01:55:33.090 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:33.085+0000 7f0c86ffd700 1 -- 172.21.15.137:0/3971307545 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0c88013f00 con 0x7f0c980f51a0 2023-11-15T01:55:33.090 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:33.085+0000 7f0c9759e700 1 -- 172.21.15.137:0/3971307545 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c98106ba0 con 0x7f0c980f51a0 2023-11-15T01:55:33.090 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:33.085+0000 7f0c86ffd700 1 -- 172.21.15.137:0/3971307545 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c88014dc0 con 0x7f0c980f51a0 2023-11-15T01:55:33.090 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:33.085+0000 7f0c9759e700 1 -- 172.21.15.137:0/3971307545 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c98107030 con 0x7f0c980f51a0 2023-11-15T01:55:33.090 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:33.086+0000 7f0c86ffd700 1 -- 172.21.15.137:0/3971307545 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0c880095f0 con 0x7f0c980f51a0 2023-11-15T01:55:33.091 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:33.086+0000 7f0c9759e700 1 -- 172.21.15.137:0/3971307545 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c98045ad0 con 0x7f0c980f51a0 2023-11-15T01:55:33.091 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:33.086+0000 7f0c86ffd700 1 --2- 172.21.15.137:0/3971307545 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f0c8002ff60 0x7f0c80032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:55:33.091 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:33.086+0000 7f0c86ffd700 1 -- 172.21.15.137:0/3971307545 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0c88041000 con 0x7f0c980f51a0 2023-11-15T01:55:33.091 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:33.087+0000 7f0c95d9b700 1 --2- 172.21.15.137:0/3971307545 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f0c8002ff60 0x7f0c80032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:55:33.091 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:33.087+0000 7f0c95d9b700 1 --2- 172.21.15.137:0/3971307545 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f0c8002ff60 0x7f0c80032420 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f0c8c0096a0 tx=0x7f0c8c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:55:33.092 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:33.091+0000 7f0c86ffd700 1 -- 172.21.15.137:0/3971307545 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0c8801c790 con 0x7f0c980f51a0 2023-11-15T01:55:33.302 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:33.301+0000 7f0c9759e700 1 -- 172.21.15.137:0/3971307545 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0c98107ee0 con 0x7f0c980f51a0 2023-11-15T01:55:33.303 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:33.302+0000 7f0c86ffd700 1 -- 172.21.15.137:0/3971307545 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0c88010230 con 0x7f0c980f51a0 2023-11-15T01:55:33.303 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:55:33.303 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:55:33.304 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:33.303+0000 7f0c9759e700 1 -- 172.21.15.137:0/3971307545 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f0c8002ff60 msgr2=0x7f0c80032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:55:33.305 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:33.303+0000 7f0c9759e700 1 --2- 172.21.15.137:0/3971307545 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f0c8002ff60 0x7f0c80032420 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f0c8c0096a0 tx=0x7f0c8c006b40).stop 2023-11-15T01:55:33.305 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:33.303+0000 7f0c9759e700 1 -- 172.21.15.137:0/3971307545 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0c980f51a0 msgr2=0x7f0c981064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:55:33.305 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:33.303+0000 7f0c9759e700 1 --2- 172.21.15.137:0/3971307545 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0c980f51a0 0x7f0c981064b0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f0c88013510 tx=0x7f0c8800b040).stop 2023-11-15T01:55:33.305 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:33.303+0000 7f0c9759e700 1 -- 172.21.15.137:0/3971307545 shutdown_connections 2023-11-15T01:55:33.305 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:33.303+0000 7f0c9759e700 1 --2- 172.21.15.137:0/3971307545 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f0c8002ff60 0x7f0c80032420 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:55:33.306 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:33.303+0000 7f0c9759e700 1 --2- 172.21.15.137:0/3971307545 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0c980f51a0 0x7f0c981064b0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:55:33.306 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:33.303+0000 7f0c9759e700 1 -- 172.21.15.137:0/3971307545 >> 172.21.15.137:0/3971307545 conn(0x7f0c980f09a0 msgr2=0x7f0c980f1400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:55:33.306 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:33.303+0000 7f0c9759e700 1 -- 172.21.15.137:0/3971307545 shutdown_connections 2023-11-15T01:55:33.306 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:33.303+0000 7f0c9759e700 1 -- 172.21.15.137:0/3971307545 wait complete. 2023-11-15T01:55:33.306 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:55:34.077 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:33 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:55:32.745929+0000 mgr.smithi055.szwngw (mgr.14164) 180 : cluster [DBG] pgmap v169: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:55:34.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:33 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:55:33.302847+0000 mon.smithi055 (mon.0) 250 : audit [DBG] from='client.? 172.21.15.137:0/3971307545' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:55:34.675 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:55:34.676 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:55:35.788 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:35.787+0000 7f3168cb5700 1 -- 172.21.15.137:0/3455629192 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f31640f5410 msgr2=0x7f31640f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:55:35.788 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:35.787+0000 7f3168cb5700 1 --2- 172.21.15.137:0/3455629192 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f31640f5410 0x7f31640f5830 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7f314c009fb0 tx=0x7f314c009b30).stop 2023-11-15T01:55:35.788 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:35.787+0000 7f3168cb5700 1 -- 172.21.15.137:0/3455629192 shutdown_connections 2023-11-15T01:55:35.789 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:35.787+0000 7f3168cb5700 1 --2- 172.21.15.137:0/3455629192 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f31640f5410 0x7f31640f5830 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:55:35.789 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:35.787+0000 7f3168cb5700 1 -- 172.21.15.137:0/3455629192 >> 172.21.15.137:0/3455629192 conn(0x7f31640f0b70 msgr2=0x7f31640f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:55:35.789 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:35.787+0000 7f3168cb5700 1 -- 172.21.15.137:0/3455629192 shutdown_connections 2023-11-15T01:55:35.790 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:35.787+0000 7f3168cb5700 1 -- 172.21.15.137:0/3455629192 wait complete. 2023-11-15T01:55:35.792 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:35.788+0000 7f3168cb5700 1 Processor -- start 2023-11-15T01:55:35.792 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:35.788+0000 7f3168cb5700 1 -- start start 2023-11-15T01:55:35.792 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:35.788+0000 7f3168cb5700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f31640f5410 0x7f3164109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:55:35.793 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:35.788+0000 7f3168cb5700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3164060240 con 0x7f31640f5410 2023-11-15T01:55:35.793 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:35.788+0000 7f316259c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f31640f5410 0x7f3164109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:55:35.793 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:35.788+0000 7f316259c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f31640f5410 0x7f3164109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:36716/0 (socket says 172.21.15.137:36716) 2023-11-15T01:55:35.793 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:35.788+0000 7f316259c700 1 -- 172.21.15.137:0/1446636268 learned_addr learned my addr 172.21.15.137:0/1446636268 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:55:35.794 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:35.788+0000 7f316259c700 1 -- 172.21.15.137:0/1446636268 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f314c005040 con 0x7f31640f5410 2023-11-15T01:55:35.794 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:35.788+0000 7f316259c700 1 --2- 172.21.15.137:0/1446636268 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f31640f5410 0x7f3164109430 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f314c014000 tx=0x7f314c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:55:35.794 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:35.789+0000 7f315affd700 1 -- 172.21.15.137:0/1446636268 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f314c0095f0 con 0x7f31640f5410 2023-11-15T01:55:35.795 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:35.789+0000 7f315affd700 1 -- 172.21.15.137:0/1446636268 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f314c007e70 con 0x7f31640f5410 2023-11-15T01:55:35.795 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:35.789+0000 7f3168cb5700 1 -- 172.21.15.137:0/1446636268 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3164060440 con 0x7f31640f5410 2023-11-15T01:55:35.795 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:35.789+0000 7f315affd700 1 -- 172.21.15.137:0/1446636268 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f314c014ea0 con 0x7f31640f5410 2023-11-15T01:55:35.795 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:35.789+0000 7f3168cb5700 1 -- 172.21.15.137:0/1446636268 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3164109ae0 con 0x7f31640f5410 2023-11-15T01:55:35.795 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:35.790+0000 7f315affd700 1 -- 172.21.15.137:0/1446636268 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f314c025070 con 0x7f31640f5410 2023-11-15T01:55:35.796 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:35.790+0000 7f315affd700 1 --2- 172.21.15.137:0/1446636268 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f315002ff60 0x7f3150032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:55:35.796 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:35.790+0000 7f3168cb5700 1 -- 172.21.15.137:0/1446636268 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3164045ad0 con 0x7f31640f5410 2023-11-15T01:55:35.796 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:35.790+0000 7f315affd700 1 -- 172.21.15.137:0/1446636268 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f314c040c40 con 0x7f31640f5410 2023-11-15T01:55:35.796 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:35.790+0000 7f3161d9b700 1 --2- 172.21.15.137:0/1446636268 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f315002ff60 0x7f3150032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:55:35.796 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:35.790+0000 7f3161d9b700 1 --2- 172.21.15.137:0/1446636268 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f315002ff60 0x7f3150032420 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f31540096a0 tx=0x7f3154006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:55:35.797 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:35.794+0000 7f315affd700 1 -- 172.21.15.137:0/1446636268 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f314c01b2c0 con 0x7f31640f5410 2023-11-15T01:55:36.002 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:36.001+0000 7f3168cb5700 1 -- 172.21.15.137:0/1446636268 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f31640faf50 con 0x7f31640f5410 2023-11-15T01:55:36.003 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:36.002+0000 7f315affd700 1 -- 172.21.15.137:0/1446636268 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f314c025b30 con 0x7f31640f5410 2023-11-15T01:55:36.004 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:55:36.004 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:55:36.005 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:36.003+0000 7f3168cb5700 1 -- 172.21.15.137:0/1446636268 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f315002ff60 msgr2=0x7f3150032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:55:36.006 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:36.003+0000 7f3168cb5700 1 --2- 172.21.15.137:0/1446636268 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f315002ff60 0x7f3150032420 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f31540096a0 tx=0x7f3154006b40).stop 2023-11-15T01:55:36.006 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:36.003+0000 7f3168cb5700 1 -- 172.21.15.137:0/1446636268 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f31640f5410 msgr2=0x7f3164109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:55:36.006 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:36.003+0000 7f3168cb5700 1 --2- 172.21.15.137:0/1446636268 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f31640f5410 0x7f3164109430 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f314c014000 tx=0x7f314c00b040).stop 2023-11-15T01:55:36.006 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:36.004+0000 7f3168cb5700 1 -- 172.21.15.137:0/1446636268 shutdown_connections 2023-11-15T01:55:36.006 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:36.004+0000 7f3168cb5700 1 --2- 172.21.15.137:0/1446636268 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f315002ff60 0x7f3150032420 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:55:36.007 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:36.004+0000 7f3168cb5700 1 --2- 172.21.15.137:0/1446636268 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f31640f5410 0x7f3164109430 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:55:36.007 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:36.004+0000 7f3168cb5700 1 -- 172.21.15.137:0/1446636268 >> 172.21.15.137:0/1446636268 conn(0x7f31640f0b70 msgr2=0x7f31640f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:55:36.007 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:36.004+0000 7f3168cb5700 1 -- 172.21.15.137:0/1446636268 shutdown_connections 2023-11-15T01:55:36.007 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:36.004+0000 7f3168cb5700 1 -- 172.21.15.137:0/1446636268 wait complete. 2023-11-15T01:55:36.008 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:55:36.077 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:35 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:55:34.746193+0000 mgr.smithi055.szwngw (mgr.14164) 181 : cluster [DBG] pgmap v170: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:55:37.077 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:36 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:55:36.003064+0000 mon.smithi055 (mon.0) 251 : audit [DBG] from='client.? 172.21.15.137:0/1446636268' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:55:37.398 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:55:37.399 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:55:38.077 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:37 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:55:36.746443+0000 mgr.smithi055.szwngw (mgr.14164) 182 : cluster [DBG] pgmap v171: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:55:38.588 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:38.587+0000 7fc245779700 1 -- 172.21.15.137:0/698568811 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc2400f5410 msgr2=0x7fc2400f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:55:38.588 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:38.587+0000 7fc245779700 1 --2- 172.21.15.137:0/698568811 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc2400f5410 0x7fc2400f5830 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7fc234009fb0 tx=0x7fc234009b30).stop 2023-11-15T01:55:38.588 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:38.587+0000 7fc245779700 1 -- 172.21.15.137:0/698568811 shutdown_connections 2023-11-15T01:55:38.588 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:38.587+0000 7fc245779700 1 --2- 172.21.15.137:0/698568811 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc2400f5410 0x7fc2400f5830 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:55:38.589 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:38.587+0000 7fc245779700 1 -- 172.21.15.137:0/698568811 >> 172.21.15.137:0/698568811 conn(0x7fc2400f0b70 msgr2=0x7fc2400f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:55:38.589 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:38.587+0000 7fc245779700 1 -- 172.21.15.137:0/698568811 shutdown_connections 2023-11-15T01:55:38.589 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:38.587+0000 7fc245779700 1 -- 172.21.15.137:0/698568811 wait complete. 2023-11-15T01:55:38.591 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:38.588+0000 7fc245779700 1 Processor -- start 2023-11-15T01:55:38.591 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:38.588+0000 7fc245779700 1 -- start start 2023-11-15T01:55:38.592 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:38.588+0000 7fc245779700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc2400f5410 0x7fc2401093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:55:38.592 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:38.588+0000 7fc245779700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc240060280 con 0x7fc2400f5410 2023-11-15T01:55:38.592 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:38.588+0000 7fc23effd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc2400f5410 0x7fc2401093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:55:38.592 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:38.588+0000 7fc23effd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc2400f5410 0x7fc2401093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:47070/0 (socket says 172.21.15.137:47070) 2023-11-15T01:55:38.592 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:38.588+0000 7fc23effd700 1 -- 172.21.15.137:0/4294635784 learned_addr learned my addr 172.21.15.137:0/4294635784 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:55:38.593 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:38.588+0000 7fc23effd700 1 -- 172.21.15.137:0/4294635784 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc234005040 con 0x7fc2400f5410 2023-11-15T01:55:38.593 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:38.588+0000 7fc23effd700 1 --2- 172.21.15.137:0/4294635784 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc2400f5410 0x7fc2401093f0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7fc234013510 tx=0x7fc23400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:55:38.593 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:38.588+0000 7fc2277fe700 1 -- 172.21.15.137:0/4294635784 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc234009320 con 0x7fc2400f5410 2023-11-15T01:55:38.593 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:38.588+0000 7fc2277fe700 1 -- 172.21.15.137:0/4294635784 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc234013f00 con 0x7fc2400f5410 2023-11-15T01:55:38.593 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:38.588+0000 7fc245779700 1 -- 172.21.15.137:0/4294635784 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc240060480 con 0x7fc2400f5410 2023-11-15T01:55:38.594 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:38.589+0000 7fc2277fe700 1 -- 172.21.15.137:0/4294635784 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc234014d90 con 0x7fc2400f5410 2023-11-15T01:55:38.594 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:38.589+0000 7fc245779700 1 -- 172.21.15.137:0/4294635784 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc240109b20 con 0x7fc2400f5410 2023-11-15T01:55:38.594 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:38.589+0000 7fc2277fe700 1 -- 172.21.15.137:0/4294635784 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc2340095f0 con 0x7fc2400f5410 2023-11-15T01:55:38.594 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:38.589+0000 7fc245779700 1 -- 172.21.15.137:0/4294635784 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc240045ad0 con 0x7fc2400f5410 2023-11-15T01:55:38.594 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:38.589+0000 7fc2277fe700 1 --2- 172.21.15.137:0/4294635784 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fc22802ff60 0x7fc228032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:55:38.595 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:38.589+0000 7fc2277fe700 1 -- 172.21.15.137:0/4294635784 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc234041080 con 0x7fc2400f5410 2023-11-15T01:55:38.595 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:38.590+0000 7fc23e7fc700 1 --2- 172.21.15.137:0/4294635784 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fc22802ff60 0x7fc228032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:55:38.595 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:38.590+0000 7fc23e7fc700 1 --2- 172.21.15.137:0/4294635784 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fc22802ff60 0x7fc228032420 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fc2300096a0 tx=0x7fc230006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:55:38.595 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:38.595+0000 7fc2277fe700 1 -- 172.21.15.137:0/4294635784 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc23401d7f0 con 0x7fc2400f5410 2023-11-15T01:55:38.801 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:38.801+0000 7fc245779700 1 -- 172.21.15.137:0/4294635784 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc24010a340 con 0x7fc2400f5410 2023-11-15T01:55:38.802 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:38.801+0000 7fc2277fe700 1 -- 172.21.15.137:0/4294635784 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc234010230 con 0x7fc2400f5410 2023-11-15T01:55:38.802 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:55:38.803 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:55:38.803 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:38.803+0000 7fc245779700 1 -- 172.21.15.137:0/4294635784 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fc22802ff60 msgr2=0x7fc228032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:55:38.804 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:38.803+0000 7fc245779700 1 --2- 172.21.15.137:0/4294635784 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fc22802ff60 0x7fc228032420 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fc2300096a0 tx=0x7fc230006b40).stop 2023-11-15T01:55:38.804 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:38.803+0000 7fc245779700 1 -- 172.21.15.137:0/4294635784 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc2400f5410 msgr2=0x7fc2401093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:55:38.804 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:38.803+0000 7fc245779700 1 --2- 172.21.15.137:0/4294635784 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc2400f5410 0x7fc2401093f0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7fc234013510 tx=0x7fc23400b040).stop 2023-11-15T01:55:38.804 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:38.803+0000 7fc245779700 1 -- 172.21.15.137:0/4294635784 shutdown_connections 2023-11-15T01:55:38.804 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:38.803+0000 7fc245779700 1 --2- 172.21.15.137:0/4294635784 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fc22802ff60 0x7fc228032420 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:55:38.805 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:38.803+0000 7fc245779700 1 --2- 172.21.15.137:0/4294635784 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc2400f5410 0x7fc2401093f0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:55:38.805 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:38.803+0000 7fc245779700 1 -- 172.21.15.137:0/4294635784 >> 172.21.15.137:0/4294635784 conn(0x7fc2400f0b70 msgr2=0x7fc2400f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:55:38.805 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:38.803+0000 7fc245779700 1 -- 172.21.15.137:0/4294635784 shutdown_connections 2023-11-15T01:55:38.805 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:38.803+0000 7fc245779700 1 -- 172.21.15.137:0/4294635784 wait complete. 2023-11-15T01:55:38.806 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:55:39.077 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:38 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:55:38.802334+0000 mon.smithi055 (mon.0) 252 : audit [DBG] from='client.? 172.21.15.137:0/4294635784' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:55:40.077 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:39 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:55:38.746703+0000 mgr.smithi055.szwngw (mgr.14164) 183 : cluster [DBG] pgmap v172: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:55:40.195 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:55:40.195 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:55:41.323 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:41.321+0000 7fd4db410700 1 -- 172.21.15.137:0/2944005966 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd4d40f5410 msgr2=0x7fd4d40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:55:41.324 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:41.321+0000 7fd4db410700 1 --2- 172.21.15.137:0/2944005966 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd4d40f5410 0x7fd4d40f5830 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7fd4c4009fb0 tx=0x7fd4c4009b30).stop 2023-11-15T01:55:41.324 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:41.321+0000 7fd4db410700 1 -- 172.21.15.137:0/2944005966 shutdown_connections 2023-11-15T01:55:41.324 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:41.321+0000 7fd4db410700 1 --2- 172.21.15.137:0/2944005966 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd4d40f5410 0x7fd4d40f5830 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:55:41.324 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:41.321+0000 7fd4db410700 1 -- 172.21.15.137:0/2944005966 >> 172.21.15.137:0/2944005966 conn(0x7fd4d40f0b70 msgr2=0x7fd4d40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:55:41.325 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:41.321+0000 7fd4db410700 1 -- 172.21.15.137:0/2944005966 shutdown_connections 2023-11-15T01:55:41.325 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:41.321+0000 7fd4db410700 1 -- 172.21.15.137:0/2944005966 wait complete. 2023-11-15T01:55:41.325 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:41.322+0000 7fd4db410700 1 Processor -- start 2023-11-15T01:55:41.325 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:41.322+0000 7fd4db410700 1 -- start start 2023-11-15T01:55:41.325 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:41.322+0000 7fd4db410700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd4d40f5410 0x7fd4d4105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:55:41.326 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:41.322+0000 7fd4db410700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4d4105b20 con 0x7fd4d40f5410 2023-11-15T01:55:41.326 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:41.322+0000 7fd4d91ac700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd4d40f5410 0x7fd4d4105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:55:41.326 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:41.322+0000 7fd4d91ac700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd4d40f5410 0x7fd4d4105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:47080/0 (socket says 172.21.15.137:47080) 2023-11-15T01:55:41.326 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:41.322+0000 7fd4d91ac700 1 -- 172.21.15.137:0/784325698 learned_addr learned my addr 172.21.15.137:0/784325698 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:55:41.326 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:41.322+0000 7fd4d91ac700 1 -- 172.21.15.137:0/784325698 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4c4005040 con 0x7fd4d40f5410 2023-11-15T01:55:41.327 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:41.322+0000 7fd4d91ac700 1 --2- 172.21.15.137:0/784325698 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd4d40f5410 0x7fd4d4105630 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7fd4c4013510 tx=0x7fd4c400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:55:41.327 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:41.323+0000 7fd4c9ffb700 1 -- 172.21.15.137:0/784325698 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd4c4009320 con 0x7fd4d40f5410 2023-11-15T01:55:41.327 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:41.323+0000 7fd4db410700 1 -- 172.21.15.137:0/784325698 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4d4105d20 con 0x7fd4d40f5410 2023-11-15T01:55:41.327 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:41.323+0000 7fd4c9ffb700 1 -- 172.21.15.137:0/784325698 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd4c4013f00 con 0x7fd4d40f5410 2023-11-15T01:55:41.327 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:41.323+0000 7fd4c9ffb700 1 -- 172.21.15.137:0/784325698 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd4c4014d90 con 0x7fd4d40f5410 2023-11-15T01:55:41.328 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:41.323+0000 7fd4db410700 1 -- 172.21.15.137:0/784325698 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4d41061b0 con 0x7fd4d40f5410 2023-11-15T01:55:41.329 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:41.324+0000 7fd4db410700 1 -- 172.21.15.137:0/784325698 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4d4045ad0 con 0x7fd4d40f5410 2023-11-15T01:55:41.329 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:41.324+0000 7fd4c9ffb700 1 -- 172.21.15.137:0/784325698 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd4c40095f0 con 0x7fd4d40f5410 2023-11-15T01:55:41.329 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:41.324+0000 7fd4c9ffb700 1 --2- 172.21.15.137:0/784325698 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fd4c002ff60 0x7fd4c0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:55:41.329 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:41.324+0000 7fd4c9ffb700 1 -- 172.21.15.137:0/784325698 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd4c4042d90 con 0x7fd4d40f5410 2023-11-15T01:55:41.329 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:41.324+0000 7fd4d89ab700 1 --2- 172.21.15.137:0/784325698 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fd4c002ff60 0x7fd4c0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:55:41.330 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:41.325+0000 7fd4d89ab700 1 --2- 172.21.15.137:0/784325698 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fd4c002ff60 0x7fd4c0032420 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7fd4d00096a0 tx=0x7fd4d0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:55:41.330 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:41.329+0000 7fd4c9ffb700 1 -- 172.21.15.137:0/784325698 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd4c4014420 con 0x7fd4d40f5410 2023-11-15T01:55:41.536 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:41.535+0000 7fd4db410700 1 -- 172.21.15.137:0/784325698 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd4d40fae60 con 0x7fd4d40f5410 2023-11-15T01:55:41.537 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:41.536+0000 7fd4c9ffb700 1 -- 172.21.15.137:0/784325698 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd4c4011290 con 0x7fd4d40f5410 2023-11-15T01:55:41.537 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:55:41.538 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:55:41.539 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:41.538+0000 7fd4db410700 1 -- 172.21.15.137:0/784325698 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fd4c002ff60 msgr2=0x7fd4c0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:55:41.539 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:41.538+0000 7fd4db410700 1 --2- 172.21.15.137:0/784325698 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fd4c002ff60 0x7fd4c0032420 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7fd4d00096a0 tx=0x7fd4d0006b40).stop 2023-11-15T01:55:41.539 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:41.538+0000 7fd4db410700 1 -- 172.21.15.137:0/784325698 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd4d40f5410 msgr2=0x7fd4d4105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:55:41.539 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:41.538+0000 7fd4db410700 1 --2- 172.21.15.137:0/784325698 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd4d40f5410 0x7fd4d4105630 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7fd4c4013510 tx=0x7fd4c400b040).stop 2023-11-15T01:55:41.540 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:41.538+0000 7fd4db410700 1 -- 172.21.15.137:0/784325698 shutdown_connections 2023-11-15T01:55:41.540 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:41.538+0000 7fd4db410700 1 --2- 172.21.15.137:0/784325698 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fd4c002ff60 0x7fd4c0032420 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:55:41.540 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:41.538+0000 7fd4db410700 1 --2- 172.21.15.137:0/784325698 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd4d40f5410 0x7fd4d4105630 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:55:41.540 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:41.538+0000 7fd4db410700 1 -- 172.21.15.137:0/784325698 >> 172.21.15.137:0/784325698 conn(0x7fd4d40f0b70 msgr2=0x7fd4d40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:55:41.540 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:41.538+0000 7fd4db410700 1 -- 172.21.15.137:0/784325698 shutdown_connections 2023-11-15T01:55:41.541 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:41.538+0000 7fd4db410700 1 -- 172.21.15.137:0/784325698 wait complete. 2023-11-15T01:55:41.541 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:55:41.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:41 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:55:40.746914+0000 mgr.smithi055.szwngw (mgr.14164) 184 : cluster [DBG] pgmap v173: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:55:42.815 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:42 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:55:41.537352+0000 mon.smithi055 (mon.0) 253 : audit [DBG] from='client.? 172.21.15.137:0/784325698' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:55:42.914 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:55:42.915 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:55:43.775 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:43 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:55:42.747206+0000 mgr.smithi055.szwngw (mgr.14164) 185 : cluster [DBG] pgmap v174: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:55:43.776 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:43 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:55:42.853563+0000 mon.smithi055 (mon.0) 254 : audit [INF] from='mgr.14164 172.21.15.55:0/59463371' entity='mgr.smithi055.szwngw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi055.szwngw/mirror_snapshot_schedule"}]: dispatch 2023-11-15T01:55:43.776 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:43 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:55:42.856114+0000 mon.smithi055 (mon.0) 255 : audit [INF] from='mgr.14164 172.21.15.55:0/59463371' entity='mgr.smithi055.szwngw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi055.szwngw/trash_purge_schedule"}]: dispatch 2023-11-15T01:55:44.047 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:44.046+0000 7fc82415e700 1 -- 172.21.15.137:0/79169413 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc81c0f5410 msgr2=0x7fc81c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:55:44.048 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:44.046+0000 7fc82415e700 1 --2- 172.21.15.137:0/79169413 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc81c0f5410 0x7fc81c0f5830 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7fc80c009fb0 tx=0x7fc80c009b30).stop 2023-11-15T01:55:44.048 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:44.046+0000 7fc82415e700 1 -- 172.21.15.137:0/79169413 shutdown_connections 2023-11-15T01:55:44.048 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:44.046+0000 7fc82415e700 1 --2- 172.21.15.137:0/79169413 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc81c0f5410 0x7fc81c0f5830 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:55:44.048 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:44.046+0000 7fc82415e700 1 -- 172.21.15.137:0/79169413 >> 172.21.15.137:0/79169413 conn(0x7fc81c0f0b70 msgr2=0x7fc81c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:55:44.049 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:44.046+0000 7fc82415e700 1 -- 172.21.15.137:0/79169413 shutdown_connections 2023-11-15T01:55:44.049 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:44.046+0000 7fc82415e700 1 -- 172.21.15.137:0/79169413 wait complete. 2023-11-15T01:55:44.049 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:44.047+0000 7fc82415e700 1 Processor -- start 2023-11-15T01:55:44.049 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:44.047+0000 7fc82415e700 1 -- start start 2023-11-15T01:55:44.051 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:44.047+0000 7fc82415e700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc81c0f5410 0x7fc81c109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:55:44.051 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:44.047+0000 7fc82415e700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc81c060240 con 0x7fc81c0f5410 2023-11-15T01:55:44.051 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:44.048+0000 7fc821efa700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc81c0f5410 0x7fc81c109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:55:44.052 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:44.048+0000 7fc821efa700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc81c0f5410 0x7fc81c109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:47092/0 (socket says 172.21.15.137:47092) 2023-11-15T01:55:44.052 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:44.048+0000 7fc821efa700 1 -- 172.21.15.137:0/514069239 learned_addr learned my addr 172.21.15.137:0/514069239 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:55:44.052 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:44.048+0000 7fc821efa700 1 -- 172.21.15.137:0/514069239 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc80c005040 con 0x7fc81c0f5410 2023-11-15T01:55:44.052 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:44.048+0000 7fc821efa700 1 --2- 172.21.15.137:0/514069239 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc81c0f5410 0x7fc81c109430 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7fc80c014000 tx=0x7fc80c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:55:44.053 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:44.048+0000 7fc8127fc700 1 -- 172.21.15.137:0/514069239 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc80c0095f0 con 0x7fc81c0f5410 2023-11-15T01:55:44.053 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:44.049+0000 7fc82415e700 1 -- 172.21.15.137:0/514069239 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc81c060440 con 0x7fc81c0f5410 2023-11-15T01:55:44.053 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:44.049+0000 7fc8127fc700 1 -- 172.21.15.137:0/514069239 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc80c007e70 con 0x7fc81c0f5410 2023-11-15T01:55:44.053 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:44.049+0000 7fc8127fc700 1 -- 172.21.15.137:0/514069239 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc80c014ea0 con 0x7fc81c0f5410 2023-11-15T01:55:44.053 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:44.049+0000 7fc82415e700 1 -- 172.21.15.137:0/514069239 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc81c109ae0 con 0x7fc81c0f5410 2023-11-15T01:55:44.053 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:44.049+0000 7fc8127fc700 1 -- 172.21.15.137:0/514069239 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc80c025070 con 0x7fc81c0f5410 2023-11-15T01:55:44.054 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:44.049+0000 7fc82415e700 1 -- 172.21.15.137:0/514069239 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc81c045ad0 con 0x7fc81c0f5410 2023-11-15T01:55:44.054 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:44.050+0000 7fc8127fc700 1 --2- 172.21.15.137:0/514069239 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fc80802ff60 0x7fc808032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:55:44.054 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:44.050+0000 7fc8127fc700 1 -- 172.21.15.137:0/514069239 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc80c040970 con 0x7fc81c0f5410 2023-11-15T01:55:44.054 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:44.050+0000 7fc8216f9700 1 --2- 172.21.15.137:0/514069239 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fc80802ff60 0x7fc808032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:55:44.054 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:44.050+0000 7fc8216f9700 1 --2- 172.21.15.137:0/514069239 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fc80802ff60 0x7fc808032420 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7fc8180096a0 tx=0x7fc818006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:55:44.055 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:44.054+0000 7fc8127fc700 1 -- 172.21.15.137:0/514069239 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc80c0079c0 con 0x7fc81c0f5410 2023-11-15T01:55:44.260 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:44.260+0000 7fc82415e700 1 -- 172.21.15.137:0/514069239 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc81c0fae60 con 0x7fc81c0f5410 2023-11-15T01:55:44.261 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:44.260+0000 7fc8127fc700 1 -- 172.21.15.137:0/514069239 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc80c00f300 con 0x7fc81c0f5410 2023-11-15T01:55:44.262 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:55:44.262 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:55:44.263 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:44.262+0000 7fc82415e700 1 -- 172.21.15.137:0/514069239 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fc80802ff60 msgr2=0x7fc808032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:55:44.263 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:44.262+0000 7fc82415e700 1 --2- 172.21.15.137:0/514069239 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fc80802ff60 0x7fc808032420 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7fc8180096a0 tx=0x7fc818006b40).stop 2023-11-15T01:55:44.263 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:44.262+0000 7fc82415e700 1 -- 172.21.15.137:0/514069239 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc81c0f5410 msgr2=0x7fc81c109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:55:44.264 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:44.262+0000 7fc82415e700 1 --2- 172.21.15.137:0/514069239 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc81c0f5410 0x7fc81c109430 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7fc80c014000 tx=0x7fc80c00b040).stop 2023-11-15T01:55:44.264 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:44.262+0000 7fc82415e700 1 -- 172.21.15.137:0/514069239 shutdown_connections 2023-11-15T01:55:44.264 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:44.262+0000 7fc82415e700 1 --2- 172.21.15.137:0/514069239 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fc80802ff60 0x7fc808032420 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:55:44.264 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:44.262+0000 7fc82415e700 1 --2- 172.21.15.137:0/514069239 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc81c0f5410 0x7fc81c109430 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:55:44.264 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:44.262+0000 7fc82415e700 1 -- 172.21.15.137:0/514069239 >> 172.21.15.137:0/514069239 conn(0x7fc81c0f0b70 msgr2=0x7fc81c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:55:44.264 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:44.262+0000 7fc82415e700 1 -- 172.21.15.137:0/514069239 shutdown_connections 2023-11-15T01:55:44.265 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:44.262+0000 7fc82415e700 1 -- 172.21.15.137:0/514069239 wait complete. 2023-11-15T01:55:44.265 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:55:45.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:44 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:55:43.602036+0000 mon.smithi055 (mon.0) 256 : audit [INF] from='mgr.14164 172.21.15.55:0/59463371' entity='mgr.smithi055.szwngw' 2023-11-15T01:55:45.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:44 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cephadm 2023-11-15T01:55:43.603571+0000 mgr.smithi055.szwngw (mgr.14164) 186 : cephadm [INF] Deploying daemon node-exporter.smithi055 on smithi055 2023-11-15T01:55:45.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:44 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:55:44.261608+0000 mon.smithi055 (mon.0) 257 : audit [DBG] from='client.? 172.21.15.137:0/514069239' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:55:45.652 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:55:45.652 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:55:46.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:45 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:55:44.747588+0000 mgr.smithi055.szwngw (mgr.14164) 187 : cluster [DBG] pgmap v175: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:55:46.795 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:46.795+0000 7fc7f3fff700 1 -- 172.21.15.137:0/786604372 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc7f40601e0 msgr2=0x7fc7f40f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:55:46.796 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:46.795+0000 7fc7f3fff700 1 --2- 172.21.15.137:0/786604372 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc7f40601e0 0x7fc7f40f8f20 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7fc7e4009fb0 tx=0x7fc7e4009b30).stop 2023-11-15T01:55:46.796 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:46.795+0000 7fc7f3fff700 1 -- 172.21.15.137:0/786604372 shutdown_connections 2023-11-15T01:55:46.796 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:46.795+0000 7fc7f3fff700 1 --2- 172.21.15.137:0/786604372 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc7f40601e0 0x7fc7f40f8f20 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:55:46.796 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:46.795+0000 7fc7f3fff700 1 -- 172.21.15.137:0/786604372 >> 172.21.15.137:0/786604372 conn(0x7fc7f40f09a0 msgr2=0x7fc7f40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:55:46.797 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:46.795+0000 7fc7f3fff700 1 -- 172.21.15.137:0/786604372 shutdown_connections 2023-11-15T01:55:46.797 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:46.795+0000 7fc7f3fff700 1 -- 172.21.15.137:0/786604372 wait complete. 2023-11-15T01:55:46.799 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:46.795+0000 7fc7f3fff700 1 Processor -- start 2023-11-15T01:55:46.799 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:46.795+0000 7fc7f3fff700 1 -- start start 2023-11-15T01:55:46.799 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:46.795+0000 7fc7f3fff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc7f40601e0 0x7fc7f4106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:55:46.800 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:46.795+0000 7fc7f3fff700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7f41069f0 con 0x7fc7f40601e0 2023-11-15T01:55:46.800 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:46.796+0000 7fc7f2ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc7f40601e0 0x7fc7f4106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:55:46.800 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:46.796+0000 7fc7f2ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc7f40601e0 0x7fc7f4106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:47120/0 (socket says 172.21.15.137:47120) 2023-11-15T01:55:46.800 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:46.796+0000 7fc7f2ffd700 1 -- 172.21.15.137:0/3507749680 learned_addr learned my addr 172.21.15.137:0/3507749680 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:55:46.800 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:46.796+0000 7fc7f2ffd700 1 -- 172.21.15.137:0/3507749680 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7e4005040 con 0x7fc7f40601e0 2023-11-15T01:55:46.801 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:46.796+0000 7fc7f2ffd700 1 --2- 172.21.15.137:0/3507749680 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc7f40601e0 0x7fc7f4106500 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7fc7e4013510 tx=0x7fc7e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:55:46.801 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:46.796+0000 7fc7db7fe700 1 -- 172.21.15.137:0/3507749680 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc7e4009320 con 0x7fc7f40601e0 2023-11-15T01:55:46.801 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:46.796+0000 7fc7f3fff700 1 -- 172.21.15.137:0/3507749680 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7f4106bf0 con 0x7fc7f40601e0 2023-11-15T01:55:46.801 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:46.796+0000 7fc7db7fe700 1 -- 172.21.15.137:0/3507749680 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc7e4013f00 con 0x7fc7f40601e0 2023-11-15T01:55:46.801 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:46.796+0000 7fc7db7fe700 1 -- 172.21.15.137:0/3507749680 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc7e4014d90 con 0x7fc7f40601e0 2023-11-15T01:55:46.801 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:46.796+0000 7fc7f3fff700 1 -- 172.21.15.137:0/3507749680 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7f4107080 con 0x7fc7f40601e0 2023-11-15T01:55:46.802 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:46.797+0000 7fc7db7fe700 1 -- 172.21.15.137:0/3507749680 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc7e4014420 con 0x7fc7f40601e0 2023-11-15T01:55:46.802 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:46.797+0000 7fc7f3fff700 1 -- 172.21.15.137:0/3507749680 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7f4045ad0 con 0x7fc7f40601e0 2023-11-15T01:55:46.802 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:46.797+0000 7fc7db7fe700 1 --2- 172.21.15.137:0/3507749680 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fc7dc02ff60 0x7fc7dc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:55:46.802 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:46.797+0000 7fc7db7fe700 1 -- 172.21.15.137:0/3507749680 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc7e4041050 con 0x7fc7f40601e0 2023-11-15T01:55:46.802 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:46.798+0000 7fc7f27fc700 1 --2- 172.21.15.137:0/3507749680 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fc7dc02ff60 0x7fc7dc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:55:46.803 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:46.798+0000 7fc7f27fc700 1 --2- 172.21.15.137:0/3507749680 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fc7dc02ff60 0x7fc7dc032420 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7fc7e80096a0 tx=0x7fc7e8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:55:46.803 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:46.802+0000 7fc7db7fe700 1 -- 172.21.15.137:0/3507749680 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc7e40095f0 con 0x7fc7f40601e0 2023-11-15T01:55:47.010 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:47.009+0000 7fc7f3fff700 1 -- 172.21.15.137:0/3507749680 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc7f4107ed0 con 0x7fc7f40601e0 2023-11-15T01:55:47.011 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:47.010+0000 7fc7db7fe700 1 -- 172.21.15.137:0/3507749680 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc7e401db70 con 0x7fc7f40601e0 2023-11-15T01:55:47.011 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:55:47.011 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:55:47.012 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:47.012+0000 7fc7f3fff700 1 -- 172.21.15.137:0/3507749680 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fc7dc02ff60 msgr2=0x7fc7dc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:55:47.012 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:47.012+0000 7fc7f3fff700 1 --2- 172.21.15.137:0/3507749680 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fc7dc02ff60 0x7fc7dc032420 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7fc7e80096a0 tx=0x7fc7e8006b40).stop 2023-11-15T01:55:47.013 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:47.012+0000 7fc7f3fff700 1 -- 172.21.15.137:0/3507749680 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc7f40601e0 msgr2=0x7fc7f4106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:55:47.013 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:47.012+0000 7fc7f3fff700 1 --2- 172.21.15.137:0/3507749680 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc7f40601e0 0x7fc7f4106500 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7fc7e4013510 tx=0x7fc7e400b040).stop 2023-11-15T01:55:47.013 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:47.012+0000 7fc7f3fff700 1 -- 172.21.15.137:0/3507749680 shutdown_connections 2023-11-15T01:55:47.013 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:47.012+0000 7fc7f3fff700 1 --2- 172.21.15.137:0/3507749680 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fc7dc02ff60 0x7fc7dc032420 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:55:47.013 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:47.012+0000 7fc7f3fff700 1 --2- 172.21.15.137:0/3507749680 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc7f40601e0 0x7fc7f4106500 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:55:47.014 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:47.012+0000 7fc7f3fff700 1 -- 172.21.15.137:0/3507749680 >> 172.21.15.137:0/3507749680 conn(0x7fc7f40f09a0 msgr2=0x7fc7f40f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:55:47.014 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:47.012+0000 7fc7f3fff700 1 -- 172.21.15.137:0/3507749680 shutdown_connections 2023-11-15T01:55:47.014 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:47.012+0000 7fc7f3fff700 1 -- 172.21.15.137:0/3507749680 wait complete. 2023-11-15T01:55:47.015 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:55:48.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:47 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:55:46.748006+0000 mgr.smithi055.szwngw (mgr.14164) 188 : cluster [DBG] pgmap v176: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:55:48.079 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:47 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:55:47.011212+0000 mon.smithi055 (mon.0) 258 : audit [DBG] from='client.? 172.21.15.137:0/3507749680' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:55:48.407 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:55:48.407 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:55:49.562 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:49.561+0000 7f04eb59e700 1 -- 172.21.15.137:0/2206085850 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f04ec0f78e0 msgr2=0x7f04ec0f7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:55:49.562 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:49.561+0000 7f04eb59e700 1 --2- 172.21.15.137:0/2206085850 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f04ec0f78e0 0x7f04ec0f7d00 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7f04dc009fb0 tx=0x7f04dc009b30).stop 2023-11-15T01:55:49.563 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:49.561+0000 7f04eb59e700 1 -- 172.21.15.137:0/2206085850 shutdown_connections 2023-11-15T01:55:49.563 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:49.561+0000 7f04eb59e700 1 --2- 172.21.15.137:0/2206085850 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f04ec0f78e0 0x7f04ec0f7d00 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:55:49.563 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:49.561+0000 7f04eb59e700 1 -- 172.21.15.137:0/2206085850 >> 172.21.15.137:0/2206085850 conn(0x7f04ec0f0120 msgr2=0x7f04ec0f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:55:49.563 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:49.561+0000 7f04eb59e700 1 -- 172.21.15.137:0/2206085850 shutdown_connections 2023-11-15T01:55:49.563 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:49.561+0000 7f04eb59e700 1 -- 172.21.15.137:0/2206085850 wait complete. 2023-11-15T01:55:49.565 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:49.562+0000 7f04eb59e700 1 Processor -- start 2023-11-15T01:55:49.566 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:49.562+0000 7f04eb59e700 1 -- start start 2023-11-15T01:55:49.566 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:49.562+0000 7f04eb59e700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f04ec0f78e0 0x7f04ec1064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:55:49.566 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:49.562+0000 7f04eb59e700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04ec1069d0 con 0x7f04ec0f78e0 2023-11-15T01:55:49.566 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:49.562+0000 7f04ea59c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f04ec0f78e0 0x7f04ec1064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:55:49.567 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:49.562+0000 7f04ea59c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f04ec0f78e0 0x7f04ec1064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:34164/0 (socket says 172.21.15.137:34164) 2023-11-15T01:55:49.567 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:49.562+0000 7f04ea59c700 1 -- 172.21.15.137:0/3201541167 learned_addr learned my addr 172.21.15.137:0/3201541167 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:55:49.567 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:49.563+0000 7f04ea59c700 1 -- 172.21.15.137:0/3201541167 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04dc005040 con 0x7f04ec0f78e0 2023-11-15T01:55:49.568 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:49.563+0000 7f04ea59c700 1 --2- 172.21.15.137:0/3201541167 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f04ec0f78e0 0x7f04ec1064e0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f04dc013510 tx=0x7f04dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:55:49.568 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:49.563+0000 7f04daffd700 1 -- 172.21.15.137:0/3201541167 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f04dc009320 con 0x7f04ec0f78e0 2023-11-15T01:55:49.568 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:49.563+0000 7f04eb59e700 1 -- 172.21.15.137:0/3201541167 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04ec106bd0 con 0x7f04ec0f78e0 2023-11-15T01:55:49.568 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:49.563+0000 7f04daffd700 1 -- 172.21.15.137:0/3201541167 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f04dc013f00 con 0x7f04ec0f78e0 2023-11-15T01:55:49.568 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:49.563+0000 7f04daffd700 1 -- 172.21.15.137:0/3201541167 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f04dc014dc0 con 0x7f04ec0f78e0 2023-11-15T01:55:49.569 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:49.563+0000 7f04eb59e700 1 -- 172.21.15.137:0/3201541167 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04ec107060 con 0x7f04ec0f78e0 2023-11-15T01:55:49.569 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:49.564+0000 7f04eb59e700 1 -- 172.21.15.137:0/3201541167 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04ec045ad0 con 0x7f04ec0f78e0 2023-11-15T01:55:49.569 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:49.564+0000 7f04daffd700 1 -- 172.21.15.137:0/3201541167 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f04dc0095f0 con 0x7f04ec0f78e0 2023-11-15T01:55:49.569 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:49.564+0000 7f04daffd700 1 --2- 172.21.15.137:0/3201541167 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f04d402ff60 0x7f04d4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:55:49.569 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:49.564+0000 7f04daffd700 1 -- 172.21.15.137:0/3201541167 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f04dc041030 con 0x7f04ec0f78e0 2023-11-15T01:55:49.570 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:49.564+0000 7f04e9d9b700 1 --2- 172.21.15.137:0/3201541167 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f04d402ff60 0x7f04d4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:55:49.570 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:49.565+0000 7f04e9d9b700 1 --2- 172.21.15.137:0/3201541167 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f04d402ff60 0x7f04d4032420 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f04e00096a0 tx=0x7f04e0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:55:49.570 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:49.569+0000 7f04daffd700 1 -- 172.21.15.137:0/3201541167 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f04dc014420 con 0x7f04ec0f78e0 2023-11-15T01:55:49.775 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:49.774+0000 7f04eb59e700 1 -- 172.21.15.137:0/3201541167 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f04ec107f10 con 0x7f04ec0f78e0 2023-11-15T01:55:49.776 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:49.775+0000 7f04daffd700 1 -- 172.21.15.137:0/3201541167 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f04dc010230 con 0x7f04ec0f78e0 2023-11-15T01:55:49.776 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:55:49.776 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:55:49.777 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:49.777+0000 7f04eb59e700 1 -- 172.21.15.137:0/3201541167 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f04d402ff60 msgr2=0x7f04d4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:55:49.778 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:49.777+0000 7f04eb59e700 1 --2- 172.21.15.137:0/3201541167 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f04d402ff60 0x7f04d4032420 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f04e00096a0 tx=0x7f04e0006b40).stop 2023-11-15T01:55:49.778 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:49.777+0000 7f04eb59e700 1 -- 172.21.15.137:0/3201541167 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f04ec0f78e0 msgr2=0x7f04ec1064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:55:49.778 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:49.777+0000 7f04eb59e700 1 --2- 172.21.15.137:0/3201541167 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f04ec0f78e0 0x7f04ec1064e0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f04dc013510 tx=0x7f04dc00b040).stop 2023-11-15T01:55:49.778 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:49.777+0000 7f04eb59e700 1 -- 172.21.15.137:0/3201541167 shutdown_connections 2023-11-15T01:55:49.779 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:49.777+0000 7f04eb59e700 1 --2- 172.21.15.137:0/3201541167 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f04d402ff60 0x7f04d4032420 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:55:49.779 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:49.777+0000 7f04eb59e700 1 --2- 172.21.15.137:0/3201541167 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f04ec0f78e0 0x7f04ec1064e0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:55:49.779 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:49.777+0000 7f04eb59e700 1 -- 172.21.15.137:0/3201541167 >> 172.21.15.137:0/3201541167 conn(0x7f04ec0f0120 msgr2=0x7f04ec0f0ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:55:49.779 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:49.777+0000 7f04eb59e700 1 -- 172.21.15.137:0/3201541167 shutdown_connections 2023-11-15T01:55:49.779 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:49.777+0000 7f04eb59e700 1 -- 172.21.15.137:0/3201541167 wait complete. 2023-11-15T01:55:49.780 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:55:50.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:49 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:55:48.748384+0000 mgr.smithi055.szwngw (mgr.14164) 189 : cluster [DBG] pgmap v177: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:55:50.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:49 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:55:49.776279+0000 mon.smithi055 (mon.0) 259 : audit [DBG] from='client.? 172.21.15.137:0/3201541167' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:55:51.164 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:55:51.165 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:55:51.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:51 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:55:50.748728+0000 mgr.smithi055.szwngw (mgr.14164) 190 : cluster [DBG] pgmap v178: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:55:52.313 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:52.310+0000 7fcabf22f700 1 -- 172.21.15.137:0/1139143937 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcab80f6510 msgr2=0x7fcab80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:55:52.313 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:52.310+0000 7fcabf22f700 1 --2- 172.21.15.137:0/1139143937 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcab80f6510 0x7fcab80f6930 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7fcaac009fb0 tx=0x7fcaac009b30).stop 2023-11-15T01:55:52.313 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:52.310+0000 7fcabf22f700 1 -- 172.21.15.137:0/1139143937 shutdown_connections 2023-11-15T01:55:52.314 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:52.310+0000 7fcabf22f700 1 --2- 172.21.15.137:0/1139143937 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcab80f6510 0x7fcab80f6930 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:55:52.314 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:52.310+0000 7fcabf22f700 1 -- 172.21.15.137:0/1139143937 >> 172.21.15.137:0/1139143937 conn(0x7fcab80f1c70 msgr2=0x7fcab80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:55:52.314 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:52.310+0000 7fcabf22f700 1 -- 172.21.15.137:0/1139143937 shutdown_connections 2023-11-15T01:55:52.314 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:52.310+0000 7fcabf22f700 1 -- 172.21.15.137:0/1139143937 wait complete. 2023-11-15T01:55:52.315 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:52.311+0000 7fcabf22f700 1 Processor -- start 2023-11-15T01:55:52.315 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:52.311+0000 7fcabf22f700 1 -- start start 2023-11-15T01:55:52.315 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:52.311+0000 7fcabf22f700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcab80f6510 0x7fcab81066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:55:52.315 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:52.311+0000 7fcabf22f700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcab8106bc0 con 0x7fcab80f6510 2023-11-15T01:55:52.315 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:52.311+0000 7fcabcfcb700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcab80f6510 0x7fcab81066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:55:52.316 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:52.311+0000 7fcabcfcb700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcab80f6510 0x7fcab81066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:34192/0 (socket says 172.21.15.137:34192) 2023-11-15T01:55:52.316 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:52.311+0000 7fcabcfcb700 1 -- 172.21.15.137:0/497668793 learned_addr learned my addr 172.21.15.137:0/497668793 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:55:52.316 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:52.312+0000 7fcabcfcb700 1 -- 172.21.15.137:0/497668793 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaac005040 con 0x7fcab80f6510 2023-11-15T01:55:52.316 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:52.312+0000 7fcabcfcb700 1 --2- 172.21.15.137:0/497668793 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcab80f6510 0x7fcab81066d0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7fcaac013510 tx=0x7fcaac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:55:52.316 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:52.312+0000 7fcab57fa700 1 -- 172.21.15.137:0/497668793 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcaac009320 con 0x7fcab80f6510 2023-11-15T01:55:52.317 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:52.312+0000 7fcabf22f700 1 -- 172.21.15.137:0/497668793 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcab8106dc0 con 0x7fcab80f6510 2023-11-15T01:55:52.317 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:52.312+0000 7fcab57fa700 1 -- 172.21.15.137:0/497668793 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcaac013f00 con 0x7fcab80f6510 2023-11-15T01:55:52.317 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:52.312+0000 7fcab57fa700 1 -- 172.21.15.137:0/497668793 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcaac014d90 con 0x7fcab80f6510 2023-11-15T01:55:52.317 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:52.312+0000 7fcabf22f700 1 -- 172.21.15.137:0/497668793 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcab8107250 con 0x7fcab80f6510 2023-11-15T01:55:52.318 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:52.313+0000 7fcab57fa700 1 -- 172.21.15.137:0/497668793 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcaac014420 con 0x7fcab80f6510 2023-11-15T01:55:52.318 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:52.313+0000 7fcabf22f700 1 -- 172.21.15.137:0/497668793 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcab8045ad0 con 0x7fcab80f6510 2023-11-15T01:55:52.319 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:52.313+0000 7fcab57fa700 1 --2- 172.21.15.137:0/497668793 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fcaa802ff60 0x7fcaa8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:55:52.319 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:52.313+0000 7fcab57fa700 1 -- 172.21.15.137:0/497668793 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcaac041070 con 0x7fcab80f6510 2023-11-15T01:55:52.319 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:52.313+0000 7fcab7fff700 1 --2- 172.21.15.137:0/497668793 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fcaa802ff60 0x7fcaa8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:55:52.319 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:52.314+0000 7fcab7fff700 1 --2- 172.21.15.137:0/497668793 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fcaa802ff60 0x7fcaa8032420 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7fcaa40096a0 tx=0x7fcaa4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:55:52.320 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:52.318+0000 7fcab57fa700 1 -- 172.21.15.137:0/497668793 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcaac0095f0 con 0x7fcab80f6510 2023-11-15T01:55:52.526 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:52.525+0000 7fcabf22f700 1 -- 172.21.15.137:0/497668793 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcab8108100 con 0x7fcab80f6510 2023-11-15T01:55:52.526 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:52.526+0000 7fcab57fa700 1 -- 172.21.15.137:0/497668793 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcaac01db70 con 0x7fcab80f6510 2023-11-15T01:55:52.527 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:55:52.527 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:55:52.528 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:52.527+0000 7fcabf22f700 1 -- 172.21.15.137:0/497668793 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fcaa802ff60 msgr2=0x7fcaa8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:55:52.529 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:52.527+0000 7fcabf22f700 1 --2- 172.21.15.137:0/497668793 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fcaa802ff60 0x7fcaa8032420 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7fcaa40096a0 tx=0x7fcaa4006b40).stop 2023-11-15T01:55:52.529 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:52.528+0000 7fcabf22f700 1 -- 172.21.15.137:0/497668793 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcab80f6510 msgr2=0x7fcab81066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:55:52.529 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:52.528+0000 7fcabf22f700 1 --2- 172.21.15.137:0/497668793 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcab80f6510 0x7fcab81066d0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7fcaac013510 tx=0x7fcaac00b040).stop 2023-11-15T01:55:52.529 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:52.528+0000 7fcabf22f700 1 -- 172.21.15.137:0/497668793 shutdown_connections 2023-11-15T01:55:52.529 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:52.528+0000 7fcabf22f700 1 --2- 172.21.15.137:0/497668793 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fcaa802ff60 0x7fcaa8032420 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:55:52.530 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:52.528+0000 7fcabf22f700 1 --2- 172.21.15.137:0/497668793 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcab80f6510 0x7fcab81066d0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:55:52.530 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:52.528+0000 7fcabf22f700 1 -- 172.21.15.137:0/497668793 >> 172.21.15.137:0/497668793 conn(0x7fcab80f1c70 msgr2=0x7fcab80f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:55:52.530 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:52.528+0000 7fcabf22f700 1 -- 172.21.15.137:0/497668793 shutdown_connections 2023-11-15T01:55:52.530 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:52.528+0000 7fcabf22f700 1 -- 172.21.15.137:0/497668793 wait complete. 2023-11-15T01:55:52.531 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:55:52.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:52 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:55:52.526909+0000 mon.smithi055 (mon.0) 260 : audit [DBG] from='client.? 172.21.15.137:0/497668793' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:55:53.915 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:55:53.916 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:55:54.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:53 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:55:52.749065+0000 mgr.smithi055.szwngw (mgr.14164) 191 : cluster [DBG] pgmap v179: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:55:55.030 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:55.029+0000 7f58b3c08700 1 -- 172.21.15.137:0/726313323 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f58ac0f6510 msgr2=0x7f58ac0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:55:55.031 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:55.029+0000 7f58b3c08700 1 --2- 172.21.15.137:0/726313323 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f58ac0f6510 0x7f58ac0f6930 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7f58a0009fb0 tx=0x7f58a0009b30).stop 2023-11-15T01:55:55.031 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:55.029+0000 7f58b3c08700 1 -- 172.21.15.137:0/726313323 shutdown_connections 2023-11-15T01:55:55.031 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:55.029+0000 7f58b3c08700 1 --2- 172.21.15.137:0/726313323 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f58ac0f6510 0x7f58ac0f6930 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:55:55.031 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:55.029+0000 7f58b3c08700 1 -- 172.21.15.137:0/726313323 >> 172.21.15.137:0/726313323 conn(0x7f58ac0f1c70 msgr2=0x7f58ac0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:55:55.031 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:55.030+0000 7f58b3c08700 1 -- 172.21.15.137:0/726313323 shutdown_connections 2023-11-15T01:55:55.032 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:55.030+0000 7f58b3c08700 1 -- 172.21.15.137:0/726313323 wait complete. 2023-11-15T01:55:55.033 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:55.030+0000 7f58b3c08700 1 Processor -- start 2023-11-15T01:55:55.034 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:55.030+0000 7f58b3c08700 1 -- start start 2023-11-15T01:55:55.034 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:55.030+0000 7f58b3c08700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f58ac0f6510 0x7f58ac106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:55:55.034 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:55.030+0000 7f58b3c08700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58ac106c20 con 0x7f58ac0f6510 2023-11-15T01:55:55.034 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:55.030+0000 7f58b19a4700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f58ac0f6510 0x7f58ac106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:55:55.034 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:55.030+0000 7f58b19a4700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f58ac0f6510 0x7f58ac106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:34208/0 (socket says 172.21.15.137:34208) 2023-11-15T01:55:55.035 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:55.030+0000 7f58b19a4700 1 -- 172.21.15.137:0/2993074938 learned_addr learned my addr 172.21.15.137:0/2993074938 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:55:55.035 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:55.031+0000 7f58b19a4700 1 -- 172.21.15.137:0/2993074938 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58a0005040 con 0x7f58ac0f6510 2023-11-15T01:55:55.035 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:55.031+0000 7f58b19a4700 1 --2- 172.21.15.137:0/2993074938 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f58ac0f6510 0x7f58ac106730 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f58a0014000 tx=0x7f58a000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:55:55.035 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:55.031+0000 7f589e7fc700 1 -- 172.21.15.137:0/2993074938 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f58a00095f0 con 0x7f58ac0f6510 2023-11-15T01:55:55.035 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:55.031+0000 7f58b3c08700 1 -- 172.21.15.137:0/2993074938 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58ac106e20 con 0x7f58ac0f6510 2023-11-15T01:55:55.036 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:55.031+0000 7f589e7fc700 1 -- 172.21.15.137:0/2993074938 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f58a0007e70 con 0x7f58ac0f6510 2023-11-15T01:55:55.036 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:55.031+0000 7f589e7fc700 1 -- 172.21.15.137:0/2993074938 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f58a0014ea0 con 0x7f58ac0f6510 2023-11-15T01:55:55.036 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:55.031+0000 7f58b3c08700 1 -- 172.21.15.137:0/2993074938 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58ac1072b0 con 0x7f58ac0f6510 2023-11-15T01:55:55.036 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:55.032+0000 7f589e7fc700 1 -- 172.21.15.137:0/2993074938 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f58a0025070 con 0x7f58ac0f6510 2023-11-15T01:55:55.036 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:55.032+0000 7f58b3c08700 1 -- 172.21.15.137:0/2993074938 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58ac045ad0 con 0x7f58ac0f6510 2023-11-15T01:55:55.037 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:55.032+0000 7f589e7fc700 1 --2- 172.21.15.137:0/2993074938 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f589802ff60 0x7f5898032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:55:55.037 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:55.032+0000 7f589e7fc700 1 -- 172.21.15.137:0/2993074938 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f58a0040970 con 0x7f58ac0f6510 2023-11-15T01:55:55.037 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:55.032+0000 7f58b11a3700 1 --2- 172.21.15.137:0/2993074938 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f589802ff60 0x7f5898032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:55:55.037 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:55.033+0000 7f58b11a3700 1 --2- 172.21.15.137:0/2993074938 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f589802ff60 0x7f5898032420 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f58a80096a0 tx=0x7f58a8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:55:55.038 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:55.037+0000 7f589e7fc700 1 -- 172.21.15.137:0/2993074938 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f58a00079c0 con 0x7f58ac0f6510 2023-11-15T01:55:55.246 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:55.246+0000 7f58b3c08700 1 -- 172.21.15.137:0/2993074938 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f58ac0fbf60 con 0x7f58ac0f6510 2023-11-15T01:55:55.247 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:55.246+0000 7f589e7fc700 1 -- 172.21.15.137:0/2993074938 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f58a000f300 con 0x7f58ac0f6510 2023-11-15T01:55:55.248 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:55:55.248 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:55:55.249 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:55.248+0000 7f58b3c08700 1 -- 172.21.15.137:0/2993074938 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f589802ff60 msgr2=0x7f5898032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:55:55.249 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:55.248+0000 7f58b3c08700 1 --2- 172.21.15.137:0/2993074938 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f589802ff60 0x7f5898032420 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f58a80096a0 tx=0x7f58a8006b40).stop 2023-11-15T01:55:55.249 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:55.248+0000 7f58b3c08700 1 -- 172.21.15.137:0/2993074938 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f58ac0f6510 msgr2=0x7f58ac106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:55:55.250 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:55.248+0000 7f58b3c08700 1 --2- 172.21.15.137:0/2993074938 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f58ac0f6510 0x7f58ac106730 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f58a0014000 tx=0x7f58a000b040).stop 2023-11-15T01:55:55.250 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:55.248+0000 7f58b3c08700 1 -- 172.21.15.137:0/2993074938 shutdown_connections 2023-11-15T01:55:55.250 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:55.248+0000 7f58b3c08700 1 --2- 172.21.15.137:0/2993074938 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f589802ff60 0x7f5898032420 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:55:55.250 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:55.248+0000 7f58b3c08700 1 --2- 172.21.15.137:0/2993074938 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f58ac0f6510 0x7f58ac106730 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:55:55.250 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:55.248+0000 7f58b3c08700 1 -- 172.21.15.137:0/2993074938 >> 172.21.15.137:0/2993074938 conn(0x7f58ac0f1c70 msgr2=0x7f58ac0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:55:55.251 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:55.248+0000 7f58b3c08700 1 -- 172.21.15.137:0/2993074938 shutdown_connections 2023-11-15T01:55:55.251 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:55.248+0000 7f58b3c08700 1 -- 172.21.15.137:0/2993074938 wait complete. 2023-11-15T01:55:55.251 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:55:56.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:55 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:55:54.749420+0000 mgr.smithi055.szwngw (mgr.14164) 192 : cluster [DBG] pgmap v180: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:55:56.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:55 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:55:55.247711+0000 mon.smithi055 (mon.0) 261 : audit [DBG] from='client.? 172.21.15.137:0/2993074938' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:55:56.635 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:55:56.636 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:55:57.776 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:57.775+0000 7f7d04b03700 1 -- 172.21.15.137:0/1720707760 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7d000f6510 msgr2=0x7f7d000f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:55:57.777 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:57.775+0000 7f7d04b03700 1 --2- 172.21.15.137:0/1720707760 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7d000f6510 0x7f7d000f6930 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7f7cf4009fb0 tx=0x7f7cf4009b30).stop 2023-11-15T01:55:57.777 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:57.775+0000 7f7d04b03700 1 -- 172.21.15.137:0/1720707760 shutdown_connections 2023-11-15T01:55:57.777 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:57.775+0000 7f7d04b03700 1 --2- 172.21.15.137:0/1720707760 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7d000f6510 0x7f7d000f6930 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:55:57.777 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:57.775+0000 7f7d04b03700 1 -- 172.21.15.137:0/1720707760 >> 172.21.15.137:0/1720707760 conn(0x7f7d000f1c70 msgr2=0x7f7d000f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:55:57.778 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:57.775+0000 7f7d04b03700 1 -- 172.21.15.137:0/1720707760 shutdown_connections 2023-11-15T01:55:57.778 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:57.775+0000 7f7d04b03700 1 -- 172.21.15.137:0/1720707760 wait complete. 2023-11-15T01:55:57.778 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:57.776+0000 7f7d04b03700 1 Processor -- start 2023-11-15T01:55:57.778 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:57.776+0000 7f7d04b03700 1 -- start start 2023-11-15T01:55:57.778 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:57.776+0000 7f7d04b03700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7d000f6510 0x7f7d00106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:55:57.779 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:57.776+0000 7f7d04b03700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d00106c20 con 0x7f7d000f6510 2023-11-15T01:55:57.780 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:57.776+0000 7f7cfe59c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7d000f6510 0x7f7d00106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:55:57.780 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:57.776+0000 7f7cfe59c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7d000f6510 0x7f7d00106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:34230/0 (socket says 172.21.15.137:34230) 2023-11-15T01:55:57.781 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:57.776+0000 7f7cfe59c700 1 -- 172.21.15.137:0/234197378 learned_addr learned my addr 172.21.15.137:0/234197378 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:55:57.781 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:57.777+0000 7f7cfe59c700 1 -- 172.21.15.137:0/234197378 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cf4005040 con 0x7f7d000f6510 2023-11-15T01:55:57.781 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:57.777+0000 7f7cfe59c700 1 --2- 172.21.15.137:0/234197378 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7d000f6510 0x7f7d00106730 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7f7cf4013510 tx=0x7f7cf400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:55:57.781 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:57.777+0000 7f7ceeffd700 1 -- 172.21.15.137:0/234197378 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7cf4009320 con 0x7f7d000f6510 2023-11-15T01:55:57.781 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:57.777+0000 7f7ceeffd700 1 -- 172.21.15.137:0/234197378 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7cf4013f00 con 0x7f7d000f6510 2023-11-15T01:55:57.782 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:57.777+0000 7f7d04b03700 1 -- 172.21.15.137:0/234197378 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d00106e20 con 0x7f7d000f6510 2023-11-15T01:55:57.782 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:57.777+0000 7f7ceeffd700 1 -- 172.21.15.137:0/234197378 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7cf4014d90 con 0x7f7d000f6510 2023-11-15T01:55:57.782 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:57.777+0000 7f7d04b03700 1 -- 172.21.15.137:0/234197378 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d001072b0 con 0x7f7d000f6510 2023-11-15T01:55:57.782 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:57.778+0000 7f7ceeffd700 1 -- 172.21.15.137:0/234197378 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7cf40095f0 con 0x7f7d000f6510 2023-11-15T01:55:57.782 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:57.778+0000 7f7d04b03700 1 -- 172.21.15.137:0/234197378 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d00045ad0 con 0x7f7d000f6510 2023-11-15T01:55:57.783 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:57.778+0000 7f7ceeffd700 1 --2- 172.21.15.137:0/234197378 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f7ce802ff60 0x7f7ce8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:55:57.783 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:57.778+0000 7f7ceeffd700 1 -- 172.21.15.137:0/234197378 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7cf4041000 con 0x7f7d000f6510 2023-11-15T01:55:57.783 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:57.778+0000 7f7cfdd9b700 1 --2- 172.21.15.137:0/234197378 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f7ce802ff60 0x7f7ce8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:55:57.783 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:57.779+0000 7f7cfdd9b700 1 --2- 172.21.15.137:0/234197378 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f7ce802ff60 0x7f7ce8032420 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f7cf00096a0 tx=0x7f7cf0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:55:57.784 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:57.783+0000 7f7ceeffd700 1 -- 172.21.15.137:0/234197378 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7cf4014420 con 0x7f7d000f6510 2023-11-15T01:55:57.990 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:57.988+0000 7f7d04b03700 1 -- 172.21.15.137:0/234197378 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7d000fbf60 con 0x7f7d000f6510 2023-11-15T01:55:57.990 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:57.989+0000 7f7ceeffd700 1 -- 172.21.15.137:0/234197378 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7cf401d7f0 con 0x7f7d000f6510 2023-11-15T01:55:57.990 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:55:57.991 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:55:57.992 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:57.990+0000 7f7d04b03700 1 -- 172.21.15.137:0/234197378 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f7ce802ff60 msgr2=0x7f7ce8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:55:57.992 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:57.990+0000 7f7d04b03700 1 --2- 172.21.15.137:0/234197378 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f7ce802ff60 0x7f7ce8032420 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f7cf00096a0 tx=0x7f7cf0006b40).stop 2023-11-15T01:55:57.992 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:57.990+0000 7f7d04b03700 1 -- 172.21.15.137:0/234197378 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7d000f6510 msgr2=0x7f7d00106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:55:57.992 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:57.990+0000 7f7d04b03700 1 --2- 172.21.15.137:0/234197378 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7d000f6510 0x7f7d00106730 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7f7cf4013510 tx=0x7f7cf400b040).stop 2023-11-15T01:55:57.993 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:57.991+0000 7f7d04b03700 1 -- 172.21.15.137:0/234197378 shutdown_connections 2023-11-15T01:55:57.993 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:57.991+0000 7f7d04b03700 1 --2- 172.21.15.137:0/234197378 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f7ce802ff60 0x7f7ce8032420 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:55:57.993 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:57.991+0000 7f7d04b03700 1 --2- 172.21.15.137:0/234197378 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7d000f6510 0x7f7d00106730 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:55:57.993 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:57.991+0000 7f7d04b03700 1 -- 172.21.15.137:0/234197378 >> 172.21.15.137:0/234197378 conn(0x7f7d000f1c70 msgr2=0x7f7d000f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:55:57.994 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:57.991+0000 7f7d04b03700 1 -- 172.21.15.137:0/234197378 shutdown_connections 2023-11-15T01:55:57.994 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:55:57.991+0000 7f7d04b03700 1 -- 172.21.15.137:0/234197378 wait complete. 2023-11-15T01:55:57.994 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:55:58.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:57 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:55:56.749730+0000 mgr.smithi055.szwngw (mgr.14164) 193 : cluster [DBG] pgmap v181: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:55:59.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:58 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:55:57.990181+0000 mon.smithi055 (mon.0) 262 : audit [DBG] from='client.? 172.21.15.137:0/234197378' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:55:59.379 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:55:59.380 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:56:00.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:55:59 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:55:58.749940+0000 mgr.smithi055.szwngw (mgr.14164) 194 : cluster [DBG] pgmap v182: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:56:00.501 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:00.499+0000 7fc2d0b03700 1 -- 172.21.15.137:0/3010443835 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc2cc0f51a0 msgr2=0x7fc2cc0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:56:00.501 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:00.499+0000 7fc2d0b03700 1 --2- 172.21.15.137:0/3010443835 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc2cc0f51a0 0x7fc2cc0f55c0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7fc2bc009fb0 tx=0x7fc2bc009b30).stop 2023-11-15T01:56:00.502 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:00.499+0000 7fc2d0b03700 1 -- 172.21.15.137:0/3010443835 shutdown_connections 2023-11-15T01:56:00.502 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:00.499+0000 7fc2d0b03700 1 --2- 172.21.15.137:0/3010443835 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc2cc0f51a0 0x7fc2cc0f55c0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:56:00.502 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:00.499+0000 7fc2d0b03700 1 -- 172.21.15.137:0/3010443835 >> 172.21.15.137:0/3010443835 conn(0x7fc2cc0f09a0 msgr2=0x7fc2cc0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:56:00.502 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:00.499+0000 7fc2d0b03700 1 -- 172.21.15.137:0/3010443835 shutdown_connections 2023-11-15T01:56:00.502 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:00.499+0000 7fc2d0b03700 1 -- 172.21.15.137:0/3010443835 wait complete. 2023-11-15T01:56:00.504 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:00.500+0000 7fc2d0b03700 1 Processor -- start 2023-11-15T01:56:00.505 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:00.500+0000 7fc2d0b03700 1 -- start start 2023-11-15T01:56:00.505 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:00.500+0000 7fc2d0b03700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc2cc0f51a0 0x7fc2cc1091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:56:00.505 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:00.500+0000 7fc2d0b03700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2cc060220 con 0x7fc2cc0f51a0 2023-11-15T01:56:00.505 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:00.500+0000 7fc2cb7fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc2cc0f51a0 0x7fc2cc1091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:56:00.505 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:00.500+0000 7fc2cb7fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc2cc0f51a0 0x7fc2cc1091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:44592/0 (socket says 172.21.15.137:44592) 2023-11-15T01:56:00.506 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:00.500+0000 7fc2cb7fe700 1 -- 172.21.15.137:0/1205053186 learned_addr learned my addr 172.21.15.137:0/1205053186 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:56:00.506 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:00.501+0000 7fc2cb7fe700 1 -- 172.21.15.137:0/1205053186 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2bc005040 con 0x7fc2cc0f51a0 2023-11-15T01:56:00.506 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:00.501+0000 7fc2cb7fe700 1 --2- 172.21.15.137:0/1205053186 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc2cc0f51a0 0x7fc2cc1091d0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7fc2bc013510 tx=0x7fc2bc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:56:00.506 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:00.501+0000 7fc2c8ff9700 1 -- 172.21.15.137:0/1205053186 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc2bc009320 con 0x7fc2cc0f51a0 2023-11-15T01:56:00.506 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:00.501+0000 7fc2d0b03700 1 -- 172.21.15.137:0/1205053186 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2cc060420 con 0x7fc2cc0f51a0 2023-11-15T01:56:00.507 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:00.501+0000 7fc2c8ff9700 1 -- 172.21.15.137:0/1205053186 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc2bc013f00 con 0x7fc2cc0f51a0 2023-11-15T01:56:00.507 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:00.501+0000 7fc2c8ff9700 1 -- 172.21.15.137:0/1205053186 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc2bc014dc0 con 0x7fc2cc0f51a0 2023-11-15T01:56:00.507 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:00.501+0000 7fc2d0b03700 1 -- 172.21.15.137:0/1205053186 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2cc109880 con 0x7fc2cc0f51a0 2023-11-15T01:56:00.507 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:00.502+0000 7fc2c8ff9700 1 -- 172.21.15.137:0/1205053186 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc2bc01d3d0 con 0x7fc2cc0f51a0 2023-11-15T01:56:00.507 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:00.502+0000 7fc2c8ff9700 1 --2- 172.21.15.137:0/1205053186 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fc2b802ff60 0x7fc2b8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:56:00.507 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:00.502+0000 7fc2c8ff9700 1 -- 172.21.15.137:0/1205053186 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc2bc041410 con 0x7fc2cc0f51a0 2023-11-15T01:56:00.508 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:00.502+0000 7fc2d0b03700 1 -- 172.21.15.137:0/1205053186 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2cc059100 con 0x7fc2cc0f51a0 2023-11-15T01:56:00.508 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:00.502+0000 7fc2c3fff700 1 --2- 172.21.15.137:0/1205053186 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fc2b802ff60 0x7fc2b8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:56:00.508 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:00.503+0000 7fc2c3fff700 1 --2- 172.21.15.137:0/1205053186 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fc2b802ff60 0x7fc2b8032420 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7fc2b40096a0 tx=0x7fc2b4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:56:00.508 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:00.507+0000 7fc2c8ff9700 1 -- 172.21.15.137:0/1205053186 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc2bc026490 con 0x7fc2cc0f51a0 2023-11-15T01:56:00.715 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:00.714+0000 7fc2d0b03700 1 -- 172.21.15.137:0/1205053186 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc2cc109f80 con 0x7fc2cc0f51a0 2023-11-15T01:56:00.716 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:00.714+0000 7fc2c8ff9700 1 -- 172.21.15.137:0/1205053186 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc2bc0126f0 con 0x7fc2cc0f51a0 2023-11-15T01:56:00.716 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:56:00.717 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:56:00.718 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:00.716+0000 7fc2d0b03700 1 -- 172.21.15.137:0/1205053186 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fc2b802ff60 msgr2=0x7fc2b8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:56:00.718 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:00.716+0000 7fc2d0b03700 1 --2- 172.21.15.137:0/1205053186 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fc2b802ff60 0x7fc2b8032420 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7fc2b40096a0 tx=0x7fc2b4006b40).stop 2023-11-15T01:56:00.718 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:00.716+0000 7fc2d0b03700 1 -- 172.21.15.137:0/1205053186 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc2cc0f51a0 msgr2=0x7fc2cc1091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:56:00.718 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:00.716+0000 7fc2d0b03700 1 --2- 172.21.15.137:0/1205053186 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc2cc0f51a0 0x7fc2cc1091d0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7fc2bc013510 tx=0x7fc2bc00b040).stop 2023-11-15T01:56:00.719 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:00.716+0000 7fc2d0b03700 1 -- 172.21.15.137:0/1205053186 shutdown_connections 2023-11-15T01:56:00.719 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:00.716+0000 7fc2d0b03700 1 --2- 172.21.15.137:0/1205053186 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fc2b802ff60 0x7fc2b8032420 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:56:00.719 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:00.716+0000 7fc2d0b03700 1 --2- 172.21.15.137:0/1205053186 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc2cc0f51a0 0x7fc2cc1091d0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:56:00.719 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:00.716+0000 7fc2d0b03700 1 -- 172.21.15.137:0/1205053186 >> 172.21.15.137:0/1205053186 conn(0x7fc2cc0f09a0 msgr2=0x7fc2cc0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:56:00.719 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:00.716+0000 7fc2d0b03700 1 -- 172.21.15.137:0/1205053186 shutdown_connections 2023-11-15T01:56:00.720 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:00.716+0000 7fc2d0b03700 1 -- 172.21.15.137:0/1205053186 wait complete. 2023-11-15T01:56:00.720 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:56:01.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:56:01 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:56:00.716552+0000 mon.smithi055 (mon.0) 263 : audit [DBG] from='client.? 172.21.15.137:0/1205053186' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:56:01.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:56:01 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:56:00.750208+0000 mgr.smithi055.szwngw (mgr.14164) 195 : cluster [DBG] pgmap v183: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:56:02.132 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:56:02.132 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:56:03.260 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:03.257+0000 7f5f1359e700 1 -- 172.21.15.137:0/2543392134 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5f140f51a0 msgr2=0x7f5f140f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:56:03.260 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:03.257+0000 7f5f1359e700 1 --2- 172.21.15.137:0/2543392134 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5f140f51a0 0x7f5f140f55c0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7f5f08009fb0 tx=0x7f5f08009b30).stop 2023-11-15T01:56:03.260 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:03.257+0000 7f5f1359e700 1 -- 172.21.15.137:0/2543392134 shutdown_connections 2023-11-15T01:56:03.260 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:03.257+0000 7f5f1359e700 1 --2- 172.21.15.137:0/2543392134 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5f140f51a0 0x7f5f140f55c0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:56:03.261 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:03.257+0000 7f5f1359e700 1 -- 172.21.15.137:0/2543392134 >> 172.21.15.137:0/2543392134 conn(0x7f5f140f09a0 msgr2=0x7f5f140f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:56:03.261 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:03.257+0000 7f5f1359e700 1 -- 172.21.15.137:0/2543392134 shutdown_connections 2023-11-15T01:56:03.261 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:03.257+0000 7f5f1359e700 1 -- 172.21.15.137:0/2543392134 wait complete. 2023-11-15T01:56:03.261 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:03.258+0000 7f5f1359e700 1 Processor -- start 2023-11-15T01:56:03.261 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:03.258+0000 7f5f1359e700 1 -- start start 2023-11-15T01:56:03.262 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:03.258+0000 7f5f1359e700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5f140f51a0 0x7f5f141091e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:56:03.262 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:03.258+0000 7f5f1359e700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f14060240 con 0x7f5f140f51a0 2023-11-15T01:56:03.262 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:03.258+0000 7f5f1259c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5f140f51a0 0x7f5f141091e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:56:03.262 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:03.258+0000 7f5f1259c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5f140f51a0 0x7f5f141091e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:44608/0 (socket says 172.21.15.137:44608) 2023-11-15T01:56:03.262 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:03.258+0000 7f5f1259c700 1 -- 172.21.15.137:0/646196339 learned_addr learned my addr 172.21.15.137:0/646196339 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:56:03.264 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:03.259+0000 7f5f1259c700 1 -- 172.21.15.137:0/646196339 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f08005040 con 0x7f5f140f51a0 2023-11-15T01:56:03.264 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:03.259+0000 7f5f1259c700 1 --2- 172.21.15.137:0/646196339 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5f140f51a0 0x7f5f141091e0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7f5f08013510 tx=0x7f5f0800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:56:03.264 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:03.259+0000 7f5f02ffd700 1 -- 172.21.15.137:0/646196339 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5f08009320 con 0x7f5f140f51a0 2023-11-15T01:56:03.265 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:03.259+0000 7f5f1359e700 1 -- 172.21.15.137:0/646196339 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f14060440 con 0x7f5f140f51a0 2023-11-15T01:56:03.265 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:03.259+0000 7f5f02ffd700 1 -- 172.21.15.137:0/646196339 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5f08013f00 con 0x7f5f140f51a0 2023-11-15T01:56:03.265 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:03.259+0000 7f5f02ffd700 1 -- 172.21.15.137:0/646196339 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5f08014dc0 con 0x7f5f140f51a0 2023-11-15T01:56:03.265 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:03.259+0000 7f5f1359e700 1 -- 172.21.15.137:0/646196339 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f14109890 con 0x7f5f140f51a0 2023-11-15T01:56:03.265 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:03.260+0000 7f5f02ffd700 1 -- 172.21.15.137:0/646196339 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5f080095f0 con 0x7f5f140f51a0 2023-11-15T01:56:03.266 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:03.260+0000 7f5f1359e700 1 -- 172.21.15.137:0/646196339 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f14045ad0 con 0x7f5f140f51a0 2023-11-15T01:56:03.266 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:03.260+0000 7f5f02ffd700 1 --2- 172.21.15.137:0/646196339 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f5efc02ff60 0x7f5efc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:56:03.266 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:03.260+0000 7f5f02ffd700 1 -- 172.21.15.137:0/646196339 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5f080410c0 con 0x7f5f140f51a0 2023-11-15T01:56:03.266 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:03.260+0000 7f5f11d9b700 1 --2- 172.21.15.137:0/646196339 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f5efc02ff60 0x7f5efc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:56:03.266 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:03.261+0000 7f5f11d9b700 1 --2- 172.21.15.137:0/646196339 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f5efc02ff60 0x7f5efc032420 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f5f040096a0 tx=0x7f5f04006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:56:03.267 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:03.265+0000 7f5f02ffd700 1 -- 172.21.15.137:0/646196339 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5f0801c790 con 0x7f5f140f51a0 2023-11-15T01:56:03.471 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:03.470+0000 7f5f1359e700 1 -- 172.21.15.137:0/646196339 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5f1410a0b0 con 0x7f5f140f51a0 2023-11-15T01:56:03.472 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:03.470+0000 7f5f02ffd700 1 -- 172.21.15.137:0/646196339 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5f08010230 con 0x7f5f140f51a0 2023-11-15T01:56:03.472 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:56:03.472 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:56:03.474 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:03.472+0000 7f5f1359e700 1 -- 172.21.15.137:0/646196339 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f5efc02ff60 msgr2=0x7f5efc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:56:03.474 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:03.472+0000 7f5f1359e700 1 --2- 172.21.15.137:0/646196339 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f5efc02ff60 0x7f5efc032420 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f5f040096a0 tx=0x7f5f04006b40).stop 2023-11-15T01:56:03.474 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:03.472+0000 7f5f1359e700 1 -- 172.21.15.137:0/646196339 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5f140f51a0 msgr2=0x7f5f141091e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:56:03.474 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:03.472+0000 7f5f1359e700 1 --2- 172.21.15.137:0/646196339 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5f140f51a0 0x7f5f141091e0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7f5f08013510 tx=0x7f5f0800b040).stop 2023-11-15T01:56:03.475 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:03.472+0000 7f5f1359e700 1 -- 172.21.15.137:0/646196339 shutdown_connections 2023-11-15T01:56:03.475 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:03.472+0000 7f5f1359e700 1 --2- 172.21.15.137:0/646196339 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f5efc02ff60 0x7f5efc032420 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:56:03.475 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:03.472+0000 7f5f1359e700 1 --2- 172.21.15.137:0/646196339 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5f140f51a0 0x7f5f141091e0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:56:03.475 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:03.472+0000 7f5f1359e700 1 -- 172.21.15.137:0/646196339 >> 172.21.15.137:0/646196339 conn(0x7f5f140f09a0 msgr2=0x7f5f140f13e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:56:03.475 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:03.472+0000 7f5f1359e700 1 -- 172.21.15.137:0/646196339 shutdown_connections 2023-11-15T01:56:03.476 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:03.472+0000 7f5f1359e700 1 -- 172.21.15.137:0/646196339 wait complete. 2023-11-15T01:56:03.476 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:56:04.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:56:03 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:56:02.750551+0000 mgr.smithi055.szwngw (mgr.14164) 196 : cluster [DBG] pgmap v184: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:56:04.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:56:03 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:56:03.472383+0000 mon.smithi055 (mon.0) 264 : audit [DBG] from='client.? 172.21.15.137:0/646196339' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:56:04.853 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:56:04.854 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:56:06.013 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:06.010+0000 7f67c0b9d700 1 -- 172.21.15.137:0/1258786381 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f67bc0f51a0 msgr2=0x7f67bc0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:56:06.013 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:06.010+0000 7f67c0b9d700 1 --2- 172.21.15.137:0/1258786381 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f67bc0f51a0 0x7f67bc0f55c0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7f67ac009fb0 tx=0x7f67ac009b30).stop 2023-11-15T01:56:06.014 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:06.010+0000 7f67c0b9d700 1 -- 172.21.15.137:0/1258786381 shutdown_connections 2023-11-15T01:56:06.014 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:06.010+0000 7f67c0b9d700 1 --2- 172.21.15.137:0/1258786381 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f67bc0f51a0 0x7f67bc0f55c0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:56:06.014 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:06.010+0000 7f67c0b9d700 1 -- 172.21.15.137:0/1258786381 >> 172.21.15.137:0/1258786381 conn(0x7f67bc0f09a0 msgr2=0x7f67bc0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:56:06.014 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:06.010+0000 7f67c0b9d700 1 -- 172.21.15.137:0/1258786381 shutdown_connections 2023-11-15T01:56:06.014 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:06.010+0000 7f67c0b9d700 1 -- 172.21.15.137:0/1258786381 wait complete. 2023-11-15T01:56:06.015 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:06.010+0000 7f67c0b9d700 1 Processor -- start 2023-11-15T01:56:06.015 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:06.011+0000 7f67c0b9d700 1 -- start start 2023-11-15T01:56:06.015 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:06.011+0000 7f67c0b9d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f67bc0f51a0 0x7f67bc1064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:56:06.015 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:06.011+0000 7f67c0b9d700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67bc1069d0 con 0x7f67bc0f51a0 2023-11-15T01:56:06.015 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:06.011+0000 7f67bb7fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f67bc0f51a0 0x7f67bc1064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:56:06.016 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:06.011+0000 7f67bb7fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f67bc0f51a0 0x7f67bc1064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:44618/0 (socket says 172.21.15.137:44618) 2023-11-15T01:56:06.016 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:06.011+0000 7f67bb7fe700 1 -- 172.21.15.137:0/932376478 learned_addr learned my addr 172.21.15.137:0/932376478 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:56:06.016 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:06.011+0000 7f67bb7fe700 1 -- 172.21.15.137:0/932376478 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67ac005040 con 0x7f67bc0f51a0 2023-11-15T01:56:06.016 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:06.012+0000 7f67bb7fe700 1 --2- 172.21.15.137:0/932376478 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f67bc0f51a0 0x7f67bc1064e0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f67ac013510 tx=0x7f67ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:56:06.017 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:06.012+0000 7f67a3fff700 1 -- 172.21.15.137:0/932376478 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f67ac009320 con 0x7f67bc0f51a0 2023-11-15T01:56:06.018 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:06.012+0000 7f67c0b9d700 1 -- 172.21.15.137:0/932376478 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67bc106bd0 con 0x7f67bc0f51a0 2023-11-15T01:56:06.018 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:06.012+0000 7f67a3fff700 1 -- 172.21.15.137:0/932376478 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f67ac013f00 con 0x7f67bc0f51a0 2023-11-15T01:56:06.018 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:06.012+0000 7f67a3fff700 1 -- 172.21.15.137:0/932376478 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f67ac014dc0 con 0x7f67bc0f51a0 2023-11-15T01:56:06.018 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:06.012+0000 7f67c0b9d700 1 -- 172.21.15.137:0/932376478 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67bc107060 con 0x7f67bc0f51a0 2023-11-15T01:56:06.019 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:06.013+0000 7f67a3fff700 1 -- 172.21.15.137:0/932376478 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f67ac01d3d0 con 0x7f67bc0f51a0 2023-11-15T01:56:06.019 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:06.013+0000 7f67a3fff700 1 --2- 172.21.15.137:0/932376478 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f67a402ff60 0x7f67a4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:56:06.019 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:06.013+0000 7f67a3fff700 1 -- 172.21.15.137:0/932376478 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f67ac041060 con 0x7f67bc0f51a0 2023-11-15T01:56:06.019 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:06.013+0000 7f67c0b9d700 1 -- 172.21.15.137:0/932376478 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67bc045ad0 con 0x7f67bc0f51a0 2023-11-15T01:56:06.019 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:06.013+0000 7f67baffd700 1 --2- 172.21.15.137:0/932376478 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f67a402ff60 0x7f67a4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:56:06.019 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:06.013+0000 7f67baffd700 1 --2- 172.21.15.137:0/932376478 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f67a402ff60 0x7f67a4032420 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f67b00096a0 tx=0x7f67b0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:56:06.020 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:06.018+0000 7f67a3fff700 1 -- 172.21.15.137:0/932376478 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f67ac012920 con 0x7f67bc0f51a0 2023-11-15T01:56:06.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:56:05 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:56:04.750957+0000 mgr.smithi055.szwngw (mgr.14164) 197 : cluster [DBG] pgmap v185: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:56:06.223 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:06.221+0000 7f67c0b9d700 1 -- 172.21.15.137:0/932376478 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f67bc0facd0 con 0x7f67bc0f51a0 2023-11-15T01:56:06.224 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:06.222+0000 7f67a3fff700 1 -- 172.21.15.137:0/932376478 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f67ac012330 con 0x7f67bc0f51a0 2023-11-15T01:56:06.224 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:56:06.224 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:56:06.225 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:06.223+0000 7f67c0b9d700 1 -- 172.21.15.137:0/932376478 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f67a402ff60 msgr2=0x7f67a4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:56:06.226 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:06.223+0000 7f67c0b9d700 1 --2- 172.21.15.137:0/932376478 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f67a402ff60 0x7f67a4032420 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f67b00096a0 tx=0x7f67b0006b40).stop 2023-11-15T01:56:06.226 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:06.223+0000 7f67c0b9d700 1 -- 172.21.15.137:0/932376478 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f67bc0f51a0 msgr2=0x7f67bc1064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:56:06.226 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:06.223+0000 7f67c0b9d700 1 --2- 172.21.15.137:0/932376478 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f67bc0f51a0 0x7f67bc1064e0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f67ac013510 tx=0x7f67ac00b040).stop 2023-11-15T01:56:06.226 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:06.223+0000 7f67c0b9d700 1 -- 172.21.15.137:0/932376478 shutdown_connections 2023-11-15T01:56:06.227 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:06.223+0000 7f67c0b9d700 1 --2- 172.21.15.137:0/932376478 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f67a402ff60 0x7f67a4032420 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:56:06.227 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:06.223+0000 7f67c0b9d700 1 --2- 172.21.15.137:0/932376478 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f67bc0f51a0 0x7f67bc1064e0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:56:06.227 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:06.223+0000 7f67c0b9d700 1 -- 172.21.15.137:0/932376478 >> 172.21.15.137:0/932376478 conn(0x7f67bc0f09a0 msgr2=0x7f67bc05fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:56:06.227 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:06.223+0000 7f67c0b9d700 1 -- 172.21.15.137:0/932376478 shutdown_connections 2023-11-15T01:56:06.227 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:06.223+0000 7f67c0b9d700 1 -- 172.21.15.137:0/932376478 wait complete. 2023-11-15T01:56:06.228 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:56:07.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:56:06 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:56:06.223968+0000 mon.smithi055 (mon.0) 265 : audit [DBG] from='client.? 172.21.15.137:0/932376478' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:56:07.621 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:56:07.622 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:56:08.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:56:07 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:56:06.751346+0000 mgr.smithi055.szwngw (mgr.14164) 198 : cluster [DBG] pgmap v186: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:56:08.744 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:08.743+0000 7f844bcfe700 1 -- 172.21.15.137:0/980327581 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f84440f5410 msgr2=0x7f84440f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:56:08.744 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:08.743+0000 7f844bcfe700 1 --2- 172.21.15.137:0/980327581 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f84440f5410 0x7f84440f5830 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7f8438009fb0 tx=0x7f8438009b30).stop 2023-11-15T01:56:08.744 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:08.743+0000 7f844bcfe700 1 -- 172.21.15.137:0/980327581 shutdown_connections 2023-11-15T01:56:08.745 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:08.743+0000 7f844bcfe700 1 --2- 172.21.15.137:0/980327581 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f84440f5410 0x7f84440f5830 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:56:08.745 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:08.743+0000 7f844bcfe700 1 -- 172.21.15.137:0/980327581 >> 172.21.15.137:0/980327581 conn(0x7f84440f0b70 msgr2=0x7f84440f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:56:08.745 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:08.743+0000 7f844bcfe700 1 -- 172.21.15.137:0/980327581 shutdown_connections 2023-11-15T01:56:08.745 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:08.743+0000 7f844bcfe700 1 -- 172.21.15.137:0/980327581 wait complete. 2023-11-15T01:56:08.745 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:08.743+0000 7f844bcfe700 1 Processor -- start 2023-11-15T01:56:08.746 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:08.743+0000 7f844bcfe700 1 -- start start 2023-11-15T01:56:08.747 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:08.744+0000 7f844bcfe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f84440f5410 0x7f8444109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:56:08.748 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:08.744+0000 7f844bcfe700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8444060240 con 0x7f84440f5410 2023-11-15T01:56:08.748 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:08.744+0000 7f8449a9a700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f84440f5410 0x7f8444109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:56:08.748 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:08.744+0000 7f8449a9a700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f84440f5410 0x7f8444109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:41914/0 (socket says 172.21.15.137:41914) 2023-11-15T01:56:08.748 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:08.744+0000 7f8449a9a700 1 -- 172.21.15.137:0/2977537605 learned_addr learned my addr 172.21.15.137:0/2977537605 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:56:08.748 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:08.744+0000 7f8449a9a700 1 -- 172.21.15.137:0/2977537605 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8438005040 con 0x7f84440f5410 2023-11-15T01:56:08.749 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:08.745+0000 7f8449a9a700 1 --2- 172.21.15.137:0/2977537605 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f84440f5410 0x7f8444109430 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f8438013510 tx=0x7f843800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:56:08.749 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:08.745+0000 7f84367fc700 1 -- 172.21.15.137:0/2977537605 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8438009320 con 0x7f84440f5410 2023-11-15T01:56:08.749 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:08.745+0000 7f844bcfe700 1 -- 172.21.15.137:0/2977537605 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8444060440 con 0x7f84440f5410 2023-11-15T01:56:08.749 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:08.745+0000 7f84367fc700 1 -- 172.21.15.137:0/2977537605 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8438013f00 con 0x7f84440f5410 2023-11-15T01:56:08.749 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:08.745+0000 7f84367fc700 1 -- 172.21.15.137:0/2977537605 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8438014d90 con 0x7f84440f5410 2023-11-15T01:56:08.750 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:08.745+0000 7f844bcfe700 1 -- 172.21.15.137:0/2977537605 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8444109ae0 con 0x7f84440f5410 2023-11-15T01:56:08.750 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:08.746+0000 7f844bcfe700 1 -- 172.21.15.137:0/2977537605 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8444045ad0 con 0x7f84440f5410 2023-11-15T01:56:08.750 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:08.746+0000 7f84367fc700 1 -- 172.21.15.137:0/2977537605 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f84380095f0 con 0x7f84440f5410 2023-11-15T01:56:08.750 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:08.746+0000 7f84367fc700 1 --2- 172.21.15.137:0/2977537605 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f843002ff60 0x7f8430032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:56:08.750 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:08.746+0000 7f84367fc700 1 -- 172.21.15.137:0/2977537605 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f84380410c0 con 0x7f84440f5410 2023-11-15T01:56:08.751 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:08.746+0000 7f8449299700 1 --2- 172.21.15.137:0/2977537605 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f843002ff60 0x7f8430032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:56:08.751 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:08.747+0000 7f8449299700 1 --2- 172.21.15.137:0/2977537605 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f843002ff60 0x7f8430032420 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f84400096a0 tx=0x7f8440006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:56:08.751 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:08.751+0000 7f84367fc700 1 -- 172.21.15.137:0/2977537605 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8438014420 con 0x7f84440f5410 2023-11-15T01:56:08.960 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:08.960+0000 7f844bcfe700 1 -- 172.21.15.137:0/2977537605 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f84440fae60 con 0x7f84440f5410 2023-11-15T01:56:08.961 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:08.960+0000 7f84367fc700 1 -- 172.21.15.137:0/2977537605 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8438010230 con 0x7f84440f5410 2023-11-15T01:56:08.961 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:56:08.962 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:56:08.963 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:08.962+0000 7f844bcfe700 1 -- 172.21.15.137:0/2977537605 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f843002ff60 msgr2=0x7f8430032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:56:08.963 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:08.962+0000 7f844bcfe700 1 --2- 172.21.15.137:0/2977537605 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f843002ff60 0x7f8430032420 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f84400096a0 tx=0x7f8440006b40).stop 2023-11-15T01:56:08.963 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:08.962+0000 7f844bcfe700 1 -- 172.21.15.137:0/2977537605 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f84440f5410 msgr2=0x7f8444109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:56:08.963 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:08.962+0000 7f844bcfe700 1 --2- 172.21.15.137:0/2977537605 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f84440f5410 0x7f8444109430 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f8438013510 tx=0x7f843800b040).stop 2023-11-15T01:56:08.963 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:08.962+0000 7f844bcfe700 1 -- 172.21.15.137:0/2977537605 shutdown_connections 2023-11-15T01:56:08.964 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:08.962+0000 7f844bcfe700 1 --2- 172.21.15.137:0/2977537605 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f843002ff60 0x7f8430032420 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:56:08.964 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:08.962+0000 7f844bcfe700 1 --2- 172.21.15.137:0/2977537605 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f84440f5410 0x7f8444109430 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:56:08.964 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:08.962+0000 7f844bcfe700 1 -- 172.21.15.137:0/2977537605 >> 172.21.15.137:0/2977537605 conn(0x7f84440f0b70 msgr2=0x7f84440f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:56:08.964 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:08.962+0000 7f844bcfe700 1 -- 172.21.15.137:0/2977537605 shutdown_connections 2023-11-15T01:56:08.964 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:08.962+0000 7f844bcfe700 1 -- 172.21.15.137:0/2977537605 wait complete. 2023-11-15T01:56:08.965 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:56:10.077 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:56:09 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:56:08.751599+0000 mgr.smithi055.szwngw (mgr.14164) 199 : cluster [DBG] pgmap v187: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:56:10.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:56:09 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:56:08.961541+0000 mon.smithi055 (mon.0) 266 : audit [DBG] from='client.? 172.21.15.137:0/2977537605' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:56:10.346 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:56:10.347 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:56:11.491 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:11.489+0000 7fe1323a9700 1 -- 172.21.15.137:0/3068760201 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe12c0f5410 msgr2=0x7fe12c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:56:11.491 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:11.489+0000 7fe1323a9700 1 --2- 172.21.15.137:0/3068760201 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe12c0f5410 0x7fe12c0f5830 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7fe11c009fb0 tx=0x7fe11c009b30).stop 2023-11-15T01:56:11.491 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:11.489+0000 7fe1323a9700 1 -- 172.21.15.137:0/3068760201 shutdown_connections 2023-11-15T01:56:11.492 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:11.489+0000 7fe1323a9700 1 --2- 172.21.15.137:0/3068760201 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe12c0f5410 0x7fe12c0f5830 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:56:11.492 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:11.489+0000 7fe1323a9700 1 -- 172.21.15.137:0/3068760201 >> 172.21.15.137:0/3068760201 conn(0x7fe12c0f0b70 msgr2=0x7fe12c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:56:11.492 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:11.489+0000 7fe1323a9700 1 -- 172.21.15.137:0/3068760201 shutdown_connections 2023-11-15T01:56:11.492 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:11.489+0000 7fe1323a9700 1 -- 172.21.15.137:0/3068760201 wait complete. 2023-11-15T01:56:11.492 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:11.490+0000 7fe1323a9700 1 Processor -- start 2023-11-15T01:56:11.493 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:11.490+0000 7fe1323a9700 1 -- start start 2023-11-15T01:56:11.493 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:11.490+0000 7fe1323a9700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe12c0f5410 0x7fe12c109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:56:11.493 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:11.490+0000 7fe1323a9700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe12c060240 con 0x7fe12c0f5410 2023-11-15T01:56:11.495 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:11.491+0000 7fe12bfff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe12c0f5410 0x7fe12c109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:56:11.495 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:11.491+0000 7fe12bfff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe12c0f5410 0x7fe12c109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:41930/0 (socket says 172.21.15.137:41930) 2023-11-15T01:56:11.495 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:11.491+0000 7fe12bfff700 1 -- 172.21.15.137:0/1846506753 learned_addr learned my addr 172.21.15.137:0/1846506753 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:56:11.496 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:11.491+0000 7fe12bfff700 1 -- 172.21.15.137:0/1846506753 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe11c005040 con 0x7fe12c0f5410 2023-11-15T01:56:11.496 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:11.491+0000 7fe12bfff700 1 --2- 172.21.15.137:0/1846506753 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe12c0f5410 0x7fe12c109430 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7fe11c014000 tx=0x7fe11c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:56:11.496 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:11.491+0000 7fe128ff9700 1 -- 172.21.15.137:0/1846506753 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe11c0095f0 con 0x7fe12c0f5410 2023-11-15T01:56:11.496 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:11.491+0000 7fe1323a9700 1 -- 172.21.15.137:0/1846506753 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe12c060440 con 0x7fe12c0f5410 2023-11-15T01:56:11.496 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:11.491+0000 7fe128ff9700 1 -- 172.21.15.137:0/1846506753 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe11c007e70 con 0x7fe12c0f5410 2023-11-15T01:56:11.497 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:11.491+0000 7fe128ff9700 1 -- 172.21.15.137:0/1846506753 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe11c014ea0 con 0x7fe12c0f5410 2023-11-15T01:56:11.497 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:11.491+0000 7fe1323a9700 1 -- 172.21.15.137:0/1846506753 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe12c109ae0 con 0x7fe12c0f5410 2023-11-15T01:56:11.497 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:11.492+0000 7fe1323a9700 1 -- 172.21.15.137:0/1846506753 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe12c045ad0 con 0x7fe12c0f5410 2023-11-15T01:56:11.497 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:11.492+0000 7fe128ff9700 1 -- 172.21.15.137:0/1846506753 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe11c025070 con 0x7fe12c0f5410 2023-11-15T01:56:11.498 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:11.492+0000 7fe128ff9700 1 --2- 172.21.15.137:0/1846506753 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fe11402ff60 0x7fe114032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:56:11.498 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:11.492+0000 7fe128ff9700 1 -- 172.21.15.137:0/1846506753 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe11c040940 con 0x7fe12c0f5410 2023-11-15T01:56:11.498 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:11.493+0000 7fe12b7fe700 1 --2- 172.21.15.137:0/1846506753 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fe11402ff60 0x7fe114032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:56:11.498 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:11.493+0000 7fe12b7fe700 1 --2- 172.21.15.137:0/1846506753 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fe11402ff60 0x7fe114032420 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7fe1200096a0 tx=0x7fe120006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:56:11.499 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:11.497+0000 7fe128ff9700 1 -- 172.21.15.137:0/1846506753 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe11c0079c0 con 0x7fe12c0f5410 2023-11-15T01:56:11.704 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:11.703+0000 7fe1323a9700 1 -- 172.21.15.137:0/1846506753 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe12c0fae60 con 0x7fe12c0f5410 2023-11-15T01:56:11.705 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:11.704+0000 7fe128ff9700 1 -- 172.21.15.137:0/1846506753 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe11c00f300 con 0x7fe12c0f5410 2023-11-15T01:56:11.705 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:56:11.706 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:56:11.707 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:11.706+0000 7fe1323a9700 1 -- 172.21.15.137:0/1846506753 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fe11402ff60 msgr2=0x7fe114032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:56:11.707 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:11.706+0000 7fe1323a9700 1 --2- 172.21.15.137:0/1846506753 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fe11402ff60 0x7fe114032420 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7fe1200096a0 tx=0x7fe120006b40).stop 2023-11-15T01:56:11.707 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:11.706+0000 7fe1323a9700 1 -- 172.21.15.137:0/1846506753 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe12c0f5410 msgr2=0x7fe12c109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:56:11.707 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:11.706+0000 7fe1323a9700 1 --2- 172.21.15.137:0/1846506753 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe12c0f5410 0x7fe12c109430 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7fe11c014000 tx=0x7fe11c00b040).stop 2023-11-15T01:56:11.707 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:11.706+0000 7fe1323a9700 1 -- 172.21.15.137:0/1846506753 shutdown_connections 2023-11-15T01:56:11.708 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:11.706+0000 7fe1323a9700 1 --2- 172.21.15.137:0/1846506753 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fe11402ff60 0x7fe114032420 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:56:11.708 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:11.706+0000 7fe1323a9700 1 --2- 172.21.15.137:0/1846506753 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe12c0f5410 0x7fe12c109430 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:56:11.708 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:11.706+0000 7fe1323a9700 1 -- 172.21.15.137:0/1846506753 >> 172.21.15.137:0/1846506753 conn(0x7fe12c0f0b70 msgr2=0x7fe12c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:56:11.708 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:11.706+0000 7fe1323a9700 1 -- 172.21.15.137:0/1846506753 shutdown_connections 2023-11-15T01:56:11.709 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:11.706+0000 7fe1323a9700 1 -- 172.21.15.137:0/1846506753 wait complete. 2023-11-15T01:56:11.709 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:56:11.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:56:11 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:56:10.751904+0000 mgr.smithi055.szwngw (mgr.14164) 200 : cluster [DBG] pgmap v188: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:56:12.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:56:12 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:56:11.705140+0000 mon.smithi055 (mon.0) 267 : audit [DBG] from='client.? 172.21.15.137:0/1846506753' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:56:13.079 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:56:13.080 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:56:13.827 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:56:13 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:56:12.752202+0000 mgr.smithi055.szwngw (mgr.14164) 201 : cluster [DBG] pgmap v189: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:56:14.257 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:14.256+0000 7f426a6c2700 1 -- 172.21.15.137:0/273404315 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f42640f5410 msgr2=0x7f42640f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:56:14.258 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:14.256+0000 7f426a6c2700 1 --2- 172.21.15.137:0/273404315 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f42640f5410 0x7f42640f5830 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7f424c009fb0 tx=0x7f424c009b30).stop 2023-11-15T01:56:14.258 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:14.256+0000 7f426a6c2700 1 -- 172.21.15.137:0/273404315 shutdown_connections 2023-11-15T01:56:14.258 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:14.256+0000 7f426a6c2700 1 --2- 172.21.15.137:0/273404315 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f42640f5410 0x7f42640f5830 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:56:14.258 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:14.256+0000 7f426a6c2700 1 -- 172.21.15.137:0/273404315 >> 172.21.15.137:0/273404315 conn(0x7f42640f0b70 msgr2=0x7f42640f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:56:14.259 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:14.256+0000 7f426a6c2700 1 -- 172.21.15.137:0/273404315 shutdown_connections 2023-11-15T01:56:14.259 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:14.256+0000 7f426a6c2700 1 -- 172.21.15.137:0/273404315 wait complete. 2023-11-15T01:56:14.259 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:14.257+0000 7f426a6c2700 1 Processor -- start 2023-11-15T01:56:14.259 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:14.257+0000 7f426a6c2700 1 -- start start 2023-11-15T01:56:14.260 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:14.257+0000 7f426a6c2700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f42640f5410 0x7f42641093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:56:14.260 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:14.257+0000 7f426a6c2700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4264060280 con 0x7f42640f5410 2023-11-15T01:56:14.261 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:14.257+0000 7f4263fff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f42640f5410 0x7f42641093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:56:14.262 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:14.257+0000 7f4263fff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f42640f5410 0x7f42641093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:41948/0 (socket says 172.21.15.137:41948) 2023-11-15T01:56:14.262 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:14.257+0000 7f4263fff700 1 -- 172.21.15.137:0/1604300735 learned_addr learned my addr 172.21.15.137:0/1604300735 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:56:14.262 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:14.258+0000 7f4263fff700 1 -- 172.21.15.137:0/1604300735 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f424c005040 con 0x7f42640f5410 2023-11-15T01:56:14.262 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:14.258+0000 7f4263fff700 1 --2- 172.21.15.137:0/1604300735 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f42640f5410 0x7f42641093f0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7f424c013510 tx=0x7f424c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:56:14.262 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:14.258+0000 7f4260ff9700 1 -- 172.21.15.137:0/1604300735 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f424c009320 con 0x7f42640f5410 2023-11-15T01:56:14.263 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:14.258+0000 7f4260ff9700 1 -- 172.21.15.137:0/1604300735 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f424c013f00 con 0x7f42640f5410 2023-11-15T01:56:14.263 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:14.258+0000 7f426a6c2700 1 -- 172.21.15.137:0/1604300735 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4264060480 con 0x7f42640f5410 2023-11-15T01:56:14.263 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:14.258+0000 7f4260ff9700 1 -- 172.21.15.137:0/1604300735 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f424c014d90 con 0x7f42640f5410 2023-11-15T01:56:14.263 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:14.258+0000 7f426a6c2700 1 -- 172.21.15.137:0/1604300735 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4264109b20 con 0x7f42640f5410 2023-11-15T01:56:14.263 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:14.259+0000 7f4260ff9700 1 -- 172.21.15.137:0/1604300735 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f424c014420 con 0x7f42640f5410 2023-11-15T01:56:14.264 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:14.259+0000 7f426a6c2700 1 -- 172.21.15.137:0/1604300735 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4264045ad0 con 0x7f42640f5410 2023-11-15T01:56:14.264 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:14.259+0000 7f4260ff9700 1 --2- 172.21.15.137:0/1604300735 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f425002ff60 0x7f4250032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:56:14.264 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:14.259+0000 7f4260ff9700 1 -- 172.21.15.137:0/1604300735 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f424c041070 con 0x7f42640f5410 2023-11-15T01:56:14.264 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:14.259+0000 7f42637fe700 1 --2- 172.21.15.137:0/1604300735 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f425002ff60 0x7f4250032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:56:14.264 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:14.259+0000 7f42637fe700 1 --2- 172.21.15.137:0/1604300735 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f425002ff60 0x7f4250032420 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f42540096a0 tx=0x7f4254006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:56:14.265 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:14.264+0000 7f4260ff9700 1 -- 172.21.15.137:0/1604300735 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f424c0095f0 con 0x7f42640f5410 2023-11-15T01:56:14.469 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:14.468+0000 7f426a6c2700 1 -- 172.21.15.137:0/1604300735 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f426410a340 con 0x7f42640f5410 2023-11-15T01:56:14.470 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:14.469+0000 7f4260ff9700 1 -- 172.21.15.137:0/1604300735 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f424c01db70 con 0x7f42640f5410 2023-11-15T01:56:14.470 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:56:14.470 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:56:14.471 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:14.471+0000 7f426a6c2700 1 -- 172.21.15.137:0/1604300735 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f425002ff60 msgr2=0x7f4250032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:56:14.471 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:14.471+0000 7f426a6c2700 1 --2- 172.21.15.137:0/1604300735 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f425002ff60 0x7f4250032420 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f42540096a0 tx=0x7f4254006b40).stop 2023-11-15T01:56:14.472 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:14.471+0000 7f426a6c2700 1 -- 172.21.15.137:0/1604300735 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f42640f5410 msgr2=0x7f42641093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:56:14.472 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:14.471+0000 7f426a6c2700 1 --2- 172.21.15.137:0/1604300735 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f42640f5410 0x7f42641093f0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7f424c013510 tx=0x7f424c00b040).stop 2023-11-15T01:56:14.472 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:14.471+0000 7f426a6c2700 1 -- 172.21.15.137:0/1604300735 shutdown_connections 2023-11-15T01:56:14.472 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:14.471+0000 7f426a6c2700 1 --2- 172.21.15.137:0/1604300735 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f425002ff60 0x7f4250032420 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:56:14.472 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:14.471+0000 7f426a6c2700 1 --2- 172.21.15.137:0/1604300735 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f42640f5410 0x7f42641093f0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:56:14.473 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:14.471+0000 7f426a6c2700 1 -- 172.21.15.137:0/1604300735 >> 172.21.15.137:0/1604300735 conn(0x7f42640f0b70 msgr2=0x7f42640f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:56:14.473 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:14.471+0000 7f426a6c2700 1 -- 172.21.15.137:0/1604300735 shutdown_connections 2023-11-15T01:56:14.473 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:14.471+0000 7f426a6c2700 1 -- 172.21.15.137:0/1604300735 wait complete. 2023-11-15T01:56:14.473 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:56:14.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:56:14 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:56:14.470264+0000 mon.smithi055 (mon.0) 268 : audit [DBG] from='client.? 172.21.15.137:0/1604300735' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:56:15.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:56:15 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:56:14.752543+0000 mgr.smithi055.szwngw (mgr.14164) 202 : cluster [DBG] pgmap v190: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:56:15.863 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:56:15.864 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:56:17.024 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:17.023+0000 7fedcd61e700 1 -- 172.21.15.137:0/263931761 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fedc80f51a0 msgr2=0x7fedc80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:56:17.024 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:17.023+0000 7fedcd61e700 1 --2- 172.21.15.137:0/263931761 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fedc80f51a0 0x7fedc80f55c0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7fedb8009fb0 tx=0x7fedb8009b30).stop 2023-11-15T01:56:17.025 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:17.023+0000 7fedcd61e700 1 -- 172.21.15.137:0/263931761 shutdown_connections 2023-11-15T01:56:17.025 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:17.023+0000 7fedcd61e700 1 --2- 172.21.15.137:0/263931761 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fedc80f51a0 0x7fedc80f55c0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:56:17.025 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:17.023+0000 7fedcd61e700 1 -- 172.21.15.137:0/263931761 >> 172.21.15.137:0/263931761 conn(0x7fedc80f09a0 msgr2=0x7fedc80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:56:17.025 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:17.023+0000 7fedcd61e700 1 -- 172.21.15.137:0/263931761 shutdown_connections 2023-11-15T01:56:17.025 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:17.023+0000 7fedcd61e700 1 -- 172.21.15.137:0/263931761 wait complete. 2023-11-15T01:56:17.026 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:17.024+0000 7fedcd61e700 1 Processor -- start 2023-11-15T01:56:17.027 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:17.024+0000 7fedcd61e700 1 -- start start 2023-11-15T01:56:17.028 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:17.024+0000 7fedcd61e700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fedc80f51a0 0x7fedc81091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:56:17.028 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:17.024+0000 7fedcd61e700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedc8060220 con 0x7fedc80f51a0 2023-11-15T01:56:17.028 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:17.024+0000 7fedc7fff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fedc80f51a0 0x7fedc81091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:56:17.028 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:17.024+0000 7fedc7fff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fedc80f51a0 0x7fedc81091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:41958/0 (socket says 172.21.15.137:41958) 2023-11-15T01:56:17.028 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:17.024+0000 7fedc7fff700 1 -- 172.21.15.137:0/3810227791 learned_addr learned my addr 172.21.15.137:0/3810227791 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:56:17.029 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:17.025+0000 7fedc7fff700 1 -- 172.21.15.137:0/3810227791 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedb8005040 con 0x7fedc80f51a0 2023-11-15T01:56:17.029 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:17.025+0000 7fedc7fff700 1 --2- 172.21.15.137:0/3810227791 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fedc80f51a0 0x7fedc81091d0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7fedb8013510 tx=0x7fedb800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:56:17.029 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:17.025+0000 7fedc4ff9700 1 -- 172.21.15.137:0/3810227791 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fedb8009320 con 0x7fedc80f51a0 2023-11-15T01:56:17.029 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:17.025+0000 7fedcd61e700 1 -- 172.21.15.137:0/3810227791 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedc8060420 con 0x7fedc80f51a0 2023-11-15T01:56:17.029 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:17.025+0000 7fedc4ff9700 1 -- 172.21.15.137:0/3810227791 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fedb8013f00 con 0x7fedc80f51a0 2023-11-15T01:56:17.030 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:17.025+0000 7fedc4ff9700 1 -- 172.21.15.137:0/3810227791 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fedb8014dc0 con 0x7fedc80f51a0 2023-11-15T01:56:17.030 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:17.025+0000 7fedcd61e700 1 -- 172.21.15.137:0/3810227791 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedc8109880 con 0x7fedc80f51a0 2023-11-15T01:56:17.030 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:17.026+0000 7fedc4ff9700 1 -- 172.21.15.137:0/3810227791 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fedb80095f0 con 0x7fedc80f51a0 2023-11-15T01:56:17.030 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:17.026+0000 7fedcd61e700 1 -- 172.21.15.137:0/3810227791 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedc8059100 con 0x7fedc80f51a0 2023-11-15T01:56:17.030 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:17.026+0000 7fedc4ff9700 1 --2- 172.21.15.137:0/3810227791 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fedb002ff60 0x7fedb0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:56:17.031 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:17.026+0000 7fedc4ff9700 1 -- 172.21.15.137:0/3810227791 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fedb80410c0 con 0x7fedc80f51a0 2023-11-15T01:56:17.031 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:17.026+0000 7fedc77fe700 1 --2- 172.21.15.137:0/3810227791 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fedb002ff60 0x7fedb0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:56:17.031 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:17.027+0000 7fedc77fe700 1 --2- 172.21.15.137:0/3810227791 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fedb002ff60 0x7fedb0032420 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7fedbc0096a0 tx=0x7fedbc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:56:17.032 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:17.031+0000 7fedc4ff9700 1 -- 172.21.15.137:0/3810227791 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fedb801c790 con 0x7fedc80f51a0 2023-11-15T01:56:17.236 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:17.236+0000 7fedcd61e700 1 -- 172.21.15.137:0/3810227791 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fedc8045ad0 con 0x7fedc80f51a0 2023-11-15T01:56:17.237 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:17.236+0000 7fedc4ff9700 1 -- 172.21.15.137:0/3810227791 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fedb8010230 con 0x7fedc80f51a0 2023-11-15T01:56:17.237 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:56:17.238 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:56:17.239 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:17.238+0000 7fedcd61e700 1 -- 172.21.15.137:0/3810227791 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fedb002ff60 msgr2=0x7fedb0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:56:17.239 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:17.238+0000 7fedcd61e700 1 --2- 172.21.15.137:0/3810227791 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fedb002ff60 0x7fedb0032420 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7fedbc0096a0 tx=0x7fedbc006b40).stop 2023-11-15T01:56:17.239 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:17.238+0000 7fedcd61e700 1 -- 172.21.15.137:0/3810227791 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fedc80f51a0 msgr2=0x7fedc81091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:56:17.239 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:17.238+0000 7fedcd61e700 1 --2- 172.21.15.137:0/3810227791 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fedc80f51a0 0x7fedc81091d0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7fedb8013510 tx=0x7fedb800b040).stop 2023-11-15T01:56:17.239 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:17.238+0000 7fedcd61e700 1 -- 172.21.15.137:0/3810227791 shutdown_connections 2023-11-15T01:56:17.240 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:17.238+0000 7fedcd61e700 1 --2- 172.21.15.137:0/3810227791 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fedb002ff60 0x7fedb0032420 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:56:17.240 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:17.238+0000 7fedcd61e700 1 --2- 172.21.15.137:0/3810227791 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fedc80f51a0 0x7fedc81091d0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:56:17.240 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:17.238+0000 7fedcd61e700 1 -- 172.21.15.137:0/3810227791 >> 172.21.15.137:0/3810227791 conn(0x7fedc80f09a0 msgr2=0x7fedc80f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:56:17.240 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:17.238+0000 7fedcd61e700 1 -- 172.21.15.137:0/3810227791 shutdown_connections 2023-11-15T01:56:17.240 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:17.238+0000 7fedcd61e700 1 -- 172.21.15.137:0/3810227791 wait complete. 2023-11-15T01:56:17.241 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:56:18.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:56:17 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:56:16.752851+0000 mgr.smithi055.szwngw (mgr.14164) 203 : cluster [DBG] pgmap v191: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:56:18.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:56:17 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:56:17.237653+0000 mon.smithi055 (mon.0) 269 : audit [DBG] from='client.? 172.21.15.137:0/3810227791' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:56:18.631 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:56:18.632 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:56:19.774 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:19.772+0000 7f1df6abf700 1 -- 172.21.15.137:0/983347230 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1df00f5410 msgr2=0x7f1df00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:56:19.774 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:19.772+0000 7f1df6abf700 1 --2- 172.21.15.137:0/983347230 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1df00f5410 0x7f1df00f5830 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7f1de4009fb0 tx=0x7f1de4009b30).stop 2023-11-15T01:56:19.774 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:19.773+0000 7f1df6abf700 1 -- 172.21.15.137:0/983347230 shutdown_connections 2023-11-15T01:56:19.775 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:19.773+0000 7f1df6abf700 1 --2- 172.21.15.137:0/983347230 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1df00f5410 0x7f1df00f5830 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:56:19.775 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:19.773+0000 7f1df6abf700 1 -- 172.21.15.137:0/983347230 >> 172.21.15.137:0/983347230 conn(0x7f1df00f0b70 msgr2=0x7f1df00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:56:19.775 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:19.773+0000 7f1df6abf700 1 -- 172.21.15.137:0/983347230 shutdown_connections 2023-11-15T01:56:19.775 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:19.773+0000 7f1df6abf700 1 -- 172.21.15.137:0/983347230 wait complete. 2023-11-15T01:56:19.775 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:19.773+0000 7f1df6abf700 1 Processor -- start 2023-11-15T01:56:19.776 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:19.773+0000 7f1df6abf700 1 -- start start 2023-11-15T01:56:19.777 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:19.774+0000 7f1df6abf700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1df00f5410 0x7f1df0105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:56:19.778 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:19.774+0000 7f1df6abf700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1df0105b20 con 0x7f1df00f5410 2023-11-15T01:56:19.778 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:19.774+0000 7f1df485b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1df00f5410 0x7f1df0105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:56:19.778 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:19.774+0000 7f1df485b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1df00f5410 0x7f1df0105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:41564/0 (socket says 172.21.15.137:41564) 2023-11-15T01:56:19.778 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:19.774+0000 7f1df485b700 1 -- 172.21.15.137:0/3717024165 learned_addr learned my addr 172.21.15.137:0/3717024165 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:56:19.778 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:19.774+0000 7f1df485b700 1 -- 172.21.15.137:0/3717024165 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1de4005040 con 0x7f1df00f5410 2023-11-15T01:56:19.779 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:19.775+0000 7f1df485b700 1 --2- 172.21.15.137:0/3717024165 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1df00f5410 0x7f1df0105630 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7f1de4013510 tx=0x7f1de400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:56:19.779 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:19.775+0000 7f1ded7fa700 1 -- 172.21.15.137:0/3717024165 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1de4009320 con 0x7f1df00f5410 2023-11-15T01:56:19.779 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:19.775+0000 7f1df6abf700 1 -- 172.21.15.137:0/3717024165 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1df0105d20 con 0x7f1df00f5410 2023-11-15T01:56:19.779 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:19.775+0000 7f1ded7fa700 1 -- 172.21.15.137:0/3717024165 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1de4013f00 con 0x7f1df00f5410 2023-11-15T01:56:19.779 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:19.775+0000 7f1ded7fa700 1 -- 172.21.15.137:0/3717024165 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1de4014d90 con 0x7f1df00f5410 2023-11-15T01:56:19.780 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:19.775+0000 7f1df6abf700 1 -- 172.21.15.137:0/3717024165 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1df01061b0 con 0x7f1df00f5410 2023-11-15T01:56:19.780 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:19.776+0000 7f1ded7fa700 1 -- 172.21.15.137:0/3717024165 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1de4014420 con 0x7f1df00f5410 2023-11-15T01:56:19.780 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:19.776+0000 7f1df6abf700 1 -- 172.21.15.137:0/3717024165 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1df0045ad0 con 0x7f1df00f5410 2023-11-15T01:56:19.780 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:19.776+0000 7f1ded7fa700 1 --2- 172.21.15.137:0/3717024165 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f1de002ff60 0x7f1de0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:56:19.780 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:19.776+0000 7f1ded7fa700 1 -- 172.21.15.137:0/3717024165 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1de4042df0 con 0x7f1df00f5410 2023-11-15T01:56:19.780 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:19.776+0000 7f1deffff700 1 --2- 172.21.15.137:0/3717024165 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f1de002ff60 0x7f1de0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:56:19.781 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:19.776+0000 7f1deffff700 1 --2- 172.21.15.137:0/3717024165 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f1de002ff60 0x7f1de0032420 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f1ddc0096a0 tx=0x7f1ddc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:56:19.781 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:19.781+0000 7f1ded7fa700 1 -- 172.21.15.137:0/3717024165 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1de40095f0 con 0x7f1df00f5410 2023-11-15T01:56:19.989 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:19.988+0000 7f1df6abf700 1 -- 172.21.15.137:0/3717024165 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1df00fae60 con 0x7f1df00f5410 2023-11-15T01:56:19.990 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:19.989+0000 7f1ded7fa700 1 -- 172.21.15.137:0/3717024165 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1de401fb70 con 0x7f1df00f5410 2023-11-15T01:56:19.990 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:56:19.990 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:56:19.991 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:19.991+0000 7f1df6abf700 1 -- 172.21.15.137:0/3717024165 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f1de002ff60 msgr2=0x7f1de0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:56:19.991 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:19.991+0000 7f1df6abf700 1 --2- 172.21.15.137:0/3717024165 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f1de002ff60 0x7f1de0032420 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f1ddc0096a0 tx=0x7f1ddc006b40).stop 2023-11-15T01:56:19.992 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:19.991+0000 7f1df6abf700 1 -- 172.21.15.137:0/3717024165 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1df00f5410 msgr2=0x7f1df0105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:56:19.992 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:19.991+0000 7f1df6abf700 1 --2- 172.21.15.137:0/3717024165 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1df00f5410 0x7f1df0105630 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7f1de4013510 tx=0x7f1de400b040).stop 2023-11-15T01:56:19.992 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:19.991+0000 7f1df6abf700 1 -- 172.21.15.137:0/3717024165 shutdown_connections 2023-11-15T01:56:19.992 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:19.991+0000 7f1df6abf700 1 --2- 172.21.15.137:0/3717024165 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f1de002ff60 0x7f1de0032420 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:56:19.993 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:19.991+0000 7f1df6abf700 1 --2- 172.21.15.137:0/3717024165 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1df00f5410 0x7f1df0105630 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:56:19.993 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:19.991+0000 7f1df6abf700 1 -- 172.21.15.137:0/3717024165 >> 172.21.15.137:0/3717024165 conn(0x7f1df00f0b70 msgr2=0x7f1df00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:56:19.993 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:19.991+0000 7f1df6abf700 1 -- 172.21.15.137:0/3717024165 shutdown_connections 2023-11-15T01:56:19.993 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:19.991+0000 7f1df6abf700 1 -- 172.21.15.137:0/3717024165 wait complete. 2023-11-15T01:56:19.994 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:56:20.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:56:19 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:56:18.753055+0000 mgr.smithi055.szwngw (mgr.14164) 204 : cluster [DBG] pgmap v192: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:56:21.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:56:20 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:56:19.989945+0000 mon.smithi055 (mon.0) 270 : audit [DBG] from='client.? 172.21.15.137:0/3717024165' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:56:21.365 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:56:21.366 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:56:22.327 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:56:21 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:56:20.753323+0000 mgr.smithi055.szwngw (mgr.14164) 205 : cluster [DBG] pgmap v193: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:56:22.525 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:22.523+0000 7f41144b1700 1 -- 172.21.15.137:0/754468931 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f410c0f6510 msgr2=0x7f410c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:56:22.525 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:22.523+0000 7f41144b1700 1 --2- 172.21.15.137:0/754468931 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f410c0f6510 0x7f410c0f6930 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7f4100009fb0 tx=0x7f4100009b30).stop 2023-11-15T01:56:22.526 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:22.523+0000 7f41144b1700 1 -- 172.21.15.137:0/754468931 shutdown_connections 2023-11-15T01:56:22.526 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:22.523+0000 7f41144b1700 1 --2- 172.21.15.137:0/754468931 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f410c0f6510 0x7f410c0f6930 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:56:22.526 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:22.523+0000 7f41144b1700 1 -- 172.21.15.137:0/754468931 >> 172.21.15.137:0/754468931 conn(0x7f410c0f1c70 msgr2=0x7f410c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:56:22.526 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:22.523+0000 7f41144b1700 1 -- 172.21.15.137:0/754468931 shutdown_connections 2023-11-15T01:56:22.527 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:22.523+0000 7f41144b1700 1 -- 172.21.15.137:0/754468931 wait complete. 2023-11-15T01:56:22.527 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:22.524+0000 7f41144b1700 1 Processor -- start 2023-11-15T01:56:22.527 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:22.524+0000 7f41144b1700 1 -- start start 2023-11-15T01:56:22.527 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:22.524+0000 7f41144b1700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f410c0f6510 0x7f410c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:56:22.527 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:22.524+0000 7f41144b1700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f410c069e20 con 0x7f410c0f6510 2023-11-15T01:56:22.528 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:22.524+0000 7f411224d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f410c0f6510 0x7f410c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:56:22.528 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:22.524+0000 7f411224d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f410c0f6510 0x7f410c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:41576/0 (socket says 172.21.15.137:41576) 2023-11-15T01:56:22.528 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:22.524+0000 7f411224d700 1 -- 172.21.15.137:0/2396425928 learned_addr learned my addr 172.21.15.137:0/2396425928 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:56:22.528 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:22.524+0000 7f411224d700 1 -- 172.21.15.137:0/2396425928 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4100005040 con 0x7f410c0f6510 2023-11-15T01:56:22.528 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:22.524+0000 7f411224d700 1 --2- 172.21.15.137:0/2396425928 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f410c0f6510 0x7f410c06b9a0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f4100013510 tx=0x7f410000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:56:22.529 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:22.525+0000 7f40feffd700 1 -- 172.21.15.137:0/2396425928 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4100009320 con 0x7f410c0f6510 2023-11-15T01:56:22.529 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:22.525+0000 7f41144b1700 1 -- 172.21.15.137:0/2396425928 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f410c069fc0 con 0x7f410c0f6510 2023-11-15T01:56:22.529 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:22.525+0000 7f40feffd700 1 -- 172.21.15.137:0/2396425928 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4100013f00 con 0x7f410c0f6510 2023-11-15T01:56:22.529 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:22.525+0000 7f40feffd700 1 -- 172.21.15.137:0/2396425928 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4100014d90 con 0x7f410c0f6510 2023-11-15T01:56:22.529 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:22.525+0000 7f41144b1700 1 -- 172.21.15.137:0/2396425928 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f410c06a450 con 0x7f410c0f6510 2023-11-15T01:56:22.530 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:22.525+0000 7f40feffd700 1 -- 172.21.15.137:0/2396425928 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f41000095f0 con 0x7f410c0f6510 2023-11-15T01:56:22.531 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:22.525+0000 7f41144b1700 1 -- 172.21.15.137:0/2396425928 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f410c059100 con 0x7f410c0f6510 2023-11-15T01:56:22.531 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:22.526+0000 7f40feffd700 1 --2- 172.21.15.137:0/2396425928 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f40f802ff60 0x7f40f8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:56:22.531 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:22.526+0000 7f40feffd700 1 -- 172.21.15.137:0/2396425928 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4100041000 con 0x7f410c0f6510 2023-11-15T01:56:22.531 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:22.526+0000 7f4111a4c700 1 --2- 172.21.15.137:0/2396425928 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f40f802ff60 0x7f40f8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:56:22.531 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:22.526+0000 7f4111a4c700 1 --2- 172.21.15.137:0/2396425928 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f40f802ff60 0x7f40f8032420 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f41080096a0 tx=0x7f4108006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:56:22.532 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:22.530+0000 7f40feffd700 1 -- 172.21.15.137:0/2396425928 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4100014420 con 0x7f410c0f6510 2023-11-15T01:56:22.737 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:22.736+0000 7f41144b1700 1 -- 172.21.15.137:0/2396425928 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f410c0fbf60 con 0x7f410c0f6510 2023-11-15T01:56:22.737 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:22.737+0000 7f40feffd700 1 -- 172.21.15.137:0/2396425928 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4100010230 con 0x7f410c0f6510 2023-11-15T01:56:22.738 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:56:22.738 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:56:22.739 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:22.738+0000 7f41144b1700 1 -- 172.21.15.137:0/2396425928 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f40f802ff60 msgr2=0x7f40f8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:56:22.739 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:22.738+0000 7f41144b1700 1 --2- 172.21.15.137:0/2396425928 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f40f802ff60 0x7f40f8032420 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f41080096a0 tx=0x7f4108006b40).stop 2023-11-15T01:56:22.740 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:22.738+0000 7f41144b1700 1 -- 172.21.15.137:0/2396425928 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f410c0f6510 msgr2=0x7f410c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:56:22.740 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:22.738+0000 7f41144b1700 1 --2- 172.21.15.137:0/2396425928 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f410c0f6510 0x7f410c06b9a0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f4100013510 tx=0x7f410000b040).stop 2023-11-15T01:56:22.740 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:22.738+0000 7f41144b1700 1 -- 172.21.15.137:0/2396425928 shutdown_connections 2023-11-15T01:56:22.740 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:22.738+0000 7f41144b1700 1 --2- 172.21.15.137:0/2396425928 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f40f802ff60 0x7f40f8032420 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:56:22.740 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:22.738+0000 7f41144b1700 1 --2- 172.21.15.137:0/2396425928 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f410c0f6510 0x7f410c06b9a0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:56:22.741 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:22.738+0000 7f41144b1700 1 -- 172.21.15.137:0/2396425928 >> 172.21.15.137:0/2396425928 conn(0x7f410c0f1c70 msgr2=0x7f410c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:56:22.741 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:22.738+0000 7f41144b1700 1 -- 172.21.15.137:0/2396425928 shutdown_connections 2023-11-15T01:56:22.741 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:22.738+0000 7f41144b1700 1 -- 172.21.15.137:0/2396425928 wait complete. 2023-11-15T01:56:22.741 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:56:23.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:56:22 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:56:22.738074+0000 mon.smithi055 (mon.0) 271 : audit [DBG] from='client.? 172.21.15.137:0/2396425928' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:56:24.119 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:56:24.120 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:56:24.327 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:56:23 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:56:22.753664+0000 mgr.smithi055.szwngw (mgr.14164) 206 : cluster [DBG] pgmap v194: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:56:25.223 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:25.221+0000 7f64e6d7b700 1 -- 172.21.15.137:0/1689625687 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f64e00f5410 msgr2=0x7f64e00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:56:25.224 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:25.221+0000 7f64e6d7b700 1 --2- 172.21.15.137:0/1689625687 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f64e00f5410 0x7f64e00f5830 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7f64d4009fb0 tx=0x7f64d4009b30).stop 2023-11-15T01:56:25.224 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:25.221+0000 7f64e6d7b700 1 -- 172.21.15.137:0/1689625687 shutdown_connections 2023-11-15T01:56:25.224 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:25.221+0000 7f64e6d7b700 1 --2- 172.21.15.137:0/1689625687 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f64e00f5410 0x7f64e00f5830 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:56:25.224 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:25.221+0000 7f64e6d7b700 1 -- 172.21.15.137:0/1689625687 >> 172.21.15.137:0/1689625687 conn(0x7f64e00f0b70 msgr2=0x7f64e00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:56:25.224 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:25.221+0000 7f64e6d7b700 1 -- 172.21.15.137:0/1689625687 shutdown_connections 2023-11-15T01:56:25.225 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:25.221+0000 7f64e6d7b700 1 -- 172.21.15.137:0/1689625687 wait complete. 2023-11-15T01:56:25.225 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:25.221+0000 7f64e6d7b700 1 Processor -- start 2023-11-15T01:56:25.225 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:25.222+0000 7f64e6d7b700 1 -- start start 2023-11-15T01:56:25.225 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:25.222+0000 7f64e6d7b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f64e00f5410 0x7f64e01093d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:56:25.225 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:25.222+0000 7f64e6d7b700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64e0060240 con 0x7f64e00f5410 2023-11-15T01:56:25.225 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:25.222+0000 7f64e4b17700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f64e00f5410 0x7f64e01093d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:56:25.226 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:25.222+0000 7f64e4b17700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f64e00f5410 0x7f64e01093d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:41590/0 (socket says 172.21.15.137:41590) 2023-11-15T01:56:25.226 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:25.222+0000 7f64e4b17700 1 -- 172.21.15.137:0/1796612218 learned_addr learned my addr 172.21.15.137:0/1796612218 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:56:25.226 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:25.222+0000 7f64e4b17700 1 -- 172.21.15.137:0/1796612218 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64d4005040 con 0x7f64e00f5410 2023-11-15T01:56:25.226 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:25.222+0000 7f64e4b17700 1 --2- 172.21.15.137:0/1796612218 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f64e00f5410 0x7f64e01093d0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7f64d4013510 tx=0x7f64d400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:56:25.227 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:25.223+0000 7f64dd7fa700 1 -- 172.21.15.137:0/1796612218 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f64d4009320 con 0x7f64e00f5410 2023-11-15T01:56:25.227 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:25.223+0000 7f64e6d7b700 1 -- 172.21.15.137:0/1796612218 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64e0060440 con 0x7f64e00f5410 2023-11-15T01:56:25.227 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:25.223+0000 7f64dd7fa700 1 -- 172.21.15.137:0/1796612218 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f64d4013f00 con 0x7f64e00f5410 2023-11-15T01:56:25.227 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:25.223+0000 7f64dd7fa700 1 -- 172.21.15.137:0/1796612218 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f64d4014d90 con 0x7f64e00f5410 2023-11-15T01:56:25.227 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:25.223+0000 7f64e6d7b700 1 -- 172.21.15.137:0/1796612218 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64e0109a80 con 0x7f64e00f5410 2023-11-15T01:56:25.228 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:25.224+0000 7f64dd7fa700 1 -- 172.21.15.137:0/1796612218 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f64d40095f0 con 0x7f64e00f5410 2023-11-15T01:56:25.229 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:25.224+0000 7f64e6d7b700 1 -- 172.21.15.137:0/1796612218 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64e0045ad0 con 0x7f64e00f5410 2023-11-15T01:56:25.229 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:25.224+0000 7f64dd7fa700 1 --2- 172.21.15.137:0/1796612218 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f64d002ff60 0x7f64d0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:56:25.229 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:25.224+0000 7f64dd7fa700 1 -- 172.21.15.137:0/1796612218 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f64d4040fc0 con 0x7f64e00f5410 2023-11-15T01:56:25.229 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:25.224+0000 7f64dffff700 1 --2- 172.21.15.137:0/1796612218 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f64d002ff60 0x7f64d0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:56:25.229 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:25.224+0000 7f64dffff700 1 --2- 172.21.15.137:0/1796612218 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f64d002ff60 0x7f64d0032420 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f64cc0096a0 tx=0x7f64cc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:56:25.230 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:25.228+0000 7f64dd7fa700 1 -- 172.21.15.137:0/1796612218 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f64d4014420 con 0x7f64e00f5410 2023-11-15T01:56:25.435 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:25.434+0000 7f64e6d7b700 1 -- 172.21.15.137:0/1796612218 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f64e010a290 con 0x7f64e00f5410 2023-11-15T01:56:25.435 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:25.435+0000 7f64dd7fa700 1 -- 172.21.15.137:0/1796612218 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f64d4010230 con 0x7f64e00f5410 2023-11-15T01:56:25.436 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:56:25.436 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:56:25.437 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:25.436+0000 7f64e6d7b700 1 -- 172.21.15.137:0/1796612218 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f64d002ff60 msgr2=0x7f64d0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:56:25.437 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:25.436+0000 7f64e6d7b700 1 --2- 172.21.15.137:0/1796612218 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f64d002ff60 0x7f64d0032420 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f64cc0096a0 tx=0x7f64cc006b40).stop 2023-11-15T01:56:25.437 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:25.436+0000 7f64e6d7b700 1 -- 172.21.15.137:0/1796612218 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f64e00f5410 msgr2=0x7f64e01093d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:56:25.438 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:25.436+0000 7f64e6d7b700 1 --2- 172.21.15.137:0/1796612218 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f64e00f5410 0x7f64e01093d0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7f64d4013510 tx=0x7f64d400b040).stop 2023-11-15T01:56:25.438 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:25.436+0000 7f64e6d7b700 1 -- 172.21.15.137:0/1796612218 shutdown_connections 2023-11-15T01:56:25.438 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:25.436+0000 7f64e6d7b700 1 --2- 172.21.15.137:0/1796612218 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f64d002ff60 0x7f64d0032420 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:56:25.438 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:25.436+0000 7f64e6d7b700 1 --2- 172.21.15.137:0/1796612218 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f64e00f5410 0x7f64e01093d0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:56:25.438 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:25.436+0000 7f64e6d7b700 1 -- 172.21.15.137:0/1796612218 >> 172.21.15.137:0/1796612218 conn(0x7f64e00f0b70 msgr2=0x7f64e00f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:56:25.439 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:25.436+0000 7f64e6d7b700 1 -- 172.21.15.137:0/1796612218 shutdown_connections 2023-11-15T01:56:25.439 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:25.436+0000 7f64e6d7b700 1 -- 172.21.15.137:0/1796612218 wait complete. 2023-11-15T01:56:25.439 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:56:26.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:56:25 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:56:24.754077+0000 mgr.smithi055.szwngw (mgr.14164) 207 : cluster [DBG] pgmap v195: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:56:26.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:56:25 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:56:25.435836+0000 mon.smithi055 (mon.0) 272 : audit [DBG] from='client.? 172.21.15.137:0/1796612218' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:56:26.815 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:56:26.816 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:56:27.949 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:27.949+0000 7fca547c2700 1 -- 172.21.15.137:0/2805875740 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fca4c0f5410 msgr2=0x7fca4c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:56:27.950 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:27.949+0000 7fca547c2700 1 --2- 172.21.15.137:0/2805875740 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fca4c0f5410 0x7fca4c0f5830 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7fca3c009fb0 tx=0x7fca3c009b30).stop 2023-11-15T01:56:27.950 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:27.949+0000 7fca547c2700 1 -- 172.21.15.137:0/2805875740 shutdown_connections 2023-11-15T01:56:27.950 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:27.949+0000 7fca547c2700 1 --2- 172.21.15.137:0/2805875740 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fca4c0f5410 0x7fca4c0f5830 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:56:27.951 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:27.949+0000 7fca547c2700 1 -- 172.21.15.137:0/2805875740 >> 172.21.15.137:0/2805875740 conn(0x7fca4c0f0b70 msgr2=0x7fca4c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:56:27.951 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:27.949+0000 7fca547c2700 1 -- 172.21.15.137:0/2805875740 shutdown_connections 2023-11-15T01:56:27.951 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:27.949+0000 7fca547c2700 1 -- 172.21.15.137:0/2805875740 wait complete. 2023-11-15T01:56:27.953 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:27.949+0000 7fca547c2700 1 Processor -- start 2023-11-15T01:56:27.953 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:27.949+0000 7fca547c2700 1 -- start start 2023-11-15T01:56:27.954 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:27.950+0000 7fca547c2700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fca4c0f5410 0x7fca4c105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:56:27.954 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:27.950+0000 7fca547c2700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca4c105b20 con 0x7fca4c0f5410 2023-11-15T01:56:27.954 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:27.950+0000 7fca5255e700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fca4c0f5410 0x7fca4c105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:56:27.954 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:27.950+0000 7fca5255e700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fca4c0f5410 0x7fca4c105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:41594/0 (socket says 172.21.15.137:41594) 2023-11-15T01:56:27.955 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:27.950+0000 7fca5255e700 1 -- 172.21.15.137:0/633746735 learned_addr learned my addr 172.21.15.137:0/633746735 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:56:27.955 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:27.950+0000 7fca5255e700 1 -- 172.21.15.137:0/633746735 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca3c005040 con 0x7fca4c0f5410 2023-11-15T01:56:27.955 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:27.951+0000 7fca5255e700 1 --2- 172.21.15.137:0/633746735 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fca4c0f5410 0x7fca4c105630 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7fca3c013510 tx=0x7fca3c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:56:27.955 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:27.951+0000 7fca42ffd700 1 -- 172.21.15.137:0/633746735 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca3c009320 con 0x7fca4c0f5410 2023-11-15T01:56:27.955 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:27.951+0000 7fca547c2700 1 -- 172.21.15.137:0/633746735 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca4c105d20 con 0x7fca4c0f5410 2023-11-15T01:56:27.956 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:27.951+0000 7fca42ffd700 1 -- 172.21.15.137:0/633746735 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fca3c013f00 con 0x7fca4c0f5410 2023-11-15T01:56:27.956 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:27.951+0000 7fca42ffd700 1 -- 172.21.15.137:0/633746735 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca3c014d90 con 0x7fca4c0f5410 2023-11-15T01:56:27.956 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:27.951+0000 7fca547c2700 1 -- 172.21.15.137:0/633746735 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca4c1061b0 con 0x7fca4c0f5410 2023-11-15T01:56:27.956 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:27.952+0000 7fca547c2700 1 -- 172.21.15.137:0/633746735 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca4c045ad0 con 0x7fca4c0f5410 2023-11-15T01:56:27.957 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:27.952+0000 7fca42ffd700 1 -- 172.21.15.137:0/633746735 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fca3c0095f0 con 0x7fca4c0f5410 2023-11-15T01:56:27.957 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:27.952+0000 7fca42ffd700 1 --2- 172.21.15.137:0/633746735 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fca3802ff60 0x7fca38032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:56:27.957 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:27.952+0000 7fca42ffd700 1 -- 172.21.15.137:0/633746735 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fca3c041000 con 0x7fca4c0f5410 2023-11-15T01:56:27.957 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:27.952+0000 7fca51d5d700 1 --2- 172.21.15.137:0/633746735 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fca3802ff60 0x7fca38032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:56:27.957 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:27.952+0000 7fca51d5d700 1 --2- 172.21.15.137:0/633746735 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fca3802ff60 0x7fca38032420 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fca480096a0 tx=0x7fca48006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:56:27.958 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:27.957+0000 7fca42ffd700 1 -- 172.21.15.137:0/633746735 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fca3c014420 con 0x7fca4c0f5410 2023-11-15T01:56:28.165 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:28.165+0000 7fca547c2700 1 -- 172.21.15.137:0/633746735 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fca4c0fae60 con 0x7fca4c0f5410 2023-11-15T01:56:28.166 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:28.165+0000 7fca42ffd700 1 -- 172.21.15.137:0/633746735 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fca3c01d7f0 con 0x7fca4c0f5410 2023-11-15T01:56:28.166 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:56:28.167 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:56:28.168 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:28.167+0000 7fca547c2700 1 -- 172.21.15.137:0/633746735 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fca3802ff60 msgr2=0x7fca38032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:56:28.168 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:28.167+0000 7fca547c2700 1 --2- 172.21.15.137:0/633746735 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fca3802ff60 0x7fca38032420 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fca480096a0 tx=0x7fca48006b40).stop 2023-11-15T01:56:28.169 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:28.167+0000 7fca547c2700 1 -- 172.21.15.137:0/633746735 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fca4c0f5410 msgr2=0x7fca4c105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:56:28.169 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:28.167+0000 7fca547c2700 1 --2- 172.21.15.137:0/633746735 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fca4c0f5410 0x7fca4c105630 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7fca3c013510 tx=0x7fca3c00b040).stop 2023-11-15T01:56:28.169 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:28.167+0000 7fca547c2700 1 -- 172.21.15.137:0/633746735 shutdown_connections 2023-11-15T01:56:28.169 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:28.167+0000 7fca547c2700 1 --2- 172.21.15.137:0/633746735 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fca3802ff60 0x7fca38032420 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:56:28.169 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:28.167+0000 7fca547c2700 1 --2- 172.21.15.137:0/633746735 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fca4c0f5410 0x7fca4c105630 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:56:28.170 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:28.167+0000 7fca547c2700 1 -- 172.21.15.137:0/633746735 >> 172.21.15.137:0/633746735 conn(0x7fca4c0f0b70 msgr2=0x7fca4c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:56:28.170 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:28.167+0000 7fca547c2700 1 -- 172.21.15.137:0/633746735 shutdown_connections 2023-11-15T01:56:28.170 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:28.167+0000 7fca547c2700 1 -- 172.21.15.137:0/633746735 wait complete. 2023-11-15T01:56:28.170 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:56:28.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:56:27 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:56:26.754460+0000 mgr.smithi055.szwngw (mgr.14164) 208 : cluster [DBG] pgmap v196: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:56:29.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:56:28 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:56:28.166558+0000 mon.smithi055 (mon.0) 273 : audit [DBG] from='client.? 172.21.15.137:0/633746735' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:56:29.531 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:56:29.532 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:56:30.327 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:56:29 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:56:28.754827+0000 mgr.smithi055.szwngw (mgr.14164) 209 : cluster [DBG] pgmap v197: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:56:30.690 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:30.689+0000 7fd302f89700 1 -- 172.21.15.137:0/3822850087 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd2fc0f5410 msgr2=0x7fd2fc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:56:30.690 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:30.689+0000 7fd302f89700 1 --2- 172.21.15.137:0/3822850087 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd2fc0f5410 0x7fd2fc0f5830 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7fd2f0009fb0 tx=0x7fd2f0009b30).stop 2023-11-15T01:56:30.691 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:30.689+0000 7fd302f89700 1 -- 172.21.15.137:0/3822850087 shutdown_connections 2023-11-15T01:56:30.691 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:30.689+0000 7fd302f89700 1 --2- 172.21.15.137:0/3822850087 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd2fc0f5410 0x7fd2fc0f5830 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:56:30.691 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:30.689+0000 7fd302f89700 1 -- 172.21.15.137:0/3822850087 >> 172.21.15.137:0/3822850087 conn(0x7fd2fc0f0b70 msgr2=0x7fd2fc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:56:30.691 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:30.689+0000 7fd302f89700 1 -- 172.21.15.137:0/3822850087 shutdown_connections 2023-11-15T01:56:30.691 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:30.689+0000 7fd302f89700 1 -- 172.21.15.137:0/3822850087 wait complete. 2023-11-15T01:56:30.692 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:30.689+0000 7fd302f89700 1 Processor -- start 2023-11-15T01:56:30.692 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:30.690+0000 7fd302f89700 1 -- start start 2023-11-15T01:56:30.693 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:30.690+0000 7fd302f89700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd2fc0f5410 0x7fd2fc1066b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:56:30.694 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:30.690+0000 7fd302f89700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2fc106ba0 con 0x7fd2fc0f5410 2023-11-15T01:56:30.694 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:30.690+0000 7fd300d25700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd2fc0f5410 0x7fd2fc1066b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:56:30.694 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:30.690+0000 7fd300d25700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd2fc0f5410 0x7fd2fc1066b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:44422/0 (socket says 172.21.15.137:44422) 2023-11-15T01:56:30.694 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:30.690+0000 7fd300d25700 1 -- 172.21.15.137:0/766363847 learned_addr learned my addr 172.21.15.137:0/766363847 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:56:30.694 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:30.690+0000 7fd300d25700 1 -- 172.21.15.137:0/766363847 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2f0005040 con 0x7fd2fc0f5410 2023-11-15T01:56:30.695 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:30.690+0000 7fd300d25700 1 --2- 172.21.15.137:0/766363847 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd2fc0f5410 0x7fd2fc1066b0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7fd2f0013510 tx=0x7fd2f000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:56:30.695 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:30.691+0000 7fd2f97fa700 1 -- 172.21.15.137:0/766363847 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd2f0009320 con 0x7fd2fc0f5410 2023-11-15T01:56:30.695 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:30.691+0000 7fd302f89700 1 -- 172.21.15.137:0/766363847 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2fc106da0 con 0x7fd2fc0f5410 2023-11-15T01:56:30.695 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:30.691+0000 7fd2f97fa700 1 -- 172.21.15.137:0/766363847 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd2f0013f00 con 0x7fd2fc0f5410 2023-11-15T01:56:30.697 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:30.691+0000 7fd2f97fa700 1 -- 172.21.15.137:0/766363847 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd2f0014d90 con 0x7fd2fc0f5410 2023-11-15T01:56:30.697 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:30.691+0000 7fd302f89700 1 -- 172.21.15.137:0/766363847 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2fc107230 con 0x7fd2fc0f5410 2023-11-15T01:56:30.697 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:30.691+0000 7fd2f97fa700 1 -- 172.21.15.137:0/766363847 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd2f0014420 con 0x7fd2fc0f5410 2023-11-15T01:56:30.697 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:30.691+0000 7fd302f89700 1 -- 172.21.15.137:0/766363847 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2fc045ad0 con 0x7fd2fc0f5410 2023-11-15T01:56:30.698 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:30.692+0000 7fd2f97fa700 1 --2- 172.21.15.137:0/766363847 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fd2ec02ff60 0x7fd2ec032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:56:30.698 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:30.692+0000 7fd2f97fa700 1 -- 172.21.15.137:0/766363847 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd2f0041000 con 0x7fd2fc0f5410 2023-11-15T01:56:30.698 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:30.692+0000 7fd2fbfff700 1 --2- 172.21.15.137:0/766363847 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fd2ec02ff60 0x7fd2ec032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:56:30.698 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:30.692+0000 7fd2fbfff700 1 --2- 172.21.15.137:0/766363847 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fd2ec02ff60 0x7fd2ec032420 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7fd2e80096a0 tx=0x7fd2e8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:56:30.699 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:30.697+0000 7fd2f97fa700 1 -- 172.21.15.137:0/766363847 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd2f00095f0 con 0x7fd2fc0f5410 2023-11-15T01:56:30.902 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:30.901+0000 7fd302f89700 1 -- 172.21.15.137:0/766363847 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd2fc1080e0 con 0x7fd2fc0f5410 2023-11-15T01:56:30.902 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:30.902+0000 7fd2f97fa700 1 -- 172.21.15.137:0/766363847 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd2f001db70 con 0x7fd2fc0f5410 2023-11-15T01:56:30.903 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:56:30.903 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:56:30.905 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:30.904+0000 7fd302f89700 1 -- 172.21.15.137:0/766363847 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fd2ec02ff60 msgr2=0x7fd2ec032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:56:30.906 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:30.904+0000 7fd302f89700 1 --2- 172.21.15.137:0/766363847 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fd2ec02ff60 0x7fd2ec032420 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7fd2e80096a0 tx=0x7fd2e8006b40).stop 2023-11-15T01:56:30.906 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:30.904+0000 7fd302f89700 1 -- 172.21.15.137:0/766363847 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd2fc0f5410 msgr2=0x7fd2fc1066b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:56:30.906 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:30.904+0000 7fd302f89700 1 --2- 172.21.15.137:0/766363847 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd2fc0f5410 0x7fd2fc1066b0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7fd2f0013510 tx=0x7fd2f000b040).stop 2023-11-15T01:56:30.906 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:30.904+0000 7fd302f89700 1 -- 172.21.15.137:0/766363847 shutdown_connections 2023-11-15T01:56:30.906 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:30.904+0000 7fd302f89700 1 --2- 172.21.15.137:0/766363847 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fd2ec02ff60 0x7fd2ec032420 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:56:30.907 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:30.904+0000 7fd302f89700 1 --2- 172.21.15.137:0/766363847 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd2fc0f5410 0x7fd2fc1066b0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:56:30.907 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:30.904+0000 7fd302f89700 1 -- 172.21.15.137:0/766363847 >> 172.21.15.137:0/766363847 conn(0x7fd2fc0f0b70 msgr2=0x7fd2fc0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:56:30.907 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:30.904+0000 7fd302f89700 1 -- 172.21.15.137:0/766363847 shutdown_connections 2023-11-15T01:56:30.907 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:30.904+0000 7fd302f89700 1 -- 172.21.15.137:0/766363847 wait complete. 2023-11-15T01:56:30.907 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:56:31.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:56:31 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:56:30.755262+0000 mgr.smithi055.szwngw (mgr.14164) 210 : cluster [DBG] pgmap v198: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:56:31.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:56:31 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:56:30.903152+0000 mon.smithi055 (mon.0) 274 : audit [DBG] from='client.? 172.21.15.137:0/766363847' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:56:32.287 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:56:32.288 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:56:33.406 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:33.405+0000 7f664c872700 1 -- 172.21.15.137:0/1117697085 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f66480f51a0 msgr2=0x7f66480f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:56:33.407 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:33.405+0000 7f664c872700 1 --2- 172.21.15.137:0/1117697085 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f66480f51a0 0x7f66480f55c0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7f6638009fb0 tx=0x7f6638009b30).stop 2023-11-15T01:56:33.407 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:33.405+0000 7f664c872700 1 -- 172.21.15.137:0/1117697085 shutdown_connections 2023-11-15T01:56:33.407 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:33.405+0000 7f664c872700 1 --2- 172.21.15.137:0/1117697085 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f66480f51a0 0x7f66480f55c0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:56:33.407 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:33.405+0000 7f664c872700 1 -- 172.21.15.137:0/1117697085 >> 172.21.15.137:0/1117697085 conn(0x7f66480f09a0 msgr2=0x7f66480f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:56:33.407 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:33.406+0000 7f664c872700 1 -- 172.21.15.137:0/1117697085 shutdown_connections 2023-11-15T01:56:33.408 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:33.406+0000 7f664c872700 1 -- 172.21.15.137:0/1117697085 wait complete. 2023-11-15T01:56:33.409 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:33.406+0000 7f664c872700 1 Processor -- start 2023-11-15T01:56:33.409 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:33.406+0000 7f664c872700 1 -- start start 2023-11-15T01:56:33.410 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:33.407+0000 7f664c872700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6648060200 0x7f6648109340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:56:33.410 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:33.407+0000 7f664c872700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6648109830 con 0x7f6648060200 2023-11-15T01:56:33.410 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:33.407+0000 7f6646d9d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6648060200 0x7f6648109340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:56:33.410 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:33.407+0000 7f6646d9d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6648060200 0x7f6648109340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:44448/0 (socket says 172.21.15.137:44448) 2023-11-15T01:56:33.410 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:33.407+0000 7f6646d9d700 1 -- 172.21.15.137:0/488394782 learned_addr learned my addr 172.21.15.137:0/488394782 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:56:33.411 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:33.407+0000 7f6646d9d700 1 -- 172.21.15.137:0/488394782 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6638005040 con 0x7f6648060200 2023-11-15T01:56:33.411 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:33.407+0000 7f6646d9d700 1 --2- 172.21.15.137:0/488394782 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6648060200 0x7f6648109340 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7f6638009f80 tx=0x7f6638009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:56:33.411 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:33.407+0000 7f662f7fe700 1 -- 172.21.15.137:0/488394782 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f66380136c0 con 0x7f6648060200 2023-11-15T01:56:33.411 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:33.408+0000 7f664c872700 1 -- 172.21.15.137:0/488394782 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6648109970 con 0x7f6648060200 2023-11-15T01:56:33.411 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:33.408+0000 7f662f7fe700 1 -- 172.21.15.137:0/488394782 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f66380095f0 con 0x7f6648060200 2023-11-15T01:56:33.412 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:33.408+0000 7f662f7fe700 1 -- 172.21.15.137:0/488394782 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6638014420 con 0x7f6648060200 2023-11-15T01:56:33.412 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:33.408+0000 7f664c872700 1 -- 172.21.15.137:0/488394782 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6648109da0 con 0x7f6648060200 2023-11-15T01:56:33.412 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:33.408+0000 7f662f7fe700 1 -- 172.21.15.137:0/488394782 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6638025060 con 0x7f6648060200 2023-11-15T01:56:33.412 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:33.408+0000 7f664c872700 1 -- 172.21.15.137:0/488394782 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6648059100 con 0x7f6648060200 2023-11-15T01:56:33.412 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:33.408+0000 7f662f7fe700 1 --2- 172.21.15.137:0/488394782 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f663002ff60 0x7f6630032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:56:33.413 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:33.408+0000 7f662f7fe700 1 -- 172.21.15.137:0/488394782 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6638040da0 con 0x7f6648060200 2023-11-15T01:56:33.413 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:33.409+0000 7f664659c700 1 --2- 172.21.15.137:0/488394782 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f663002ff60 0x7f6630032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:56:33.413 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:33.409+0000 7f664659c700 1 --2- 172.21.15.137:0/488394782 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f663002ff60 0x7f6630032420 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f663c0096a0 tx=0x7f663c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:56:33.413 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:33.413+0000 7f662f7fe700 1 -- 172.21.15.137:0/488394782 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f663801c950 con 0x7f6648060200 2023-11-15T01:56:33.619 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:33.619+0000 7f664c872700 1 -- 172.21.15.137:0/488394782 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6648045ad0 con 0x7f6648060200 2023-11-15T01:56:33.620 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:33.619+0000 7f662f7fe700 1 -- 172.21.15.137:0/488394782 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f663801cb60 con 0x7f6648060200 2023-11-15T01:56:33.621 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:56:33.621 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:56:33.622 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:33.621+0000 7f664c872700 1 -- 172.21.15.137:0/488394782 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f663002ff60 msgr2=0x7f6630032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:56:33.622 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:33.621+0000 7f664c872700 1 --2- 172.21.15.137:0/488394782 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f663002ff60 0x7f6630032420 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f663c0096a0 tx=0x7f663c006b40).stop 2023-11-15T01:56:33.622 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:33.621+0000 7f664c872700 1 -- 172.21.15.137:0/488394782 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6648060200 msgr2=0x7f6648109340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:56:33.623 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:33.621+0000 7f664c872700 1 --2- 172.21.15.137:0/488394782 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6648060200 0x7f6648109340 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7f6638009f80 tx=0x7f6638009b30).stop 2023-11-15T01:56:33.623 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:33.621+0000 7f664c872700 1 -- 172.21.15.137:0/488394782 shutdown_connections 2023-11-15T01:56:33.623 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:33.621+0000 7f664c872700 1 --2- 172.21.15.137:0/488394782 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f663002ff60 0x7f6630032420 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:56:33.623 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:33.621+0000 7f664c872700 1 --2- 172.21.15.137:0/488394782 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6648060200 0x7f6648109340 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:56:33.623 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:33.621+0000 7f664c872700 1 -- 172.21.15.137:0/488394782 >> 172.21.15.137:0/488394782 conn(0x7f66480f09a0 msgr2=0x7f66480f1500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:56:33.624 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:33.621+0000 7f664c872700 1 -- 172.21.15.137:0/488394782 shutdown_connections 2023-11-15T01:56:33.624 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:33.621+0000 7f664c872700 1 -- 172.21.15.137:0/488394782 wait complete. 2023-11-15T01:56:33.624 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:56:34.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:56:33 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:56:32.755697+0000 mgr.smithi055.szwngw (mgr.14164) 211 : cluster [DBG] pgmap v199: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:56:34.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:56:33 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:56:33.620584+0000 mon.smithi055 (mon.0) 275 : audit [DBG] from='client.? 172.21.15.137:0/488394782' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:56:35.010 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:56:35.011 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:56:36.129 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:36.128+0000 7f5721c5f700 1 -- 172.21.15.137:0/3234526158 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f571c0f6480 msgr2=0x7f571c0f68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:56:36.129 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:36.128+0000 7f5721c5f700 1 --2- 172.21.15.137:0/3234526158 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f571c0f6480 0x7f571c0f68a0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7f570c009fb0 tx=0x7f570c009b30).stop 2023-11-15T01:56:36.130 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:36.128+0000 7f5721c5f700 1 -- 172.21.15.137:0/3234526158 shutdown_connections 2023-11-15T01:56:36.130 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:36.128+0000 7f5721c5f700 1 --2- 172.21.15.137:0/3234526158 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f571c0f6480 0x7f571c0f68a0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:56:36.130 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:36.128+0000 7f5721c5f700 1 -- 172.21.15.137:0/3234526158 >> 172.21.15.137:0/3234526158 conn(0x7f571c0f1c00 msgr2=0x7f571c0f4020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:56:36.130 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:36.128+0000 7f5721c5f700 1 -- 172.21.15.137:0/3234526158 shutdown_connections 2023-11-15T01:56:36.130 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:36.128+0000 7f5721c5f700 1 -- 172.21.15.137:0/3234526158 wait complete. 2023-11-15T01:56:36.131 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:36.128+0000 7f5721c5f700 1 Processor -- start 2023-11-15T01:56:36.131 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:36.128+0000 7f5721c5f700 1 -- start start 2023-11-15T01:56:36.131 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:36.129+0000 7f5721c5f700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f571c0f6480 0x7f571c1066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:56:36.131 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:36.129+0000 7f5721c5f700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f571c106bc0 con 0x7f571c0f6480 2023-11-15T01:56:36.133 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:36.129+0000 7f571b7fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f571c0f6480 0x7f571c1066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:56:36.133 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:36.129+0000 7f571b7fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f571c0f6480 0x7f571c1066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:44460/0 (socket says 172.21.15.137:44460) 2023-11-15T01:56:36.133 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:36.129+0000 7f571b7fe700 1 -- 172.21.15.137:0/3070414396 learned_addr learned my addr 172.21.15.137:0/3070414396 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:56:36.134 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:36.129+0000 7f571b7fe700 1 -- 172.21.15.137:0/3070414396 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f570c005040 con 0x7f571c0f6480 2023-11-15T01:56:36.134 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:36.129+0000 7f571b7fe700 1 --2- 172.21.15.137:0/3070414396 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f571c0f6480 0x7f571c1066d0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7f570c013510 tx=0x7f570c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:56:36.134 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:36.129+0000 7f5718ff9700 1 -- 172.21.15.137:0/3070414396 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f570c009320 con 0x7f571c0f6480 2023-11-15T01:56:36.134 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:36.130+0000 7f5721c5f700 1 -- 172.21.15.137:0/3070414396 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f571c106dc0 con 0x7f571c0f6480 2023-11-15T01:56:36.134 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:36.130+0000 7f5718ff9700 1 -- 172.21.15.137:0/3070414396 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f570c013f00 con 0x7f571c0f6480 2023-11-15T01:56:36.135 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:36.130+0000 7f5718ff9700 1 -- 172.21.15.137:0/3070414396 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f570c014d90 con 0x7f571c0f6480 2023-11-15T01:56:36.135 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:36.130+0000 7f5721c5f700 1 -- 172.21.15.137:0/3070414396 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f571c107250 con 0x7f571c0f6480 2023-11-15T01:56:36.135 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:36.130+0000 7f5718ff9700 1 -- 172.21.15.137:0/3070414396 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f570c0263a0 con 0x7f571c0f6480 2023-11-15T01:56:36.135 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:36.130+0000 7f5718ff9700 1 --2- 172.21.15.137:0/3070414396 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f570802ff60 0x7f5708032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:56:36.135 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:36.131+0000 7f5718ff9700 1 -- 172.21.15.137:0/3070414396 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f570c041010 con 0x7f571c0f6480 2023-11-15T01:56:36.136 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:36.131+0000 7f5721c5f700 1 -- 172.21.15.137:0/3070414396 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f571c045ad0 con 0x7f571c0f6480 2023-11-15T01:56:36.136 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:36.131+0000 7f5713fff700 1 --2- 172.21.15.137:0/3070414396 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f570802ff60 0x7f5708032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:56:36.136 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:36.131+0000 7f5713fff700 1 --2- 172.21.15.137:0/3070414396 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f570802ff60 0x7f5708032420 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f57040096a0 tx=0x7f5704006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:56:36.136 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:36.136+0000 7f5718ff9700 1 -- 172.21.15.137:0/3070414396 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f570c012920 con 0x7f571c0f6480 2023-11-15T01:56:36.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:56:35 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:56:34.755978+0000 mgr.smithi055.szwngw (mgr.14164) 212 : cluster [DBG] pgmap v200: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:56:36.344 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:36.343+0000 7f5721c5f700 1 -- 172.21.15.137:0/3070414396 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f571c107f70 con 0x7f571c0f6480 2023-11-15T01:56:36.344 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:36.344+0000 7f5718ff9700 1 -- 172.21.15.137:0/3070414396 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f570c02b020 con 0x7f571c0f6480 2023-11-15T01:56:36.345 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:56:36.345 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:56:36.346 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:36.345+0000 7f5721c5f700 1 -- 172.21.15.137:0/3070414396 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f570802ff60 msgr2=0x7f5708032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:56:36.346 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:36.345+0000 7f5721c5f700 1 --2- 172.21.15.137:0/3070414396 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f570802ff60 0x7f5708032420 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f57040096a0 tx=0x7f5704006b40).stop 2023-11-15T01:56:36.346 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:36.345+0000 7f5721c5f700 1 -- 172.21.15.137:0/3070414396 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f571c0f6480 msgr2=0x7f571c1066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:56:36.347 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:36.345+0000 7f5721c5f700 1 --2- 172.21.15.137:0/3070414396 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f571c0f6480 0x7f571c1066d0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7f570c013510 tx=0x7f570c00b040).stop 2023-11-15T01:56:36.347 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:36.345+0000 7f5721c5f700 1 -- 172.21.15.137:0/3070414396 shutdown_connections 2023-11-15T01:56:36.347 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:36.345+0000 7f5721c5f700 1 --2- 172.21.15.137:0/3070414396 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f570802ff60 0x7f5708032420 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:56:36.347 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:36.345+0000 7f5721c5f700 1 --2- 172.21.15.137:0/3070414396 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f571c0f6480 0x7f571c1066d0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:56:36.347 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:36.345+0000 7f5721c5f700 1 -- 172.21.15.137:0/3070414396 >> 172.21.15.137:0/3070414396 conn(0x7f571c0f1c00 msgr2=0x7f571c0f2720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:56:36.348 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:36.345+0000 7f5721c5f700 1 -- 172.21.15.137:0/3070414396 shutdown_connections 2023-11-15T01:56:36.348 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:36.345+0000 7f5721c5f700 1 -- 172.21.15.137:0/3070414396 wait complete. 2023-11-15T01:56:36.348 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:56:37.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:56:36 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:56:36.344986+0000 mon.smithi055 (mon.0) 276 : audit [DBG] from='client.? 172.21.15.137:0/3070414396' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:56:37.728 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:56:37.728 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:56:38.327 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:56:37 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:56:36.756278+0000 mgr.smithi055.szwngw (mgr.14164) 213 : cluster [DBG] pgmap v201: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:56:38.884 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:38.882+0000 7f633af9d700 1 -- 172.21.15.137:0/2513635792 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f63340f5810 msgr2=0x7f63340f5c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:56:38.884 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:38.882+0000 7f633af9d700 1 --2- 172.21.15.137:0/2513635792 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f63340f5810 0x7f63340f5c30 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7f6330009fb0 tx=0x7f6330009b30).stop 2023-11-15T01:56:38.884 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:38.882+0000 7f633af9d700 1 -- 172.21.15.137:0/2513635792 shutdown_connections 2023-11-15T01:56:38.884 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:38.882+0000 7f633af9d700 1 --2- 172.21.15.137:0/2513635792 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f63340f5810 0x7f63340f5c30 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:56:38.885 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:38.882+0000 7f633af9d700 1 -- 172.21.15.137:0/2513635792 >> 172.21.15.137:0/2513635792 conn(0x7f63340f0120 msgr2=0x7f63340f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:56:38.885 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:38.882+0000 7f633af9d700 1 -- 172.21.15.137:0/2513635792 shutdown_connections 2023-11-15T01:56:38.885 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:38.882+0000 7f633af9d700 1 -- 172.21.15.137:0/2513635792 wait complete. 2023-11-15T01:56:38.885 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:38.883+0000 7f633af9d700 1 Processor -- start 2023-11-15T01:56:38.885 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:38.883+0000 7f633af9d700 1 -- start start 2023-11-15T01:56:38.886 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:38.883+0000 7f633af9d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f63340f5810 0x7f6334109200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:56:38.886 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:38.883+0000 7f633af9d700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63340f51d0 con 0x7f63340f5810 2023-11-15T01:56:38.886 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:38.883+0000 7f6339f9b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f63340f5810 0x7f6334109200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:56:38.886 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:38.883+0000 7f6339f9b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f63340f5810 0x7f6334109200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:37076/0 (socket says 172.21.15.137:37076) 2023-11-15T01:56:38.886 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:38.883+0000 7f6339f9b700 1 -- 172.21.15.137:0/2112945416 learned_addr learned my addr 172.21.15.137:0/2112945416 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:56:38.890 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:38.884+0000 7f6339f9b700 1 -- 172.21.15.137:0/2112945416 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6330005040 con 0x7f63340f5810 2023-11-15T01:56:38.891 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:38.884+0000 7f6339f9b700 1 --2- 172.21.15.137:0/2112945416 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f63340f5810 0x7f6334109200 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f6330004dd0 tx=0x7f6330009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:56:38.891 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:38.884+0000 7f63267fc700 1 -- 172.21.15.137:0/2112945416 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6330018070 con 0x7f63340f5810 2023-11-15T01:56:38.891 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:38.884+0000 7f633af9d700 1 -- 172.21.15.137:0/2112945416 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63340f53d0 con 0x7f63340f5810 2023-11-15T01:56:38.891 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:38.884+0000 7f63267fc700 1 -- 172.21.15.137:0/2112945416 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6330013410 con 0x7f63340f5810 2023-11-15T01:56:38.891 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:38.884+0000 7f63267fc700 1 -- 172.21.15.137:0/2112945416 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f633001c3c0 con 0x7f63340f5810 2023-11-15T01:56:38.892 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:38.884+0000 7f633af9d700 1 -- 172.21.15.137:0/2112945416 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63341098b0 con 0x7f63340f5810 2023-11-15T01:56:38.892 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:38.885+0000 7f63267fc700 1 -- 172.21.15.137:0/2112945416 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f633001d420 con 0x7f63340f5810 2023-11-15T01:56:38.892 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:38.885+0000 7f633af9d700 1 -- 172.21.15.137:0/2112945416 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6334059100 con 0x7f63340f5810 2023-11-15T01:56:38.892 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:38.885+0000 7f63267fc700 1 --2- 172.21.15.137:0/2112945416 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f632002ff60 0x7f6320032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:56:38.892 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:38.885+0000 7f63267fc700 1 -- 172.21.15.137:0/2112945416 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6330041330 con 0x7f63340f5810 2023-11-15T01:56:38.892 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:38.885+0000 7f633979a700 1 --2- 172.21.15.137:0/2112945416 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f632002ff60 0x7f6320032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:56:38.893 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:38.886+0000 7f633979a700 1 --2- 172.21.15.137:0/2112945416 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f632002ff60 0x7f6320032420 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f63280096a0 tx=0x7f6328006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:56:38.893 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:38.890+0000 7f63267fc700 1 -- 172.21.15.137:0/2112945416 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f633001d7e0 con 0x7f63340f5810 2023-11-15T01:56:39.098 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:39.098+0000 7f633af9d700 1 -- 172.21.15.137:0/2112945416 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6334045ad0 con 0x7f63340f5810 2023-11-15T01:56:39.099 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:39.099+0000 7f63267fc700 1 -- 172.21.15.137:0/2112945416 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f633001b370 con 0x7f63340f5810 2023-11-15T01:56:39.099 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:56:39.100 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:56:39.101 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:39.100+0000 7f633af9d700 1 -- 172.21.15.137:0/2112945416 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f632002ff60 msgr2=0x7f6320032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:56:39.101 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:39.100+0000 7f633af9d700 1 --2- 172.21.15.137:0/2112945416 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f632002ff60 0x7f6320032420 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f63280096a0 tx=0x7f6328006b40).stop 2023-11-15T01:56:39.101 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:39.100+0000 7f633af9d700 1 -- 172.21.15.137:0/2112945416 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f63340f5810 msgr2=0x7f6334109200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:56:39.101 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:39.100+0000 7f633af9d700 1 --2- 172.21.15.137:0/2112945416 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f63340f5810 0x7f6334109200 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f6330004dd0 tx=0x7f6330009b30).stop 2023-11-15T01:56:39.101 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:39.100+0000 7f633af9d700 1 -- 172.21.15.137:0/2112945416 shutdown_connections 2023-11-15T01:56:39.102 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:39.100+0000 7f633af9d700 1 --2- 172.21.15.137:0/2112945416 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f632002ff60 0x7f6320032420 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:56:39.102 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:39.100+0000 7f633af9d700 1 --2- 172.21.15.137:0/2112945416 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f63340f5810 0x7f6334109200 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:56:39.102 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:39.100+0000 7f633af9d700 1 -- 172.21.15.137:0/2112945416 >> 172.21.15.137:0/2112945416 conn(0x7f63340f0120 msgr2=0x7f63340f0bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:56:39.102 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:39.100+0000 7f633af9d700 1 -- 172.21.15.137:0/2112945416 shutdown_connections 2023-11-15T01:56:39.102 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:39.100+0000 7f633af9d700 1 -- 172.21.15.137:0/2112945416 wait complete. 2023-11-15T01:56:39.103 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:56:40.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:56:39 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:56:38.756640+0000 mgr.smithi055.szwngw (mgr.14164) 214 : cluster [DBG] pgmap v202: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:56:40.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:56:39 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:56:39.099763+0000 mon.smithi055 (mon.0) 277 : audit [DBG] from='client.? 172.21.15.137:0/2112945416' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:56:40.498 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:56:40.498 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:56:41.655 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:41.654+0000 7f8f0ecad700 1 -- 172.21.15.137:0/2389398646 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8f080f5410 msgr2=0x7f8f080f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:56:41.656 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:41.654+0000 7f8f0ecad700 1 --2- 172.21.15.137:0/2389398646 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8f080f5410 0x7f8f080f5830 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7f8efc009fb0 tx=0x7f8efc009b30).stop 2023-11-15T01:56:41.656 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:41.655+0000 7f8f0ecad700 1 -- 172.21.15.137:0/2389398646 shutdown_connections 2023-11-15T01:56:41.656 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:41.655+0000 7f8f0ecad700 1 --2- 172.21.15.137:0/2389398646 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8f080f5410 0x7f8f080f5830 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:56:41.657 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:41.655+0000 7f8f0ecad700 1 -- 172.21.15.137:0/2389398646 >> 172.21.15.137:0/2389398646 conn(0x7f8f080f0b70 msgr2=0x7f8f080f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:56:41.657 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:41.655+0000 7f8f0ecad700 1 -- 172.21.15.137:0/2389398646 shutdown_connections 2023-11-15T01:56:41.657 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:41.655+0000 7f8f0ecad700 1 -- 172.21.15.137:0/2389398646 wait complete. 2023-11-15T01:56:41.657 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:41.655+0000 7f8f0ecad700 1 Processor -- start 2023-11-15T01:56:41.657 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:41.655+0000 7f8f0ecad700 1 -- start start 2023-11-15T01:56:41.660 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:41.655+0000 7f8f0ecad700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8f080f5410 0x7f8f08105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:56:41.660 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:41.655+0000 7f8f0ecad700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f08105b20 con 0x7f8f080f5410 2023-11-15T01:56:41.660 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:41.656+0000 7f8f0ca49700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8f080f5410 0x7f8f08105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:56:41.660 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:41.656+0000 7f8f0ca49700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8f080f5410 0x7f8f08105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:37090/0 (socket says 172.21.15.137:37090) 2023-11-15T01:56:41.661 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:41.656+0000 7f8f0ca49700 1 -- 172.21.15.137:0/3213473445 learned_addr learned my addr 172.21.15.137:0/3213473445 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:56:41.661 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:41.656+0000 7f8f0ca49700 1 -- 172.21.15.137:0/3213473445 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8efc005040 con 0x7f8f080f5410 2023-11-15T01:56:41.661 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:41.656+0000 7f8f0ca49700 1 --2- 172.21.15.137:0/3213473445 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8f080f5410 0x7f8f08105630 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f8efc013510 tx=0x7f8efc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:56:41.661 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:41.656+0000 7f8f057fa700 1 -- 172.21.15.137:0/3213473445 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8efc009320 con 0x7f8f080f5410 2023-11-15T01:56:41.661 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:41.656+0000 7f8f0ecad700 1 -- 172.21.15.137:0/3213473445 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f08105d20 con 0x7f8f080f5410 2023-11-15T01:56:41.662 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:41.656+0000 7f8f057fa700 1 -- 172.21.15.137:0/3213473445 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8efc013f00 con 0x7f8f080f5410 2023-11-15T01:56:41.662 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:41.656+0000 7f8f057fa700 1 -- 172.21.15.137:0/3213473445 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8efc014d90 con 0x7f8f080f5410 2023-11-15T01:56:41.662 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:41.656+0000 7f8f0ecad700 1 -- 172.21.15.137:0/3213473445 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f081061b0 con 0x7f8f080f5410 2023-11-15T01:56:41.662 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:41.657+0000 7f8f057fa700 1 -- 172.21.15.137:0/3213473445 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8efc0095f0 con 0x7f8f080f5410 2023-11-15T01:56:41.662 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:41.657+0000 7f8f0ecad700 1 -- 172.21.15.137:0/3213473445 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f08045ad0 con 0x7f8f080f5410 2023-11-15T01:56:41.663 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:41.657+0000 7f8f057fa700 1 --2- 172.21.15.137:0/3213473445 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f8ef802ff60 0x7f8ef8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:56:41.663 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:41.657+0000 7f8f057fa700 1 -- 172.21.15.137:0/3213473445 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8efc040fe0 con 0x7f8f080f5410 2023-11-15T01:56:41.663 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:41.658+0000 7f8f07fff700 1 --2- 172.21.15.137:0/3213473445 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f8ef802ff60 0x7f8ef8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:56:41.663 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:41.658+0000 7f8f07fff700 1 --2- 172.21.15.137:0/3213473445 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f8ef802ff60 0x7f8ef8032420 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f8ef40096a0 tx=0x7f8ef4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:56:41.664 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:41.662+0000 7f8f057fa700 1 -- 172.21.15.137:0/3213473445 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8efc014420 con 0x7f8f080f5410 2023-11-15T01:56:41.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:56:41 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:56:40.757012+0000 mgr.smithi055.szwngw (mgr.14164) 215 : cluster [DBG] pgmap v203: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:56:41.868 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:41.867+0000 7f8f0ecad700 1 -- 172.21.15.137:0/3213473445 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8f080fae60 con 0x7f8f080f5410 2023-11-15T01:56:41.869 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:41.868+0000 7f8f057fa700 1 -- 172.21.15.137:0/3213473445 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8efc010230 con 0x7f8f080f5410 2023-11-15T01:56:41.870 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:56:41.870 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:56:41.871 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:41.869+0000 7f8f0ecad700 1 -- 172.21.15.137:0/3213473445 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f8ef802ff60 msgr2=0x7f8ef8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:56:41.872 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:41.869+0000 7f8f0ecad700 1 --2- 172.21.15.137:0/3213473445 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f8ef802ff60 0x7f8ef8032420 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f8ef40096a0 tx=0x7f8ef4006b40).stop 2023-11-15T01:56:41.872 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:41.869+0000 7f8f0ecad700 1 -- 172.21.15.137:0/3213473445 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8f080f5410 msgr2=0x7f8f08105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:56:41.873 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:41.869+0000 7f8f0ecad700 1 --2- 172.21.15.137:0/3213473445 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8f080f5410 0x7f8f08105630 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f8efc013510 tx=0x7f8efc00b040).stop 2023-11-15T01:56:41.873 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:41.869+0000 7f8f0ecad700 1 -- 172.21.15.137:0/3213473445 shutdown_connections 2023-11-15T01:56:41.874 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:41.869+0000 7f8f0ecad700 1 --2- 172.21.15.137:0/3213473445 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f8ef802ff60 0x7f8ef8032420 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:56:41.875 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:41.869+0000 7f8f0ecad700 1 --2- 172.21.15.137:0/3213473445 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8f080f5410 0x7f8f08105630 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:56:41.875 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:41.869+0000 7f8f0ecad700 1 -- 172.21.15.137:0/3213473445 >> 172.21.15.137:0/3213473445 conn(0x7f8f080f0b70 msgr2=0x7f8f080f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:56:41.876 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:41.870+0000 7f8f0ecad700 1 -- 172.21.15.137:0/3213473445 shutdown_connections 2023-11-15T01:56:41.876 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:41.870+0000 7f8f0ecad700 1 -- 172.21.15.137:0/3213473445 wait complete. 2023-11-15T01:56:41.876 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:56:42.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:56:42 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:56:41.869023+0000 mon.smithi055 (mon.0) 278 : audit [DBG] from='client.? 172.21.15.137:0/3213473445' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:56:43.256 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:56:43.257 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:56:43.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:56:43 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:56:42.757250+0000 mgr.smithi055.szwngw (mgr.14164) 216 : cluster [DBG] pgmap v204: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:56:43.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:56:43 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:56:42.854577+0000 mon.smithi055 (mon.0) 279 : audit [INF] from='mgr.14164 172.21.15.55:0/59463371' entity='mgr.smithi055.szwngw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi055.szwngw/mirror_snapshot_schedule"}]: dispatch 2023-11-15T01:56:43.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:56:43 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:56:42.857064+0000 mon.smithi055 (mon.0) 280 : audit [INF] from='mgr.14164 172.21.15.55:0/59463371' entity='mgr.smithi055.szwngw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi055.szwngw/trash_purge_schedule"}]: dispatch 2023-11-15T01:56:44.388 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:44.386+0000 7ff63836e700 1 -- 172.21.15.137:0/1152817677 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff630060280 msgr2=0x7ff6300fb4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:56:44.389 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:44.386+0000 7ff63836e700 1 --2- 172.21.15.137:0/1152817677 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff630060280 0x7ff6300fb4f0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7ff620009fb0 tx=0x7ff620009b30).stop 2023-11-15T01:56:44.389 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:44.386+0000 7ff63836e700 1 -- 172.21.15.137:0/1152817677 shutdown_connections 2023-11-15T01:56:44.389 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:44.386+0000 7ff63836e700 1 --2- 172.21.15.137:0/1152817677 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff630060280 0x7ff6300fb4f0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:56:44.389 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:44.386+0000 7ff63836e700 1 -- 172.21.15.137:0/1152817677 >> 172.21.15.137:0/1152817677 conn(0x7ff6300f0b70 msgr2=0x7ff6300f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:56:44.390 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:44.386+0000 7ff63836e700 1 -- 172.21.15.137:0/1152817677 shutdown_connections 2023-11-15T01:56:44.390 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:44.386+0000 7ff63836e700 1 -- 172.21.15.137:0/1152817677 wait complete. 2023-11-15T01:56:44.390 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:44.387+0000 7ff63836e700 1 Processor -- start 2023-11-15T01:56:44.390 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:44.387+0000 7ff63836e700 1 -- start start 2023-11-15T01:56:44.390 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:44.387+0000 7ff63836e700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff630060280 0x7ff6301088e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:56:44.391 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:44.387+0000 7ff63836e700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff630108dd0 con 0x7ff630060280 2023-11-15T01:56:44.391 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:44.387+0000 7ff63610a700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff630060280 0x7ff6301088e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:56:44.391 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:44.387+0000 7ff63610a700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff630060280 0x7ff6301088e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:37102/0 (socket says 172.21.15.137:37102) 2023-11-15T01:56:44.391 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:44.387+0000 7ff63610a700 1 -- 172.21.15.137:0/1757932249 learned_addr learned my addr 172.21.15.137:0/1757932249 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:56:44.391 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:44.387+0000 7ff63610a700 1 -- 172.21.15.137:0/1757932249 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff620005040 con 0x7ff630060280 2023-11-15T01:56:44.392 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:44.388+0000 7ff63610a700 1 --2- 172.21.15.137:0/1757932249 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff630060280 0x7ff6301088e0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7ff620013510 tx=0x7ff62000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:56:44.392 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:44.388+0000 7ff626ffd700 1 -- 172.21.15.137:0/1757932249 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff620009320 con 0x7ff630060280 2023-11-15T01:56:44.392 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:44.388+0000 7ff63836e700 1 -- 172.21.15.137:0/1757932249 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff630108fd0 con 0x7ff630060280 2023-11-15T01:56:44.392 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:44.388+0000 7ff626ffd700 1 -- 172.21.15.137:0/1757932249 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff620013f00 con 0x7ff630060280 2023-11-15T01:56:44.392 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:44.388+0000 7ff63836e700 1 -- 172.21.15.137:0/1757932249 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff630109460 con 0x7ff630060280 2023-11-15T01:56:44.393 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:44.388+0000 7ff626ffd700 1 -- 172.21.15.137:0/1757932249 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff620014dc0 con 0x7ff630060280 2023-11-15T01:56:44.394 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:44.389+0000 7ff63836e700 1 -- 172.21.15.137:0/1757932249 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff630045ad0 con 0x7ff630060280 2023-11-15T01:56:44.394 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:44.389+0000 7ff626ffd700 1 -- 172.21.15.137:0/1757932249 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff6200095f0 con 0x7ff630060280 2023-11-15T01:56:44.394 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:44.389+0000 7ff626ffd700 1 --2- 172.21.15.137:0/1757932249 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7ff61c02ff60 0x7ff61c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:56:44.394 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:44.389+0000 7ff626ffd700 1 -- 172.21.15.137:0/1757932249 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff620041030 con 0x7ff630060280 2023-11-15T01:56:44.394 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:44.389+0000 7ff635909700 1 --2- 172.21.15.137:0/1757932249 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7ff61c02ff60 0x7ff61c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:56:44.395 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:44.390+0000 7ff635909700 1 --2- 172.21.15.137:0/1757932249 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7ff61c02ff60 0x7ff61c032420 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7ff62c0096a0 tx=0x7ff62c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:56:44.395 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:44.394+0000 7ff626ffd700 1 -- 172.21.15.137:0/1757932249 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff62001c790 con 0x7ff630060280 2023-11-15T01:56:44.601 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:44.601+0000 7ff63836e700 1 -- 172.21.15.137:0/1757932249 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff63010a2b0 con 0x7ff630060280 2023-11-15T01:56:44.602 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:44.601+0000 7ff626ffd700 1 -- 172.21.15.137:0/1757932249 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff620010230 con 0x7ff630060280 2023-11-15T01:56:44.602 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:56:44.603 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:56:44.603 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:44.603+0000 7ff63836e700 1 -- 172.21.15.137:0/1757932249 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7ff61c02ff60 msgr2=0x7ff61c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:56:44.604 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:44.603+0000 7ff63836e700 1 --2- 172.21.15.137:0/1757932249 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7ff61c02ff60 0x7ff61c032420 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7ff62c0096a0 tx=0x7ff62c006b40).stop 2023-11-15T01:56:44.604 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:44.603+0000 7ff63836e700 1 -- 172.21.15.137:0/1757932249 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff630060280 msgr2=0x7ff6301088e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:56:44.604 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:44.603+0000 7ff63836e700 1 --2- 172.21.15.137:0/1757932249 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff630060280 0x7ff6301088e0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7ff620013510 tx=0x7ff62000b040).stop 2023-11-15T01:56:44.604 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:44.603+0000 7ff63836e700 1 -- 172.21.15.137:0/1757932249 shutdown_connections 2023-11-15T01:56:44.604 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:44.603+0000 7ff63836e700 1 --2- 172.21.15.137:0/1757932249 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7ff61c02ff60 0x7ff61c032420 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:56:44.605 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:44.603+0000 7ff63836e700 1 --2- 172.21.15.137:0/1757932249 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff630060280 0x7ff6301088e0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:56:44.605 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:44.603+0000 7ff63836e700 1 -- 172.21.15.137:0/1757932249 >> 172.21.15.137:0/1757932249 conn(0x7ff6300f0b70 msgr2=0x7ff6300f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:56:44.605 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:44.603+0000 7ff63836e700 1 -- 172.21.15.137:0/1757932249 shutdown_connections 2023-11-15T01:56:44.605 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:44.603+0000 7ff63836e700 1 -- 172.21.15.137:0/1757932249 wait complete. 2023-11-15T01:56:44.606 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:56:45.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:56:44 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:56:44.602406+0000 mon.smithi055 (mon.0) 281 : audit [DBG] from='client.? 172.21.15.137:0/1757932249' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:56:45.992 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:56:45.993 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:56:46.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:56:45 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:56:44.757581+0000 mgr.smithi055.szwngw (mgr.14164) 217 : cluster [DBG] pgmap v205: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:56:47.188 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:47.188+0000 7f4bfed42700 1 -- 172.21.15.137:0/2370097312 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4bf80f6510 msgr2=0x7f4bf80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:56:47.189 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:47.188+0000 7f4bfed42700 1 --2- 172.21.15.137:0/2370097312 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4bf80f6510 0x7f4bf80f6930 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7f4bec009fb0 tx=0x7f4bec009b30).stop 2023-11-15T01:56:47.189 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:47.188+0000 7f4bfed42700 1 -- 172.21.15.137:0/2370097312 shutdown_connections 2023-11-15T01:56:47.189 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:47.188+0000 7f4bfed42700 1 --2- 172.21.15.137:0/2370097312 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4bf80f6510 0x7f4bf80f6930 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:56:47.189 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:47.188+0000 7f4bfed42700 1 -- 172.21.15.137:0/2370097312 >> 172.21.15.137:0/2370097312 conn(0x7f4bf80f1c70 msgr2=0x7f4bf80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:56:47.189 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:47.188+0000 7f4bfed42700 1 -- 172.21.15.137:0/2370097312 shutdown_connections 2023-11-15T01:56:47.190 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:47.188+0000 7f4bfed42700 1 -- 172.21.15.137:0/2370097312 wait complete. 2023-11-15T01:56:47.192 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:47.188+0000 7f4bfed42700 1 Processor -- start 2023-11-15T01:56:47.192 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:47.188+0000 7f4bfed42700 1 -- start start 2023-11-15T01:56:47.192 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:47.189+0000 7f4bfed42700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4bf80f6510 0x7f4bf806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:56:47.192 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:47.189+0000 7f4bfed42700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bf8069e20 con 0x7f4bf80f6510 2023-11-15T01:56:47.193 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:47.189+0000 7f4bfcade700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4bf80f6510 0x7f4bf806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:56:47.193 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:47.189+0000 7f4bfcade700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4bf80f6510 0x7f4bf806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:37126/0 (socket says 172.21.15.137:37126) 2023-11-15T01:56:47.193 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:47.189+0000 7f4bfcade700 1 -- 172.21.15.137:0/248345071 learned_addr learned my addr 172.21.15.137:0/248345071 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:56:47.193 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:47.189+0000 7f4bfcade700 1 -- 172.21.15.137:0/248345071 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bec005040 con 0x7f4bf80f6510 2023-11-15T01:56:47.193 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:47.189+0000 7f4bfcade700 1 --2- 172.21.15.137:0/248345071 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4bf80f6510 0x7f4bf806b9a0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f4bec013510 tx=0x7f4bec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:56:47.193 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:47.189+0000 7f4bf57fa700 1 -- 172.21.15.137:0/248345071 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4bec009320 con 0x7f4bf80f6510 2023-11-15T01:56:47.194 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:47.190+0000 7f4bf57fa700 1 -- 172.21.15.137:0/248345071 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4bec013f00 con 0x7f4bf80f6510 2023-11-15T01:56:47.194 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:47.190+0000 7f4bfed42700 1 -- 172.21.15.137:0/248345071 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bf8069fc0 con 0x7f4bf80f6510 2023-11-15T01:56:47.194 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:47.190+0000 7f4bf57fa700 1 -- 172.21.15.137:0/248345071 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4bec014d90 con 0x7f4bf80f6510 2023-11-15T01:56:47.194 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:47.190+0000 7f4bfed42700 1 -- 172.21.15.137:0/248345071 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bf806a450 con 0x7f4bf80f6510 2023-11-15T01:56:47.194 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:47.190+0000 7f4bf57fa700 1 -- 172.21.15.137:0/248345071 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4bec0095f0 con 0x7f4bf80f6510 2023-11-15T01:56:47.195 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:47.190+0000 7f4bfed42700 1 -- 172.21.15.137:0/248345071 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bf8059100 con 0x7f4bf80f6510 2023-11-15T01:56:47.195 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:47.191+0000 7f4bf57fa700 1 --2- 172.21.15.137:0/248345071 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f4be802ffb0 0x7f4be8032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:56:47.195 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:47.191+0000 7f4bf57fa700 1 -- 172.21.15.137:0/248345071 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4bec040f90 con 0x7f4bf80f6510 2023-11-15T01:56:47.196 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:47.191+0000 7f4bf7fff700 1 --2- 172.21.15.137:0/248345071 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f4be802ffb0 0x7f4be8032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:56:47.196 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:47.191+0000 7f4bf7fff700 1 --2- 172.21.15.137:0/248345071 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f4be802ffb0 0x7f4be8032470 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f4be40096a0 tx=0x7f4be4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:56:47.196 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:47.195+0000 7f4bf57fa700 1 -- 172.21.15.137:0/248345071 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4bec014420 con 0x7f4bf80f6510 2023-11-15T01:56:47.400 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:47.400+0000 7f4bfed42700 1 -- 172.21.15.137:0/248345071 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4bf80fbf60 con 0x7f4bf80f6510 2023-11-15T01:56:47.401 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:47.401+0000 7f4bf57fa700 1 -- 172.21.15.137:0/248345071 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4bec01d7f0 con 0x7f4bf80f6510 2023-11-15T01:56:47.402 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:56:47.402 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:56:47.403 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:47.402+0000 7f4bfed42700 1 -- 172.21.15.137:0/248345071 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f4be802ffb0 msgr2=0x7f4be8032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:56:47.403 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:47.402+0000 7f4bfed42700 1 --2- 172.21.15.137:0/248345071 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f4be802ffb0 0x7f4be8032470 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f4be40096a0 tx=0x7f4be4006b40).stop 2023-11-15T01:56:47.403 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:47.402+0000 7f4bfed42700 1 -- 172.21.15.137:0/248345071 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4bf80f6510 msgr2=0x7f4bf806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:56:47.404 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:47.402+0000 7f4bfed42700 1 --2- 172.21.15.137:0/248345071 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4bf80f6510 0x7f4bf806b9a0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f4bec013510 tx=0x7f4bec00b040).stop 2023-11-15T01:56:47.404 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:47.402+0000 7f4bfed42700 1 -- 172.21.15.137:0/248345071 shutdown_connections 2023-11-15T01:56:47.404 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:47.402+0000 7f4bfed42700 1 --2- 172.21.15.137:0/248345071 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f4be802ffb0 0x7f4be8032470 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:56:47.404 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:47.402+0000 7f4bfed42700 1 --2- 172.21.15.137:0/248345071 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4bf80f6510 0x7f4bf806b9a0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:56:47.405 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:47.402+0000 7f4bfed42700 1 -- 172.21.15.137:0/248345071 >> 172.21.15.137:0/248345071 conn(0x7f4bf80f1c70 msgr2=0x7f4bf80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:56:47.405 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:47.402+0000 7f4bfed42700 1 -- 172.21.15.137:0/248345071 shutdown_connections 2023-11-15T01:56:47.405 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:47.402+0000 7f4bfed42700 1 -- 172.21.15.137:0/248345071 wait complete. 2023-11-15T01:56:47.406 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:56:48.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:56:47 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:56:46.758028+0000 mgr.smithi055.szwngw (mgr.14164) 218 : cluster [DBG] pgmap v206: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:56:48.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:56:47 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:56:47.401807+0000 mon.smithi055 (mon.0) 282 : audit [DBG] from='client.? 172.21.15.137:0/248345071' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:56:48.791 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:56:48.792 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:56:49.912 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:49.911+0000 7f2bcc69d700 1 -- 172.21.15.137:0/4236717441 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2bc40f5410 msgr2=0x7f2bc40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:56:49.913 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:49.911+0000 7f2bcc69d700 1 --2- 172.21.15.137:0/4236717441 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2bc40f5410 0x7f2bc40f5830 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7f2bb4009fb0 tx=0x7f2bb4009b30).stop 2023-11-15T01:56:49.913 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:49.911+0000 7f2bcc69d700 1 -- 172.21.15.137:0/4236717441 shutdown_connections 2023-11-15T01:56:49.913 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:49.911+0000 7f2bcc69d700 1 --2- 172.21.15.137:0/4236717441 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2bc40f5410 0x7f2bc40f5830 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:56:49.913 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:49.911+0000 7f2bcc69d700 1 -- 172.21.15.137:0/4236717441 >> 172.21.15.137:0/4236717441 conn(0x7f2bc40f0b70 msgr2=0x7f2bc40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:56:49.914 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:49.911+0000 7f2bcc69d700 1 -- 172.21.15.137:0/4236717441 shutdown_connections 2023-11-15T01:56:49.914 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:49.911+0000 7f2bcc69d700 1 -- 172.21.15.137:0/4236717441 wait complete. 2023-11-15T01:56:49.914 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:49.912+0000 7f2bcc69d700 1 Processor -- start 2023-11-15T01:56:49.914 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:49.912+0000 7f2bcc69d700 1 -- start start 2023-11-15T01:56:49.915 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:49.912+0000 7f2bcc69d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2bc40f5410 0x7f2bc4106710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:56:49.915 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:49.912+0000 7f2bcc69d700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bc4106c00 con 0x7f2bc40f5410 2023-11-15T01:56:49.915 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:49.912+0000 7f2bca439700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2bc40f5410 0x7f2bc4106710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:56:49.915 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:49.912+0000 7f2bca439700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2bc40f5410 0x7f2bc4106710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:44552/0 (socket says 172.21.15.137:44552) 2023-11-15T01:56:49.915 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:49.912+0000 7f2bca439700 1 -- 172.21.15.137:0/3746206063 learned_addr learned my addr 172.21.15.137:0/3746206063 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:56:49.917 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:49.912+0000 7f2bca439700 1 -- 172.21.15.137:0/3746206063 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bb4005040 con 0x7f2bc40f5410 2023-11-15T01:56:49.917 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:49.913+0000 7f2bca439700 1 --2- 172.21.15.137:0/3746206063 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2bc40f5410 0x7f2bc4106710 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f2bb4013510 tx=0x7f2bb400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:56:49.917 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:49.913+0000 7f2bbaffd700 1 -- 172.21.15.137:0/3746206063 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2bb4009320 con 0x7f2bc40f5410 2023-11-15T01:56:49.917 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:49.913+0000 7f2bcc69d700 1 -- 172.21.15.137:0/3746206063 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bc4106e00 con 0x7f2bc40f5410 2023-11-15T01:56:49.918 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:49.913+0000 7f2bbaffd700 1 -- 172.21.15.137:0/3746206063 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2bb4013f00 con 0x7f2bc40f5410 2023-11-15T01:56:49.918 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:49.913+0000 7f2bbaffd700 1 -- 172.21.15.137:0/3746206063 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2bb4014d90 con 0x7f2bc40f5410 2023-11-15T01:56:49.918 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:49.913+0000 7f2bcc69d700 1 -- 172.21.15.137:0/3746206063 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bc4107290 con 0x7f2bc40f5410 2023-11-15T01:56:49.918 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:49.914+0000 7f2bcc69d700 1 -- 172.21.15.137:0/3746206063 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bc4045ad0 con 0x7f2bc40f5410 2023-11-15T01:56:49.919 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:49.914+0000 7f2bbaffd700 1 -- 172.21.15.137:0/3746206063 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2bb40095f0 con 0x7f2bc40f5410 2023-11-15T01:56:49.919 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:49.914+0000 7f2bbaffd700 1 --2- 172.21.15.137:0/3746206063 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f2bb002ff60 0x7f2bb0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:56:49.919 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:49.914+0000 7f2bbaffd700 1 -- 172.21.15.137:0/3746206063 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2bb4040fe0 con 0x7f2bc40f5410 2023-11-15T01:56:49.919 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:49.914+0000 7f2bc9c38700 1 --2- 172.21.15.137:0/3746206063 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f2bb002ff60 0x7f2bb0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:56:49.920 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:49.915+0000 7f2bc9c38700 1 --2- 172.21.15.137:0/3746206063 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f2bb002ff60 0x7f2bb0032420 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f2bc00096a0 tx=0x7f2bc0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:56:49.920 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:49.919+0000 7f2bbaffd700 1 -- 172.21.15.137:0/3746206063 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2bb401d7f0 con 0x7f2bc40f5410 2023-11-15T01:56:50.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:56:49 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:56:48.758351+0000 mgr.smithi055.szwngw (mgr.14164) 219 : cluster [DBG] pgmap v207: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:56:50.126 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:50.126+0000 7f2bcc69d700 1 -- 172.21.15.137:0/3746206063 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2bc40fae60 con 0x7f2bc40f5410 2023-11-15T01:56:50.127 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:50.126+0000 7f2bbaffd700 1 -- 172.21.15.137:0/3746206063 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2bb4010230 con 0x7f2bc40f5410 2023-11-15T01:56:50.127 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:56:50.128 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:56:50.129 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:50.128+0000 7f2bcc69d700 1 -- 172.21.15.137:0/3746206063 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f2bb002ff60 msgr2=0x7f2bb0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:56:50.129 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:50.128+0000 7f2bcc69d700 1 --2- 172.21.15.137:0/3746206063 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f2bb002ff60 0x7f2bb0032420 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f2bc00096a0 tx=0x7f2bc0006b40).stop 2023-11-15T01:56:50.129 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:50.128+0000 7f2bcc69d700 1 -- 172.21.15.137:0/3746206063 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2bc40f5410 msgr2=0x7f2bc4106710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:56:50.129 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:50.128+0000 7f2bcc69d700 1 --2- 172.21.15.137:0/3746206063 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2bc40f5410 0x7f2bc4106710 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f2bb4013510 tx=0x7f2bb400b040).stop 2023-11-15T01:56:50.129 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:50.128+0000 7f2bcc69d700 1 -- 172.21.15.137:0/3746206063 shutdown_connections 2023-11-15T01:56:50.130 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:50.128+0000 7f2bcc69d700 1 --2- 172.21.15.137:0/3746206063 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f2bb002ff60 0x7f2bb0032420 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:56:50.130 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:50.128+0000 7f2bcc69d700 1 --2- 172.21.15.137:0/3746206063 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2bc40f5410 0x7f2bc4106710 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:56:50.130 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:50.128+0000 7f2bcc69d700 1 -- 172.21.15.137:0/3746206063 >> 172.21.15.137:0/3746206063 conn(0x7f2bc40f0b70 msgr2=0x7f2bc405fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:56:50.130 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:50.128+0000 7f2bcc69d700 1 -- 172.21.15.137:0/3746206063 shutdown_connections 2023-11-15T01:56:50.131 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:50.128+0000 7f2bcc69d700 1 -- 172.21.15.137:0/3746206063 wait complete. 2023-11-15T01:56:50.131 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:56:51.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:56:50 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:56:50.127343+0000 mon.smithi055 (mon.0) 283 : audit [DBG] from='client.? 172.21.15.137:0/3746206063' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:56:51.516 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:56:51.517 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:56:52.327 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:56:51 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:56:50.758759+0000 mgr.smithi055.szwngw (mgr.14164) 220 : cluster [DBG] pgmap v208: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:56:52.721 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:52.719+0000 7f68f6b98700 1 -- 172.21.15.137:0/2479530449 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f68f0060280 msgr2=0x7f68f00f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:56:52.721 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:52.719+0000 7f68f6b98700 1 --2- 172.21.15.137:0/2479530449 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f68f0060280 0x7f68f00f90f0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7f68e4009fb0 tx=0x7f68e4009b30).stop 2023-11-15T01:56:52.722 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:52.719+0000 7f68f6b98700 1 -- 172.21.15.137:0/2479530449 shutdown_connections 2023-11-15T01:56:52.722 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:52.719+0000 7f68f6b98700 1 --2- 172.21.15.137:0/2479530449 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f68f0060280 0x7f68f00f90f0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:56:52.722 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:52.719+0000 7f68f6b98700 1 -- 172.21.15.137:0/2479530449 >> 172.21.15.137:0/2479530449 conn(0x7f68f00f0b70 msgr2=0x7f68f00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:56:52.722 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:52.719+0000 7f68f6b98700 1 -- 172.21.15.137:0/2479530449 shutdown_connections 2023-11-15T01:56:52.722 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:52.719+0000 7f68f6b98700 1 -- 172.21.15.137:0/2479530449 wait complete. 2023-11-15T01:56:52.723 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:52.720+0000 7f68f6b98700 1 Processor -- start 2023-11-15T01:56:52.723 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:52.720+0000 7f68f6b98700 1 -- start start 2023-11-15T01:56:52.723 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:52.720+0000 7f68f6b98700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f68f0060280 0x7f68f01066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:56:52.723 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:52.720+0000 7f68f6b98700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68f0106bd0 con 0x7f68f0060280 2023-11-15T01:56:52.723 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:52.720+0000 7f68f4934700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f68f0060280 0x7f68f01066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:56:52.724 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:52.720+0000 7f68f4934700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f68f0060280 0x7f68f01066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:44580/0 (socket says 172.21.15.137:44580) 2023-11-15T01:56:52.724 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:52.720+0000 7f68f4934700 1 -- 172.21.15.137:0/3215603072 learned_addr learned my addr 172.21.15.137:0/3215603072 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:56:52.725 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:52.721+0000 7f68f4934700 1 -- 172.21.15.137:0/3215603072 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68e4005040 con 0x7f68f0060280 2023-11-15T01:56:52.726 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:52.721+0000 7f68f4934700 1 --2- 172.21.15.137:0/3215603072 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f68f0060280 0x7f68f01066e0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f68e4013510 tx=0x7f68e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:56:52.726 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:52.721+0000 7f68ed7fa700 1 -- 172.21.15.137:0/3215603072 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f68e4009320 con 0x7f68f0060280 2023-11-15T01:56:52.726 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:52.721+0000 7f68ed7fa700 1 -- 172.21.15.137:0/3215603072 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f68e4013f00 con 0x7f68f0060280 2023-11-15T01:56:52.726 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:52.721+0000 7f68ed7fa700 1 -- 172.21.15.137:0/3215603072 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f68e4014dc0 con 0x7f68f0060280 2023-11-15T01:56:52.726 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:52.721+0000 7f68f6b98700 1 -- 172.21.15.137:0/3215603072 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68f0106dd0 con 0x7f68f0060280 2023-11-15T01:56:52.727 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:52.721+0000 7f68f6b98700 1 -- 172.21.15.137:0/3215603072 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68f0107260 con 0x7f68f0060280 2023-11-15T01:56:52.727 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:52.722+0000 7f68ed7fa700 1 -- 172.21.15.137:0/3215603072 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f68e401d3d0 con 0x7f68f0060280 2023-11-15T01:56:52.727 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:52.722+0000 7f68ed7fa700 1 --2- 172.21.15.137:0/3215603072 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f68e002ff60 0x7f68e0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:56:52.727 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:52.722+0000 7f68ed7fa700 1 -- 172.21.15.137:0/3215603072 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f68e4041100 con 0x7f68f0060280 2023-11-15T01:56:52.727 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:52.722+0000 7f68f6b98700 1 -- 172.21.15.137:0/3215603072 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68f0045ad0 con 0x7f68f0060280 2023-11-15T01:56:52.728 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:52.722+0000 7f68effff700 1 --2- 172.21.15.137:0/3215603072 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f68e002ff60 0x7f68e0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:56:52.728 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:52.723+0000 7f68effff700 1 --2- 172.21.15.137:0/3215603072 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f68e002ff60 0x7f68e0032420 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f68dc0096a0 tx=0x7f68dc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:56:52.728 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:52.727+0000 7f68ed7fa700 1 -- 172.21.15.137:0/3215603072 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f68e4012920 con 0x7f68f0060280 2023-11-15T01:56:52.931 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:52.931+0000 7f68f6b98700 1 -- 172.21.15.137:0/3215603072 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f68f0107f80 con 0x7f68f0060280 2023-11-15T01:56:52.932 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:52.931+0000 7f68ed7fa700 1 -- 172.21.15.137:0/3215603072 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f68e402b020 con 0x7f68f0060280 2023-11-15T01:56:52.932 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:56:52.933 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:56:52.933 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:52.933+0000 7f68f6b98700 1 -- 172.21.15.137:0/3215603072 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f68e002ff60 msgr2=0x7f68e0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:56:52.934 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:52.933+0000 7f68f6b98700 1 --2- 172.21.15.137:0/3215603072 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f68e002ff60 0x7f68e0032420 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f68dc0096a0 tx=0x7f68dc006b40).stop 2023-11-15T01:56:52.934 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:52.933+0000 7f68f6b98700 1 -- 172.21.15.137:0/3215603072 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f68f0060280 msgr2=0x7f68f01066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:56:52.934 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:52.933+0000 7f68f6b98700 1 --2- 172.21.15.137:0/3215603072 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f68f0060280 0x7f68f01066e0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f68e4013510 tx=0x7f68e400b040).stop 2023-11-15T01:56:52.934 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:52.933+0000 7f68f6b98700 1 -- 172.21.15.137:0/3215603072 shutdown_connections 2023-11-15T01:56:52.935 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:52.933+0000 7f68f6b98700 1 --2- 172.21.15.137:0/3215603072 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f68e002ff60 0x7f68e0032420 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:56:52.935 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:52.933+0000 7f68f6b98700 1 --2- 172.21.15.137:0/3215603072 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f68f0060280 0x7f68f01066e0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:56:52.935 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:52.933+0000 7f68f6b98700 1 -- 172.21.15.137:0/3215603072 >> 172.21.15.137:0/3215603072 conn(0x7f68f00f0b70 msgr2=0x7f68f00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:56:52.935 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:52.933+0000 7f68f6b98700 1 -- 172.21.15.137:0/3215603072 shutdown_connections 2023-11-15T01:56:52.936 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:52.933+0000 7f68f6b98700 1 -- 172.21.15.137:0/3215603072 wait complete. 2023-11-15T01:56:52.936 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:56:54.324 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:56:54.325 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:56:54.327 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:56:53 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:56:52.759056+0000 mgr.smithi055.szwngw (mgr.14164) 221 : cluster [DBG] pgmap v209: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:56:54.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:56:53 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:56:52.932513+0000 mon.smithi055 (mon.0) 284 : audit [DBG] from='client.? 172.21.15.137:0/3215603072' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:56:55.454 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:55.449+0000 7fa69b99e700 1 -- 172.21.15.137:0/15709818 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa694060280 msgr2=0x7fa6940f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:56:55.455 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:55.449+0000 7fa69b99e700 1 --2- 172.21.15.137:0/15709818 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa694060280 0x7fa6940f90f0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7fa684009fb0 tx=0x7fa684009b30).stop 2023-11-15T01:56:55.455 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:55.449+0000 7fa69b99e700 1 -- 172.21.15.137:0/15709818 shutdown_connections 2023-11-15T01:56:55.456 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:55.450+0000 7fa69b99e700 1 --2- 172.21.15.137:0/15709818 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa694060280 0x7fa6940f90f0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:56:55.456 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:55.450+0000 7fa69b99e700 1 -- 172.21.15.137:0/15709818 >> 172.21.15.137:0/15709818 conn(0x7fa6940f0b70 msgr2=0x7fa6940f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:56:55.456 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:55.450+0000 7fa69b99e700 1 -- 172.21.15.137:0/15709818 shutdown_connections 2023-11-15T01:56:55.457 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:55.450+0000 7fa69b99e700 1 -- 172.21.15.137:0/15709818 wait complete. 2023-11-15T01:56:55.457 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:55.450+0000 7fa69b99e700 1 Processor -- start 2023-11-15T01:56:55.457 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:55.450+0000 7fa69b99e700 1 -- start start 2023-11-15T01:56:55.458 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:55.450+0000 7fa69b99e700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa694060280 0x7fa6941066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:56:55.458 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:55.450+0000 7fa69b99e700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa694106bd0 con 0x7fa694060280 2023-11-15T01:56:55.458 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:55.450+0000 7fa69973a700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa694060280 0x7fa6941066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:56:55.459 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:55.451+0000 7fa69973a700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa694060280 0x7fa6941066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:44594/0 (socket says 172.21.15.137:44594) 2023-11-15T01:56:55.459 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:55.451+0000 7fa69973a700 1 -- 172.21.15.137:0/299583743 learned_addr learned my addr 172.21.15.137:0/299583743 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:56:55.459 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:55.451+0000 7fa69973a700 1 -- 172.21.15.137:0/299583743 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa684005040 con 0x7fa694060280 2023-11-15T01:56:55.459 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:55.451+0000 7fa69973a700 1 --2- 172.21.15.137:0/299583743 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa694060280 0x7fa6941066e0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7fa684013510 tx=0x7fa68400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:56:55.460 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:55.451+0000 7fa689ffb700 1 -- 172.21.15.137:0/299583743 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa684009320 con 0x7fa694060280 2023-11-15T01:56:55.460 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:55.451+0000 7fa69b99e700 1 -- 172.21.15.137:0/299583743 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa694106dd0 con 0x7fa694060280 2023-11-15T01:56:55.460 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:55.451+0000 7fa689ffb700 1 -- 172.21.15.137:0/299583743 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa684013f00 con 0x7fa694060280 2023-11-15T01:56:55.460 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:55.451+0000 7fa689ffb700 1 -- 172.21.15.137:0/299583743 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa684014dc0 con 0x7fa694060280 2023-11-15T01:56:55.460 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:55.451+0000 7fa69b99e700 1 -- 172.21.15.137:0/299583743 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa694107260 con 0x7fa694060280 2023-11-15T01:56:55.461 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:55.452+0000 7fa689ffb700 1 -- 172.21.15.137:0/299583743 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa6840095f0 con 0x7fa694060280 2023-11-15T01:56:55.461 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:55.452+0000 7fa69b99e700 1 -- 172.21.15.137:0/299583743 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa694045ad0 con 0x7fa694060280 2023-11-15T01:56:55.461 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:55.452+0000 7fa689ffb700 1 --2- 172.21.15.137:0/299583743 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fa68002ff60 0x7fa680032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:56:55.461 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:55.452+0000 7fa689ffb700 1 -- 172.21.15.137:0/299583743 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa684041100 con 0x7fa694060280 2023-11-15T01:56:55.462 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:55.453+0000 7fa698f39700 1 --2- 172.21.15.137:0/299583743 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fa68002ff60 0x7fa680032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:56:55.462 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:55.453+0000 7fa698f39700 1 --2- 172.21.15.137:0/299583743 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fa68002ff60 0x7fa680032420 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fa6900096a0 tx=0x7fa690006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:56:55.462 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:55.457+0000 7fa689ffb700 1 -- 172.21.15.137:0/299583743 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa684014420 con 0x7fa694060280 2023-11-15T01:56:55.668 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:55.668+0000 7fa69b99e700 1 -- 172.21.15.137:0/299583743 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa694108110 con 0x7fa694060280 2023-11-15T01:56:55.669 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:55.668+0000 7fa689ffb700 1 -- 172.21.15.137:0/299583743 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa684010230 con 0x7fa694060280 2023-11-15T01:56:55.670 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:56:55.670 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:56:55.671 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:55.670+0000 7fa69b99e700 1 -- 172.21.15.137:0/299583743 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fa68002ff60 msgr2=0x7fa680032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:56:55.672 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:55.670+0000 7fa69b99e700 1 --2- 172.21.15.137:0/299583743 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fa68002ff60 0x7fa680032420 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fa6900096a0 tx=0x7fa690006b40).stop 2023-11-15T01:56:55.672 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:55.670+0000 7fa69b99e700 1 -- 172.21.15.137:0/299583743 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa694060280 msgr2=0x7fa6941066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:56:55.672 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:55.670+0000 7fa69b99e700 1 --2- 172.21.15.137:0/299583743 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa694060280 0x7fa6941066e0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7fa684013510 tx=0x7fa68400b040).stop 2023-11-15T01:56:55.672 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:55.670+0000 7fa69b99e700 1 -- 172.21.15.137:0/299583743 shutdown_connections 2023-11-15T01:56:55.672 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:55.670+0000 7fa69b99e700 1 --2- 172.21.15.137:0/299583743 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fa68002ff60 0x7fa680032420 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:56:55.673 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:55.670+0000 7fa69b99e700 1 --2- 172.21.15.137:0/299583743 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa694060280 0x7fa6941066e0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:56:55.673 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:55.670+0000 7fa69b99e700 1 -- 172.21.15.137:0/299583743 >> 172.21.15.137:0/299583743 conn(0x7fa6940f0b70 msgr2=0x7fa6940f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:56:55.673 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:55.670+0000 7fa69b99e700 1 -- 172.21.15.137:0/299583743 shutdown_connections 2023-11-15T01:56:55.673 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:55.670+0000 7fa69b99e700 1 -- 172.21.15.137:0/299583743 wait complete. 2023-11-15T01:56:55.674 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:56:56.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:56:55 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:56:54.759404+0000 mgr.smithi055.szwngw (mgr.14164) 222 : cluster [DBG] pgmap v210: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:56:56.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:56:55 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:56:55.669467+0000 mon.smithi055 (mon.0) 285 : audit [DBG] from='client.? 172.21.15.137:0/299583743' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:56:57.064 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:56:57.064 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:56:58.211 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:58.208+0000 7f9161df1700 1 -- 172.21.15.137:0/405253176 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f915c0601e0 msgr2=0x7f915c0f9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:56:58.211 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:58.208+0000 7f9161df1700 1 --2- 172.21.15.137:0/405253176 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f915c0601e0 0x7f915c0f9190 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7f914c009fb0 tx=0x7f914c009b30).stop 2023-11-15T01:56:58.211 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:58.208+0000 7f9161df1700 1 -- 172.21.15.137:0/405253176 shutdown_connections 2023-11-15T01:56:58.212 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:58.208+0000 7f9161df1700 1 --2- 172.21.15.137:0/405253176 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f915c0601e0 0x7f915c0f9190 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:56:58.212 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:58.208+0000 7f9161df1700 1 -- 172.21.15.137:0/405253176 >> 172.21.15.137:0/405253176 conn(0x7f915c0f0b70 msgr2=0x7f915c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:56:58.212 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:58.208+0000 7f9161df1700 1 -- 172.21.15.137:0/405253176 shutdown_connections 2023-11-15T01:56:58.212 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:58.208+0000 7f9161df1700 1 -- 172.21.15.137:0/405253176 wait complete. 2023-11-15T01:56:58.212 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:58.209+0000 7f9161df1700 1 Processor -- start 2023-11-15T01:56:58.213 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:58.209+0000 7f9161df1700 1 -- start start 2023-11-15T01:56:58.213 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:58.209+0000 7f9161df1700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f915c0601e0 0x7f915c1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:56:58.213 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:58.209+0000 7f9161df1700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f915c106bd0 con 0x7f915c0601e0 2023-11-15T01:56:58.213 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:58.209+0000 7f915b7fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f915c0601e0 0x7f915c1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:56:58.214 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:58.209+0000 7f915b7fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f915c0601e0 0x7f915c1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:44606/0 (socket says 172.21.15.137:44606) 2023-11-15T01:56:58.214 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:58.209+0000 7f915b7fe700 1 -- 172.21.15.137:0/3959989693 learned_addr learned my addr 172.21.15.137:0/3959989693 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:56:58.215 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:58.210+0000 7f915b7fe700 1 -- 172.21.15.137:0/3959989693 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f914c005040 con 0x7f915c0601e0 2023-11-15T01:56:58.216 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:58.210+0000 7f915b7fe700 1 --2- 172.21.15.137:0/3959989693 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f915c0601e0 0x7f915c1066e0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7f914c004dd0 tx=0x7f914c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:56:58.216 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:58.210+0000 7f9143fff700 1 -- 172.21.15.137:0/3959989693 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f914c018070 con 0x7f915c0601e0 2023-11-15T01:56:58.216 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:58.210+0000 7f9161df1700 1 -- 172.21.15.137:0/3959989693 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f915c106dd0 con 0x7f915c0601e0 2023-11-15T01:56:58.216 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:58.210+0000 7f9143fff700 1 -- 172.21.15.137:0/3959989693 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f914c013410 con 0x7f915c0601e0 2023-11-15T01:56:58.217 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:58.210+0000 7f9161df1700 1 -- 172.21.15.137:0/3959989693 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f915c107260 con 0x7f915c0601e0 2023-11-15T01:56:58.217 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:58.210+0000 7f9143fff700 1 -- 172.21.15.137:0/3959989693 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f914c01c3c0 con 0x7f915c0601e0 2023-11-15T01:56:58.217 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:58.211+0000 7f9143fff700 1 -- 172.21.15.137:0/3959989693 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f914c01d420 con 0x7f915c0601e0 2023-11-15T01:56:58.217 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:58.211+0000 7f9161df1700 1 -- 172.21.15.137:0/3959989693 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f915c045ad0 con 0x7f915c0601e0 2023-11-15T01:56:58.217 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:58.211+0000 7f9143fff700 1 --2- 172.21.15.137:0/3959989693 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f914402ff60 0x7f9144032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:56:58.218 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:58.211+0000 7f9143fff700 1 -- 172.21.15.137:0/3959989693 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f914c0413d0 con 0x7f915c0601e0 2023-11-15T01:56:58.218 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:58.211+0000 7f915affd700 1 --2- 172.21.15.137:0/3959989693 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f914402ff60 0x7f9144032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:56:58.218 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:58.212+0000 7f915affd700 1 --2- 172.21.15.137:0/3959989693 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f914402ff60 0x7f9144032420 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f91500096a0 tx=0x7f9150006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:56:58.219 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:58.216+0000 7f9143fff700 1 -- 172.21.15.137:0/3959989693 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f914c01d820 con 0x7f915c0601e0 2023-11-15T01:56:58.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:56:57 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:56:56.759735+0000 mgr.smithi055.szwngw (mgr.14164) 223 : cluster [DBG] pgmap v211: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:56:58.426 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:58.424+0000 7f9161df1700 1 -- 172.21.15.137:0/3959989693 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f915c1080b0 con 0x7f915c0601e0 2023-11-15T01:56:58.427 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:58.426+0000 7f9143fff700 1 -- 172.21.15.137:0/3959989693 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f914c01b370 con 0x7f915c0601e0 2023-11-15T01:56:58.427 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:56:58.428 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:56:58.429 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:58.427+0000 7f9161df1700 1 -- 172.21.15.137:0/3959989693 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f914402ff60 msgr2=0x7f9144032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:56:58.429 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:58.427+0000 7f9161df1700 1 --2- 172.21.15.137:0/3959989693 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f914402ff60 0x7f9144032420 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f91500096a0 tx=0x7f9150006b40).stop 2023-11-15T01:56:58.429 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:58.427+0000 7f9161df1700 1 -- 172.21.15.137:0/3959989693 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f915c0601e0 msgr2=0x7f915c1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:56:58.429 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:58.427+0000 7f9161df1700 1 --2- 172.21.15.137:0/3959989693 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f915c0601e0 0x7f915c1066e0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7f914c004dd0 tx=0x7f914c009b30).stop 2023-11-15T01:56:58.430 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:58.427+0000 7f9161df1700 1 -- 172.21.15.137:0/3959989693 shutdown_connections 2023-11-15T01:56:58.430 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:58.427+0000 7f9161df1700 1 --2- 172.21.15.137:0/3959989693 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f914402ff60 0x7f9144032420 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:56:58.430 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:58.427+0000 7f9161df1700 1 --2- 172.21.15.137:0/3959989693 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f915c0601e0 0x7f915c1066e0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:56:58.430 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:58.427+0000 7f9161df1700 1 -- 172.21.15.137:0/3959989693 >> 172.21.15.137:0/3959989693 conn(0x7f915c0f0b70 msgr2=0x7f915c0f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:56:58.431 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:58.427+0000 7f9161df1700 1 -- 172.21.15.137:0/3959989693 shutdown_connections 2023-11-15T01:56:58.431 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:56:58.427+0000 7f9161df1700 1 -- 172.21.15.137:0/3959989693 wait complete. 2023-11-15T01:56:58.432 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:56:59.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:56:58 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:56:58.427208+0000 mon.smithi055 (mon.0) 286 : audit [DBG] from='client.? 172.21.15.137:0/3959989693' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:56:59.810 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:56:59.810 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:57:00.327 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:56:59 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:56:58.760096+0000 mgr.smithi055.szwngw (mgr.14164) 224 : cluster [DBG] pgmap v212: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:57:00.941 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:00.939+0000 7f2f2d6c1700 1 -- 172.21.15.137:0/419421537 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2f28060280 msgr2=0x7f2f280f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:57:00.941 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:00.939+0000 7f2f2d6c1700 1 --2- 172.21.15.137:0/419421537 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2f28060280 0x7f2f280f90f0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7f2f1c009fb0 tx=0x7f2f1c009b30).stop 2023-11-15T01:57:00.942 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:00.939+0000 7f2f2d6c1700 1 -- 172.21.15.137:0/419421537 shutdown_connections 2023-11-15T01:57:00.942 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:00.939+0000 7f2f2d6c1700 1 --2- 172.21.15.137:0/419421537 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2f28060280 0x7f2f280f90f0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:57:00.942 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:00.939+0000 7f2f2d6c1700 1 -- 172.21.15.137:0/419421537 >> 172.21.15.137:0/419421537 conn(0x7f2f280f0b70 msgr2=0x7f2f280f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:57:00.942 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:00.939+0000 7f2f2d6c1700 1 -- 172.21.15.137:0/419421537 shutdown_connections 2023-11-15T01:57:00.943 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:00.939+0000 7f2f2d6c1700 1 -- 172.21.15.137:0/419421537 wait complete. 2023-11-15T01:57:00.945 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:00.940+0000 7f2f2d6c1700 1 Processor -- start 2023-11-15T01:57:00.945 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:00.940+0000 7f2f2d6c1700 1 -- start start 2023-11-15T01:57:00.945 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:00.940+0000 7f2f2d6c1700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2f28060280 0x7f2f280f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:57:00.946 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:00.940+0000 7f2f2d6c1700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f280f4ff0 con 0x7f2f28060280 2023-11-15T01:57:00.946 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:00.941+0000 7f2f26ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2f28060280 0x7f2f280f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:57:00.946 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:00.941+0000 7f2f26ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2f28060280 0x7f2f280f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:56864/0 (socket says 172.21.15.137:56864) 2023-11-15T01:57:00.946 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:00.941+0000 7f2f26ffd700 1 -- 172.21.15.137:0/2404998282 learned_addr learned my addr 172.21.15.137:0/2404998282 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:57:00.947 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:00.941+0000 7f2f26ffd700 1 -- 172.21.15.137:0/2404998282 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f1c005040 con 0x7f2f28060280 2023-11-15T01:57:00.947 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:00.941+0000 7f2f26ffd700 1 --2- 172.21.15.137:0/2404998282 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2f28060280 0x7f2f280f6b70 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f2f1c009830 tx=0x7f2f1c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:57:00.947 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:00.941+0000 7f2f0f7fe700 1 -- 172.21.15.137:0/2404998282 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2f1c009320 con 0x7f2f28060280 2023-11-15T01:57:00.947 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:00.941+0000 7f2f2d6c1700 1 -- 172.21.15.137:0/2404998282 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f280f5190 con 0x7f2f28060280 2023-11-15T01:57:00.947 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:00.941+0000 7f2f0f7fe700 1 -- 172.21.15.137:0/2404998282 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2f1c0136c0 con 0x7f2f28060280 2023-11-15T01:57:00.948 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:00.941+0000 7f2f0f7fe700 1 -- 172.21.15.137:0/2404998282 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2f1c01ee90 con 0x7f2f28060280 2023-11-15T01:57:00.948 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:00.941+0000 7f2f2d6c1700 1 -- 172.21.15.137:0/2404998282 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f280f5620 con 0x7f2f28060280 2023-11-15T01:57:00.948 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:00.942+0000 7f2f0f7fe700 1 -- 172.21.15.137:0/2404998282 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2f1c01e420 con 0x7f2f28060280 2023-11-15T01:57:00.948 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:00.942+0000 7f2f2d6c1700 1 -- 172.21.15.137:0/2404998282 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f28059100 con 0x7f2f28060280 2023-11-15T01:57:00.949 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:00.942+0000 7f2f0f7fe700 1 --2- 172.21.15.137:0/2404998282 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f2f1002ff10 0x7f2f100323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:57:00.949 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:00.942+0000 7f2f0f7fe700 1 -- 172.21.15.137:0/2404998282 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2f1c040ff0 con 0x7f2f28060280 2023-11-15T01:57:00.949 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:00.943+0000 7f2f267fc700 1 --2- 172.21.15.137:0/2404998282 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f2f1002ff10 0x7f2f100323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:57:00.949 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:00.943+0000 7f2f267fc700 1 --2- 172.21.15.137:0/2404998282 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f2f1002ff10 0x7f2f100323d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f2f180096a0 tx=0x7f2f18006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:57:00.950 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:00.947+0000 7f2f0f7fe700 1 -- 172.21.15.137:0/2404998282 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2f1c0095f0 con 0x7f2f28060280 2023-11-15T01:57:01.157 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:01.155+0000 7f2f2d6c1700 1 -- 172.21.15.137:0/2404998282 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2f28045ad0 con 0x7f2f28060280 2023-11-15T01:57:01.157 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:01.155+0000 7f2f0f7fe700 1 -- 172.21.15.137:0/2404998282 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2f1c01cb10 con 0x7f2f28060280 2023-11-15T01:57:01.157 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:57:01.158 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:57:01.158 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:01.157+0000 7f2f2d6c1700 1 -- 172.21.15.137:0/2404998282 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f2f1002ff10 msgr2=0x7f2f100323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:57:01.159 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:01.157+0000 7f2f2d6c1700 1 --2- 172.21.15.137:0/2404998282 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f2f1002ff10 0x7f2f100323d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f2f180096a0 tx=0x7f2f18006b40).stop 2023-11-15T01:57:01.159 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:01.157+0000 7f2f2d6c1700 1 -- 172.21.15.137:0/2404998282 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2f28060280 msgr2=0x7f2f280f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:57:01.159 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:01.157+0000 7f2f2d6c1700 1 --2- 172.21.15.137:0/2404998282 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2f28060280 0x7f2f280f6b70 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f2f1c009830 tx=0x7f2f1c00b040).stop 2023-11-15T01:57:01.159 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:01.157+0000 7f2f2d6c1700 1 -- 172.21.15.137:0/2404998282 shutdown_connections 2023-11-15T01:57:01.160 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:01.157+0000 7f2f2d6c1700 1 --2- 172.21.15.137:0/2404998282 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f2f1002ff10 0x7f2f100323d0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:57:01.160 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:01.157+0000 7f2f2d6c1700 1 --2- 172.21.15.137:0/2404998282 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2f28060280 0x7f2f280f6b70 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:57:01.160 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:01.157+0000 7f2f2d6c1700 1 -- 172.21.15.137:0/2404998282 >> 172.21.15.137:0/2404998282 conn(0x7f2f280f0b70 msgr2=0x7f2f280f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:57:01.160 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:01.157+0000 7f2f2d6c1700 1 -- 172.21.15.137:0/2404998282 shutdown_connections 2023-11-15T01:57:01.160 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:01.157+0000 7f2f2d6c1700 1 -- 172.21.15.137:0/2404998282 wait complete. 2023-11-15T01:57:01.161 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:57:01.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:01 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:57:00.760470+0000 mgr.smithi055.szwngw (mgr.14164) 225 : cluster [DBG] pgmap v213: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:57:01.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:01 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:57:01.157399+0000 mon.smithi055 (mon.0) 287 : audit [DBG] from='client.? 172.21.15.137:0/2404998282' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:57:02.532 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:57:02.532 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:57:03.709 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:03.706+0000 7f49beef7700 1 -- 172.21.15.137:0/3097891560 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f49b80f51a0 msgr2=0x7f49b80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:57:03.710 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:03.706+0000 7f49beef7700 1 --2- 172.21.15.137:0/3097891560 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f49b80f51a0 0x7f49b80f55c0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7f49a8009fb0 tx=0x7f49a8009b30).stop 2023-11-15T01:57:03.710 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:03.706+0000 7f49beef7700 1 -- 172.21.15.137:0/3097891560 shutdown_connections 2023-11-15T01:57:03.710 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:03.706+0000 7f49beef7700 1 --2- 172.21.15.137:0/3097891560 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f49b80f51a0 0x7f49b80f55c0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:57:03.710 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:03.706+0000 7f49beef7700 1 -- 172.21.15.137:0/3097891560 >> 172.21.15.137:0/3097891560 conn(0x7f49b80f09a0 msgr2=0x7f49b80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:57:03.711 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:03.707+0000 7f49beef7700 1 -- 172.21.15.137:0/3097891560 shutdown_connections 2023-11-15T01:57:03.711 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:03.707+0000 7f49beef7700 1 -- 172.21.15.137:0/3097891560 wait complete. 2023-11-15T01:57:03.711 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:03.707+0000 7f49beef7700 1 Processor -- start 2023-11-15T01:57:03.711 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:03.707+0000 7f49beef7700 1 -- start start 2023-11-15T01:57:03.711 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:03.707+0000 7f49beef7700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f49b80f51a0 0x7f49b81053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:57:03.712 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:03.707+0000 7f49beef7700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49b81058a0 con 0x7f49b80f51a0 2023-11-15T01:57:03.712 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:03.708+0000 7f49bdef5700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f49b80f51a0 0x7f49b81053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:57:03.712 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:03.708+0000 7f49bdef5700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f49b80f51a0 0x7f49b81053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:56876/0 (socket says 172.21.15.137:56876) 2023-11-15T01:57:03.712 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:03.708+0000 7f49bdef5700 1 -- 172.21.15.137:0/1342979495 learned_addr learned my addr 172.21.15.137:0/1342979495 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:57:03.713 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:03.708+0000 7f49bdef5700 1 -- 172.21.15.137:0/1342979495 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49a8005040 con 0x7f49b80f51a0 2023-11-15T01:57:03.713 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:03.708+0000 7f49bdef5700 1 --2- 172.21.15.137:0/1342979495 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f49b80f51a0 0x7f49b81053b0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f49a8013510 tx=0x7f49a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:57:03.714 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:03.708+0000 7f49ae7fc700 1 -- 172.21.15.137:0/1342979495 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f49a8009320 con 0x7f49b80f51a0 2023-11-15T01:57:03.714 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:03.708+0000 7f49beef7700 1 -- 172.21.15.137:0/1342979495 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49b8105aa0 con 0x7f49b80f51a0 2023-11-15T01:57:03.715 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:03.708+0000 7f49ae7fc700 1 -- 172.21.15.137:0/1342979495 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f49a8013f00 con 0x7f49b80f51a0 2023-11-15T01:57:03.715 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:03.708+0000 7f49ae7fc700 1 -- 172.21.15.137:0/1342979495 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f49a8014dc0 con 0x7f49b80f51a0 2023-11-15T01:57:03.715 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:03.708+0000 7f49beef7700 1 -- 172.21.15.137:0/1342979495 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49b8105f30 con 0x7f49b80f51a0 2023-11-15T01:57:03.715 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:03.709+0000 7f49ae7fc700 1 -- 172.21.15.137:0/1342979495 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f49a80095f0 con 0x7f49b80f51a0 2023-11-15T01:57:03.715 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:03.709+0000 7f49beef7700 1 -- 172.21.15.137:0/1342979495 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49b8045ad0 con 0x7f49b80f51a0 2023-11-15T01:57:03.716 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:03.709+0000 7f49ae7fc700 1 --2- 172.21.15.137:0/1342979495 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f49a402ff60 0x7f49a4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:57:03.716 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:03.709+0000 7f49ae7fc700 1 -- 172.21.15.137:0/1342979495 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f49a8041020 con 0x7f49b80f51a0 2023-11-15T01:57:03.716 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:03.710+0000 7f49bd6f4700 1 --2- 172.21.15.137:0/1342979495 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f49a402ff60 0x7f49a4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:57:03.716 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:03.710+0000 7f49bd6f4700 1 --2- 172.21.15.137:0/1342979495 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f49a402ff60 0x7f49a4032420 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f49b40096a0 tx=0x7f49b4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:57:03.717 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:03.715+0000 7f49ae7fc700 1 -- 172.21.15.137:0/1342979495 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f49a8014420 con 0x7f49b80f51a0 2023-11-15T01:57:03.922 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:03.920+0000 7f49beef7700 1 -- 172.21.15.137:0/1342979495 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f49b8106d60 con 0x7f49b80f51a0 2023-11-15T01:57:03.922 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:03.921+0000 7f49ae7fc700 1 -- 172.21.15.137:0/1342979495 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f49a801c920 con 0x7f49b80f51a0 2023-11-15T01:57:03.923 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:57:03.923 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:57:03.924 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:03.922+0000 7f49beef7700 1 -- 172.21.15.137:0/1342979495 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f49a402ff60 msgr2=0x7f49a4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:57:03.924 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:03.922+0000 7f49beef7700 1 --2- 172.21.15.137:0/1342979495 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f49a402ff60 0x7f49a4032420 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f49b40096a0 tx=0x7f49b4006b40).stop 2023-11-15T01:57:03.925 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:03.922+0000 7f49beef7700 1 -- 172.21.15.137:0/1342979495 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f49b80f51a0 msgr2=0x7f49b81053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:57:03.925 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:03.922+0000 7f49beef7700 1 --2- 172.21.15.137:0/1342979495 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f49b80f51a0 0x7f49b81053b0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f49a8013510 tx=0x7f49a800b040).stop 2023-11-15T01:57:03.925 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:03.922+0000 7f49beef7700 1 -- 172.21.15.137:0/1342979495 shutdown_connections 2023-11-15T01:57:03.925 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:03.922+0000 7f49beef7700 1 --2- 172.21.15.137:0/1342979495 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f49a402ff60 0x7f49a4032420 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:57:03.925 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:03.922+0000 7f49beef7700 1 --2- 172.21.15.137:0/1342979495 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f49b80f51a0 0x7f49b81053b0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:57:03.926 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:03.922+0000 7f49beef7700 1 -- 172.21.15.137:0/1342979495 >> 172.21.15.137:0/1342979495 conn(0x7f49b80f09a0 msgr2=0x7f49b80f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:57:03.926 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:03.922+0000 7f49beef7700 1 -- 172.21.15.137:0/1342979495 shutdown_connections 2023-11-15T01:57:03.926 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:03.922+0000 7f49beef7700 1 -- 172.21.15.137:0/1342979495 wait complete. 2023-11-15T01:57:03.926 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:57:04.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:03 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:57:02.760909+0000 mgr.smithi055.szwngw (mgr.14164) 226 : cluster [DBG] pgmap v214: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:57:05.310 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:57:05.311 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:57:05.327 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:04 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:57:03.922937+0000 mon.smithi055 (mon.0) 288 : audit [DBG] from='client.? 172.21.15.137:0/1342979495' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:57:06.327 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:05 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:57:04.761348+0000 mgr.smithi055.szwngw (mgr.14164) 227 : cluster [DBG] pgmap v215: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:57:06.503 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:06.500+0000 7ff55c0ec700 1 -- 172.21.15.137:0/1373037517 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff554060280 msgr2=0x7ff5540f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:57:06.503 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:06.500+0000 7ff55c0ec700 1 --2- 172.21.15.137:0/1373037517 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff554060280 0x7ff5540f90f0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7ff550009fb0 tx=0x7ff550009b30).stop 2023-11-15T01:57:06.503 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:06.500+0000 7ff55c0ec700 1 -- 172.21.15.137:0/1373037517 shutdown_connections 2023-11-15T01:57:06.503 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:06.500+0000 7ff55c0ec700 1 --2- 172.21.15.137:0/1373037517 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff554060280 0x7ff5540f90f0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:57:06.504 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:06.500+0000 7ff55c0ec700 1 -- 172.21.15.137:0/1373037517 >> 172.21.15.137:0/1373037517 conn(0x7ff5540f0b70 msgr2=0x7ff5540f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:57:06.504 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:06.500+0000 7ff55c0ec700 1 -- 172.21.15.137:0/1373037517 shutdown_connections 2023-11-15T01:57:06.504 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:06.500+0000 7ff55c0ec700 1 -- 172.21.15.137:0/1373037517 wait complete. 2023-11-15T01:57:06.504 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:06.501+0000 7ff55c0ec700 1 Processor -- start 2023-11-15T01:57:06.504 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:06.501+0000 7ff55c0ec700 1 -- start start 2023-11-15T01:57:06.505 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:06.501+0000 7ff55c0ec700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff554060280 0x7ff5541066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:57:06.505 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:06.501+0000 7ff55c0ec700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff554106bd0 con 0x7ff554060280 2023-11-15T01:57:06.505 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:06.501+0000 7ff559e88700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff554060280 0x7ff5541066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:57:06.505 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:06.501+0000 7ff559e88700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff554060280 0x7ff5541066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:56888/0 (socket says 172.21.15.137:56888) 2023-11-15T01:57:06.505 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:06.501+0000 7ff559e88700 1 -- 172.21.15.137:0/2870354580 learned_addr learned my addr 172.21.15.137:0/2870354580 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:57:06.506 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:06.502+0000 7ff559e88700 1 -- 172.21.15.137:0/2870354580 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff550005040 con 0x7ff554060280 2023-11-15T01:57:06.506 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:06.502+0000 7ff559e88700 1 --2- 172.21.15.137:0/2870354580 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff554060280 0x7ff5541066e0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7ff550013510 tx=0x7ff55000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:57:06.506 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:06.502+0000 7ff54a7fc700 1 -- 172.21.15.137:0/2870354580 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff550009320 con 0x7ff554060280 2023-11-15T01:57:06.506 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:06.502+0000 7ff54a7fc700 1 -- 172.21.15.137:0/2870354580 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff550013f00 con 0x7ff554060280 2023-11-15T01:57:06.507 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:06.502+0000 7ff55c0ec700 1 -- 172.21.15.137:0/2870354580 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff554106dd0 con 0x7ff554060280 2023-11-15T01:57:06.507 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:06.502+0000 7ff54a7fc700 1 -- 172.21.15.137:0/2870354580 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff550014dc0 con 0x7ff554060280 2023-11-15T01:57:06.507 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:06.502+0000 7ff55c0ec700 1 -- 172.21.15.137:0/2870354580 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff554107260 con 0x7ff554060280 2023-11-15T01:57:06.508 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:06.503+0000 7ff54a7fc700 1 -- 172.21.15.137:0/2870354580 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff55001d3d0 con 0x7ff554060280 2023-11-15T01:57:06.508 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:06.503+0000 7ff54a7fc700 1 --2- 172.21.15.137:0/2870354580 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7ff54002ff60 0x7ff540032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:57:06.508 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:06.503+0000 7ff55c0ec700 1 -- 172.21.15.137:0/2870354580 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff554045ad0 con 0x7ff554060280 2023-11-15T01:57:06.509 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:06.503+0000 7ff54a7fc700 1 -- 172.21.15.137:0/2870354580 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff5500410b0 con 0x7ff554060280 2023-11-15T01:57:06.509 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:06.503+0000 7ff559687700 1 --2- 172.21.15.137:0/2870354580 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7ff54002ff60 0x7ff540032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:57:06.509 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:06.504+0000 7ff559687700 1 --2- 172.21.15.137:0/2870354580 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7ff54002ff60 0x7ff540032420 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7ff5440096a0 tx=0x7ff544006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:57:06.509 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:06.508+0000 7ff54a7fc700 1 -- 172.21.15.137:0/2870354580 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff550012900 con 0x7ff554060280 2023-11-15T01:57:06.715 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:06.715+0000 7ff55c0ec700 1 -- 172.21.15.137:0/2870354580 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff554108110 con 0x7ff554060280 2023-11-15T01:57:06.716 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:06.716+0000 7ff54a7fc700 1 -- 172.21.15.137:0/2870354580 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff550012330 con 0x7ff554060280 2023-11-15T01:57:06.716 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:57:06.717 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:57:06.718 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:06.717+0000 7ff55c0ec700 1 -- 172.21.15.137:0/2870354580 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7ff54002ff60 msgr2=0x7ff540032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:57:06.719 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:06.717+0000 7ff55c0ec700 1 --2- 172.21.15.137:0/2870354580 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7ff54002ff60 0x7ff540032420 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7ff5440096a0 tx=0x7ff544006b40).stop 2023-11-15T01:57:06.719 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:06.717+0000 7ff55c0ec700 1 -- 172.21.15.137:0/2870354580 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff554060280 msgr2=0x7ff5541066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:57:06.719 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:06.717+0000 7ff55c0ec700 1 --2- 172.21.15.137:0/2870354580 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff554060280 0x7ff5541066e0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7ff550013510 tx=0x7ff55000b040).stop 2023-11-15T01:57:06.719 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:06.717+0000 7ff55c0ec700 1 -- 172.21.15.137:0/2870354580 shutdown_connections 2023-11-15T01:57:06.720 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:06.717+0000 7ff55c0ec700 1 --2- 172.21.15.137:0/2870354580 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7ff54002ff60 0x7ff540032420 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:57:06.720 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:06.717+0000 7ff55c0ec700 1 --2- 172.21.15.137:0/2870354580 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff554060280 0x7ff5541066e0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:57:06.720 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:06.717+0000 7ff55c0ec700 1 -- 172.21.15.137:0/2870354580 >> 172.21.15.137:0/2870354580 conn(0x7ff5540f0b70 msgr2=0x7ff5540f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:57:06.720 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:06.717+0000 7ff55c0ec700 1 -- 172.21.15.137:0/2870354580 shutdown_connections 2023-11-15T01:57:06.720 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:06.717+0000 7ff55c0ec700 1 -- 172.21.15.137:0/2870354580 wait complete. 2023-11-15T01:57:06.721 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:57:07.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:06 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:57:06.716600+0000 mon.smithi055 (mon.0) 289 : audit [DBG] from='client.? 172.21.15.137:0/2870354580' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:57:08.111 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:57:08.112 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:57:08.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:07 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:57:06.761770+0000 mgr.smithi055.szwngw (mgr.14164) 228 : cluster [DBG] pgmap v216: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:57:09.240 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:09.239+0000 7f5cd46c6700 1 -- 172.21.15.137:0/1998374972 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5ccc0f5410 msgr2=0x7f5ccc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:57:09.241 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:09.239+0000 7f5cd46c6700 1 --2- 172.21.15.137:0/1998374972 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5ccc0f5410 0x7f5ccc0f5830 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7f5cc8009fb0 tx=0x7f5cc8009b30).stop 2023-11-15T01:57:09.241 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:09.239+0000 7f5cd46c6700 1 -- 172.21.15.137:0/1998374972 shutdown_connections 2023-11-15T01:57:09.241 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:09.239+0000 7f5cd46c6700 1 --2- 172.21.15.137:0/1998374972 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5ccc0f5410 0x7f5ccc0f5830 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:57:09.241 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:09.239+0000 7f5cd46c6700 1 -- 172.21.15.137:0/1998374972 >> 172.21.15.137:0/1998374972 conn(0x7f5ccc0f0b70 msgr2=0x7f5ccc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:57:09.241 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:09.239+0000 7f5cd46c6700 1 -- 172.21.15.137:0/1998374972 shutdown_connections 2023-11-15T01:57:09.242 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:09.239+0000 7f5cd46c6700 1 -- 172.21.15.137:0/1998374972 wait complete. 2023-11-15T01:57:09.242 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:09.240+0000 7f5cd46c6700 1 Processor -- start 2023-11-15T01:57:09.242 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:09.240+0000 7f5cd46c6700 1 -- start start 2023-11-15T01:57:09.242 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:09.240+0000 7f5cd46c6700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5ccc0f5410 0x7f5ccc1093d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:57:09.242 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:09.240+0000 7f5cd46c6700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ccc060240 con 0x7f5ccc0f5410 2023-11-15T01:57:09.244 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:09.240+0000 7f5cd2462700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5ccc0f5410 0x7f5ccc1093d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:57:09.244 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:09.240+0000 7f5cd2462700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5ccc0f5410 0x7f5ccc1093d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:51982/0 (socket says 172.21.15.137:51982) 2023-11-15T01:57:09.244 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:09.240+0000 7f5cd2462700 1 -- 172.21.15.137:0/109346799 learned_addr learned my addr 172.21.15.137:0/109346799 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:57:09.245 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:09.240+0000 7f5cd2462700 1 -- 172.21.15.137:0/109346799 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cc8005040 con 0x7f5ccc0f5410 2023-11-15T01:57:09.245 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:09.241+0000 7f5cd2462700 1 --2- 172.21.15.137:0/109346799 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5ccc0f5410 0x7f5ccc1093d0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f5cc8013510 tx=0x7f5cc800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:57:09.245 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:09.241+0000 7f5cc2ffd700 1 -- 172.21.15.137:0/109346799 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5cc8009320 con 0x7f5ccc0f5410 2023-11-15T01:57:09.245 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:09.241+0000 7f5cd46c6700 1 -- 172.21.15.137:0/109346799 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ccc060440 con 0x7f5ccc0f5410 2023-11-15T01:57:09.245 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:09.241+0000 7f5cc2ffd700 1 -- 172.21.15.137:0/109346799 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5cc8013f00 con 0x7f5ccc0f5410 2023-11-15T01:57:09.246 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:09.241+0000 7f5cc2ffd700 1 -- 172.21.15.137:0/109346799 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5cc8014d90 con 0x7f5ccc0f5410 2023-11-15T01:57:09.246 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:09.241+0000 7f5cd46c6700 1 -- 172.21.15.137:0/109346799 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ccc109a80 con 0x7f5ccc0f5410 2023-11-15T01:57:09.246 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:09.242+0000 7f5cd46c6700 1 -- 172.21.15.137:0/109346799 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ccc045ad0 con 0x7f5ccc0f5410 2023-11-15T01:57:09.246 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:09.242+0000 7f5cc2ffd700 1 -- 172.21.15.137:0/109346799 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5cc80095f0 con 0x7f5ccc0f5410 2023-11-15T01:57:09.246 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:09.242+0000 7f5cc2ffd700 1 --2- 172.21.15.137:0/109346799 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f5cb802ff60 0x7f5cb8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:57:09.247 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:09.242+0000 7f5cc2ffd700 1 -- 172.21.15.137:0/109346799 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5cc8041080 con 0x7f5ccc0f5410 2023-11-15T01:57:09.248 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:09.242+0000 7f5cd1c61700 1 --2- 172.21.15.137:0/109346799 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f5cb802ff60 0x7f5cb8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:57:09.248 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:09.243+0000 7f5cd1c61700 1 --2- 172.21.15.137:0/109346799 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f5cb802ff60 0x7f5cb8032420 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f5cbc0096a0 tx=0x7f5cbc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:57:09.249 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:09.247+0000 7f5cc2ffd700 1 -- 172.21.15.137:0/109346799 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5cc8014420 con 0x7f5ccc0f5410 2023-11-15T01:57:09.451 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:09.450+0000 7f5cd46c6700 1 -- 172.21.15.137:0/109346799 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5ccc10a290 con 0x7f5ccc0f5410 2023-11-15T01:57:09.452 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:09.451+0000 7f5cc2ffd700 1 -- 172.21.15.137:0/109346799 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5cc8010230 con 0x7f5ccc0f5410 2023-11-15T01:57:09.452 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:57:09.453 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:57:09.454 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:09.452+0000 7f5cd46c6700 1 -- 172.21.15.137:0/109346799 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f5cb802ff60 msgr2=0x7f5cb8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:57:09.454 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:09.452+0000 7f5cd46c6700 1 --2- 172.21.15.137:0/109346799 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f5cb802ff60 0x7f5cb8032420 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f5cbc0096a0 tx=0x7f5cbc006b40).stop 2023-11-15T01:57:09.454 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:09.452+0000 7f5cd46c6700 1 -- 172.21.15.137:0/109346799 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5ccc0f5410 msgr2=0x7f5ccc1093d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:57:09.454 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:09.452+0000 7f5cd46c6700 1 --2- 172.21.15.137:0/109346799 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5ccc0f5410 0x7f5ccc1093d0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f5cc8013510 tx=0x7f5cc800b040).stop 2023-11-15T01:57:09.455 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:09.453+0000 7f5cd46c6700 1 -- 172.21.15.137:0/109346799 shutdown_connections 2023-11-15T01:57:09.455 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:09.453+0000 7f5cd46c6700 1 --2- 172.21.15.137:0/109346799 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f5cb802ff60 0x7f5cb8032420 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:57:09.455 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:09.453+0000 7f5cd46c6700 1 --2- 172.21.15.137:0/109346799 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5ccc0f5410 0x7f5ccc1093d0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:57:09.455 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:09.453+0000 7f5cd46c6700 1 -- 172.21.15.137:0/109346799 >> 172.21.15.137:0/109346799 conn(0x7f5ccc0f0b70 msgr2=0x7f5ccc0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:57:09.456 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:09.453+0000 7f5cd46c6700 1 -- 172.21.15.137:0/109346799 shutdown_connections 2023-11-15T01:57:09.456 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:09.453+0000 7f5cd46c6700 1 -- 172.21.15.137:0/109346799 wait complete. 2023-11-15T01:57:09.456 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:57:10.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:09 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:57:08.762045+0000 mgr.smithi055.szwngw (mgr.14164) 229 : cluster [DBG] pgmap v217: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:57:10.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:09 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:57:09.452197+0000 mon.smithi055 (mon.0) 290 : audit [DBG] from='client.? 172.21.15.137:0/109346799' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:57:10.830 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:57:10.830 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:57:11.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:11 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:57:10.762372+0000 mgr.smithi055.szwngw (mgr.14164) 230 : cluster [DBG] pgmap v218: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:57:11.919 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:11.917+0000 7f5ce1223700 1 -- 172.21.15.137:0/991240146 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5cdc0f5410 msgr2=0x7f5cdc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:57:11.919 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:11.917+0000 7f5ce1223700 1 --2- 172.21.15.137:0/991240146 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5cdc0f5410 0x7f5cdc0f5830 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7f5cd0009fb0 tx=0x7f5cd0009b30).stop 2023-11-15T01:57:11.920 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:11.917+0000 7f5ce1223700 1 -- 172.21.15.137:0/991240146 shutdown_connections 2023-11-15T01:57:11.920 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:11.917+0000 7f5ce1223700 1 --2- 172.21.15.137:0/991240146 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5cdc0f5410 0x7f5cdc0f5830 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:57:11.920 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:11.917+0000 7f5ce1223700 1 -- 172.21.15.137:0/991240146 >> 172.21.15.137:0/991240146 conn(0x7f5cdc0f0b70 msgr2=0x7f5cdc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:57:11.920 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:11.918+0000 7f5ce1223700 1 -- 172.21.15.137:0/991240146 shutdown_connections 2023-11-15T01:57:11.920 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:11.918+0000 7f5ce1223700 1 -- 172.21.15.137:0/991240146 wait complete. 2023-11-15T01:57:11.921 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:11.918+0000 7f5ce1223700 1 Processor -- start 2023-11-15T01:57:11.921 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:11.918+0000 7f5ce1223700 1 -- start start 2023-11-15T01:57:11.921 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:11.918+0000 7f5ce1223700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5cdc060240 0x7f5cdc109540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:57:11.921 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:11.918+0000 7f5ce1223700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cdc109a30 con 0x7f5cdc060240 2023-11-15T01:57:11.921 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:11.919+0000 7f5cdad9d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5cdc060240 0x7f5cdc109540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:57:11.922 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:11.919+0000 7f5cdad9d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5cdc060240 0x7f5cdc109540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:52002/0 (socket says 172.21.15.137:52002) 2023-11-15T01:57:11.922 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:11.919+0000 7f5cdad9d700 1 -- 172.21.15.137:0/767402955 learned_addr learned my addr 172.21.15.137:0/767402955 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:57:11.923 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:11.919+0000 7f5cdad9d700 1 -- 172.21.15.137:0/767402955 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cd0005040 con 0x7f5cdc060240 2023-11-15T01:57:11.923 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:11.919+0000 7f5cdad9d700 1 --2- 172.21.15.137:0/767402955 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5cdc060240 0x7f5cdc109540 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7f5cd0009f80 tx=0x7f5cd0009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:57:11.924 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:11.919+0000 7f5cc37fe700 1 -- 172.21.15.137:0/767402955 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5cd00136c0 con 0x7f5cdc060240 2023-11-15T01:57:11.924 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:11.919+0000 7f5cc37fe700 1 -- 172.21.15.137:0/767402955 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5cd00095f0 con 0x7f5cdc060240 2023-11-15T01:57:11.924 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:11.919+0000 7f5ce1223700 1 -- 172.21.15.137:0/767402955 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cdc109b70 con 0x7f5cdc060240 2023-11-15T01:57:11.924 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:11.919+0000 7f5cc37fe700 1 -- 172.21.15.137:0/767402955 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5cd0014420 con 0x7f5cdc060240 2023-11-15T01:57:11.925 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:11.919+0000 7f5ce1223700 1 -- 172.21.15.137:0/767402955 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cdc10a000 con 0x7f5cdc060240 2023-11-15T01:57:11.925 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:11.920+0000 7f5cc37fe700 1 -- 172.21.15.137:0/767402955 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5cd0025060 con 0x7f5cdc060240 2023-11-15T01:57:11.925 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:11.920+0000 7f5ce1223700 1 -- 172.21.15.137:0/767402955 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cdc045ad0 con 0x7f5cdc060240 2023-11-15T01:57:11.925 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:11.920+0000 7f5cc37fe700 1 --2- 172.21.15.137:0/767402955 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f5cc402ff60 0x7f5cc4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:57:11.926 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:11.920+0000 7f5cc37fe700 1 -- 172.21.15.137:0/767402955 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5cd0040de0 con 0x7f5cdc060240 2023-11-15T01:57:11.926 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:11.920+0000 7f5cda59c700 1 --2- 172.21.15.137:0/767402955 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f5cc402ff60 0x7f5cc4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:57:11.926 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:11.921+0000 7f5cda59c700 1 --2- 172.21.15.137:0/767402955 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f5cc402ff60 0x7f5cc4032420 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f5ccc0096a0 tx=0x7f5ccc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:57:11.927 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:11.925+0000 7f5cc37fe700 1 -- 172.21.15.137:0/767402955 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5cd001cb60 con 0x7f5cdc060240 2023-11-15T01:57:12.132 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:12.132+0000 7f5ce1223700 1 -- 172.21.15.137:0/767402955 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5cdc0fae60 con 0x7f5cdc060240 2023-11-15T01:57:12.133 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:12.132+0000 7f5cc37fe700 1 -- 172.21.15.137:0/767402955 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5cd000f280 con 0x7f5cdc060240 2023-11-15T01:57:12.133 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:57:12.134 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:57:12.134 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:12.134+0000 7f5ce1223700 1 -- 172.21.15.137:0/767402955 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f5cc402ff60 msgr2=0x7f5cc4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:57:12.135 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:12.134+0000 7f5ce1223700 1 --2- 172.21.15.137:0/767402955 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f5cc402ff60 0x7f5cc4032420 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f5ccc0096a0 tx=0x7f5ccc006b40).stop 2023-11-15T01:57:12.135 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:12.134+0000 7f5ce1223700 1 -- 172.21.15.137:0/767402955 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5cdc060240 msgr2=0x7f5cdc109540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:57:12.135 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:12.134+0000 7f5ce1223700 1 --2- 172.21.15.137:0/767402955 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5cdc060240 0x7f5cdc109540 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7f5cd0009f80 tx=0x7f5cd0009b30).stop 2023-11-15T01:57:12.135 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:12.134+0000 7f5ce1223700 1 -- 172.21.15.137:0/767402955 shutdown_connections 2023-11-15T01:57:12.135 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:12.134+0000 7f5ce1223700 1 --2- 172.21.15.137:0/767402955 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f5cc402ff60 0x7f5cc4032420 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:57:12.136 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:12.134+0000 7f5ce1223700 1 --2- 172.21.15.137:0/767402955 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5cdc060240 0x7f5cdc109540 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:57:12.136 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:12.134+0000 7f5ce1223700 1 -- 172.21.15.137:0/767402955 >> 172.21.15.137:0/767402955 conn(0x7f5cdc0f0b70 msgr2=0x7f5cdc0f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:57:12.136 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:12.134+0000 7f5ce1223700 1 -- 172.21.15.137:0/767402955 shutdown_connections 2023-11-15T01:57:12.136 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:12.134+0000 7f5ce1223700 1 -- 172.21.15.137:0/767402955 wait complete. 2023-11-15T01:57:12.137 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:57:12.827 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:12 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:57:12.133331+0000 mon.smithi055 (mon.0) 291 : audit [DBG] from='client.? 172.21.15.137:0/767402955' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:57:13.527 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:57:13.528 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:57:13.827 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:13 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:57:12.762743+0000 mgr.smithi055.szwngw (mgr.14164) 231 : cluster [DBG] pgmap v219: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:57:14.708 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:14.706+0000 7f65775fa700 1 -- 172.21.15.137:0/1267382748 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f65700f5410 msgr2=0x7f65700f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:57:14.709 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:14.706+0000 7f65775fa700 1 --2- 172.21.15.137:0/1267382748 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f65700f5410 0x7f65700f5830 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7f6560009fb0 tx=0x7f6560009b30).stop 2023-11-15T01:57:14.709 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:14.706+0000 7f65775fa700 1 -- 172.21.15.137:0/1267382748 shutdown_connections 2023-11-15T01:57:14.709 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:14.706+0000 7f65775fa700 1 --2- 172.21.15.137:0/1267382748 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f65700f5410 0x7f65700f5830 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:57:14.709 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:14.706+0000 7f65775fa700 1 -- 172.21.15.137:0/1267382748 >> 172.21.15.137:0/1267382748 conn(0x7f65700f0b70 msgr2=0x7f65700f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:57:14.710 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:14.706+0000 7f65775fa700 1 -- 172.21.15.137:0/1267382748 shutdown_connections 2023-11-15T01:57:14.710 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:14.706+0000 7f65775fa700 1 -- 172.21.15.137:0/1267382748 wait complete. 2023-11-15T01:57:14.710 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:14.707+0000 7f65775fa700 1 Processor -- start 2023-11-15T01:57:14.710 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:14.707+0000 7f65775fa700 1 -- start start 2023-11-15T01:57:14.711 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:14.707+0000 7f65775fa700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f65700f5410 0x7f6570106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:57:14.711 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:14.707+0000 7f65775fa700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6570106c20 con 0x7f65700f5410 2023-11-15T01:57:14.711 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:14.707+0000 7f6575396700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f65700f5410 0x7f6570106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:57:14.711 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:14.707+0000 7f6575396700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f65700f5410 0x7f6570106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:52024/0 (socket says 172.21.15.137:52024) 2023-11-15T01:57:14.711 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:14.707+0000 7f6575396700 1 -- 172.21.15.137:0/868216644 learned_addr learned my addr 172.21.15.137:0/868216644 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:57:14.712 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:14.707+0000 7f6575396700 1 -- 172.21.15.137:0/868216644 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6560005040 con 0x7f65700f5410 2023-11-15T01:57:14.712 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:14.708+0000 7f6575396700 1 --2- 172.21.15.137:0/868216644 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f65700f5410 0x7f6570106730 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f6560013510 tx=0x7f656000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:57:14.712 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:14.708+0000 7f6565ffb700 1 -- 172.21.15.137:0/868216644 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6560009320 con 0x7f65700f5410 2023-11-15T01:57:14.712 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:14.708+0000 7f65775fa700 1 -- 172.21.15.137:0/868216644 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6570106e20 con 0x7f65700f5410 2023-11-15T01:57:14.712 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:14.708+0000 7f6565ffb700 1 -- 172.21.15.137:0/868216644 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6560013f00 con 0x7f65700f5410 2023-11-15T01:57:14.713 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:14.708+0000 7f6565ffb700 1 -- 172.21.15.137:0/868216644 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6560014d90 con 0x7f65700f5410 2023-11-15T01:57:14.713 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:14.708+0000 7f65775fa700 1 -- 172.21.15.137:0/868216644 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65701072b0 con 0x7f65700f5410 2023-11-15T01:57:14.714 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:14.709+0000 7f6565ffb700 1 -- 172.21.15.137:0/868216644 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f65600095f0 con 0x7f65700f5410 2023-11-15T01:57:14.715 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:14.709+0000 7f65775fa700 1 -- 172.21.15.137:0/868216644 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6570045ad0 con 0x7f65700f5410 2023-11-15T01:57:14.715 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:14.709+0000 7f6565ffb700 1 --2- 172.21.15.137:0/868216644 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f655c02ff60 0x7f655c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:57:14.715 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:14.709+0000 7f6565ffb700 1 -- 172.21.15.137:0/868216644 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6560041000 con 0x7f65700f5410 2023-11-15T01:57:14.716 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:14.709+0000 7f6574b95700 1 --2- 172.21.15.137:0/868216644 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f655c02ff60 0x7f655c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:57:14.716 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:14.710+0000 7f6574b95700 1 --2- 172.21.15.137:0/868216644 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f655c02ff60 0x7f655c032420 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f656c0096a0 tx=0x7f656c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:57:14.716 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:14.714+0000 7f6565ffb700 1 -- 172.21.15.137:0/868216644 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6560014420 con 0x7f65700f5410 2023-11-15T01:57:14.920 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:14.920+0000 7f65775fa700 1 -- 172.21.15.137:0/868216644 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f65700fae60 con 0x7f65700f5410 2023-11-15T01:57:14.921 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:14.921+0000 7f6565ffb700 1 -- 172.21.15.137:0/868216644 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f656001d7f0 con 0x7f65700f5410 2023-11-15T01:57:14.922 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:57:14.922 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:57:14.923 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:14.922+0000 7f65775fa700 1 -- 172.21.15.137:0/868216644 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f655c02ff60 msgr2=0x7f655c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:57:14.923 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:14.922+0000 7f65775fa700 1 --2- 172.21.15.137:0/868216644 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f655c02ff60 0x7f655c032420 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f656c0096a0 tx=0x7f656c006b40).stop 2023-11-15T01:57:14.923 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:14.922+0000 7f65775fa700 1 -- 172.21.15.137:0/868216644 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f65700f5410 msgr2=0x7f6570106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:57:14.923 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:14.922+0000 7f65775fa700 1 --2- 172.21.15.137:0/868216644 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f65700f5410 0x7f6570106730 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f6560013510 tx=0x7f656000b040).stop 2023-11-15T01:57:14.923 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:14.922+0000 7f65775fa700 1 -- 172.21.15.137:0/868216644 shutdown_connections 2023-11-15T01:57:14.924 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:14.922+0000 7f65775fa700 1 --2- 172.21.15.137:0/868216644 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f655c02ff60 0x7f655c032420 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:57:14.924 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:14.922+0000 7f65775fa700 1 --2- 172.21.15.137:0/868216644 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f65700f5410 0x7f6570106730 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:57:14.924 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:14.922+0000 7f65775fa700 1 -- 172.21.15.137:0/868216644 >> 172.21.15.137:0/868216644 conn(0x7f65700f0b70 msgr2=0x7f65700f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:57:14.924 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:14.922+0000 7f65775fa700 1 -- 172.21.15.137:0/868216644 shutdown_connections 2023-11-15T01:57:14.924 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:14.922+0000 7f65775fa700 1 -- 172.21.15.137:0/868216644 wait complete. 2023-11-15T01:57:14.925 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:57:16.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:15 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:57:14.763145+0000 mgr.smithi055.szwngw (mgr.14164) 232 : cluster [DBG] pgmap v220: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:57:16.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:15 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:57:14.921784+0000 mon.smithi055 (mon.0) 292 : audit [DBG] from='client.? 172.21.15.137:0/868216644' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:57:16.307 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:57:16.307 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:57:17.447 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:17.445+0000 7fc188e13700 1 -- 172.21.15.137:0/3144122100 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc1840f62a0 msgr2=0x7fc1840f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:57:17.447 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:17.445+0000 7fc188e13700 1 --2- 172.21.15.137:0/3144122100 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc1840f62a0 0x7fc1840f66c0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7fc174009fb0 tx=0x7fc174009b30).stop 2023-11-15T01:57:17.447 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:17.445+0000 7fc188e13700 1 -- 172.21.15.137:0/3144122100 shutdown_connections 2023-11-15T01:57:17.448 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:17.445+0000 7fc188e13700 1 --2- 172.21.15.137:0/3144122100 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc1840f62a0 0x7fc1840f66c0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:57:17.448 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:17.445+0000 7fc188e13700 1 -- 172.21.15.137:0/3144122100 >> 172.21.15.137:0/3144122100 conn(0x7fc1840f1aa0 msgr2=0x7fc1840f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:57:17.448 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:17.445+0000 7fc188e13700 1 -- 172.21.15.137:0/3144122100 shutdown_connections 2023-11-15T01:57:17.448 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:17.445+0000 7fc188e13700 1 -- 172.21.15.137:0/3144122100 wait complete. 2023-11-15T01:57:17.449 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:17.445+0000 7fc188e13700 1 Processor -- start 2023-11-15T01:57:17.449 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:17.445+0000 7fc188e13700 1 -- start start 2023-11-15T01:57:17.449 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:17.446+0000 7fc188e13700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc1840f62a0 0x7fc1841064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:57:17.449 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:17.446+0000 7fc188e13700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1841069a0 con 0x7fc1840f62a0 2023-11-15T01:57:17.449 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:17.446+0000 7fc1837fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc1840f62a0 0x7fc1841064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:57:17.450 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:17.446+0000 7fc1837fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc1840f62a0 0x7fc1841064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:52036/0 (socket says 172.21.15.137:52036) 2023-11-15T01:57:17.450 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:17.446+0000 7fc1837fe700 1 -- 172.21.15.137:0/1744937782 learned_addr learned my addr 172.21.15.137:0/1744937782 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:57:17.450 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:17.446+0000 7fc1837fe700 1 -- 172.21.15.137:0/1744937782 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc174005040 con 0x7fc1840f62a0 2023-11-15T01:57:17.450 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:17.446+0000 7fc1837fe700 1 --2- 172.21.15.137:0/1744937782 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc1840f62a0 0x7fc1841064b0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7fc174013510 tx=0x7fc17400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:57:17.450 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:17.446+0000 7fc180ff9700 1 -- 172.21.15.137:0/1744937782 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc174009320 con 0x7fc1840f62a0 2023-11-15T01:57:17.451 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:17.447+0000 7fc188e13700 1 -- 172.21.15.137:0/1744937782 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc184106ba0 con 0x7fc1840f62a0 2023-11-15T01:57:17.451 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:17.447+0000 7fc180ff9700 1 -- 172.21.15.137:0/1744937782 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc174013f00 con 0x7fc1840f62a0 2023-11-15T01:57:17.451 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:17.447+0000 7fc180ff9700 1 -- 172.21.15.137:0/1744937782 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc174014dc0 con 0x7fc1840f62a0 2023-11-15T01:57:17.451 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:17.447+0000 7fc188e13700 1 -- 172.21.15.137:0/1744937782 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc184107030 con 0x7fc1840f62a0 2023-11-15T01:57:17.452 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:17.447+0000 7fc180ff9700 1 -- 172.21.15.137:0/1744937782 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc1740095f0 con 0x7fc1840f62a0 2023-11-15T01:57:17.452 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:17.447+0000 7fc188e13700 1 -- 172.21.15.137:0/1744937782 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc184045ad0 con 0x7fc1840f62a0 2023-11-15T01:57:17.453 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:17.448+0000 7fc180ff9700 1 --2- 172.21.15.137:0/1744937782 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fc17002ff60 0x7fc170032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:57:17.453 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:17.448+0000 7fc180ff9700 1 -- 172.21.15.137:0/1744937782 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc174041000 con 0x7fc1840f62a0 2023-11-15T01:57:17.453 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:17.448+0000 7fc17bfff700 1 --2- 172.21.15.137:0/1744937782 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fc17002ff60 0x7fc170032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:57:17.453 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:17.448+0000 7fc17bfff700 1 --2- 172.21.15.137:0/1744937782 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fc17002ff60 0x7fc170032420 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fc16c0096a0 tx=0x7fc16c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:57:17.454 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:17.453+0000 7fc180ff9700 1 -- 172.21.15.137:0/1744937782 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc174014420 con 0x7fc1840f62a0 2023-11-15T01:57:17.662 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:17.661+0000 7fc188e13700 1 -- 172.21.15.137:0/1744937782 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc184107ee0 con 0x7fc1840f62a0 2023-11-15T01:57:17.663 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:17.662+0000 7fc180ff9700 1 -- 172.21.15.137:0/1744937782 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc174010230 con 0x7fc1840f62a0 2023-11-15T01:57:17.663 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:57:17.663 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:57:17.664 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:17.664+0000 7fc188e13700 1 -- 172.21.15.137:0/1744937782 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fc17002ff60 msgr2=0x7fc170032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:57:17.665 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:17.664+0000 7fc188e13700 1 --2- 172.21.15.137:0/1744937782 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fc17002ff60 0x7fc170032420 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fc16c0096a0 tx=0x7fc16c006b40).stop 2023-11-15T01:57:17.665 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:17.664+0000 7fc188e13700 1 -- 172.21.15.137:0/1744937782 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc1840f62a0 msgr2=0x7fc1841064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:57:17.665 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:17.664+0000 7fc188e13700 1 --2- 172.21.15.137:0/1744937782 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc1840f62a0 0x7fc1841064b0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7fc174013510 tx=0x7fc17400b040).stop 2023-11-15T01:57:17.665 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:17.664+0000 7fc188e13700 1 -- 172.21.15.137:0/1744937782 shutdown_connections 2023-11-15T01:57:17.666 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:17.664+0000 7fc188e13700 1 --2- 172.21.15.137:0/1744937782 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fc17002ff60 0x7fc170032420 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:57:17.666 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:17.664+0000 7fc188e13700 1 --2- 172.21.15.137:0/1744937782 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc1840f62a0 0x7fc1841064b0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:57:17.666 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:17.664+0000 7fc188e13700 1 -- 172.21.15.137:0/1744937782 >> 172.21.15.137:0/1744937782 conn(0x7fc1840f1aa0 msgr2=0x7fc1840f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:57:17.666 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:17.664+0000 7fc188e13700 1 -- 172.21.15.137:0/1744937782 shutdown_connections 2023-11-15T01:57:17.666 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:17.664+0000 7fc188e13700 1 -- 172.21.15.137:0/1744937782 wait complete. 2023-11-15T01:57:17.667 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:57:18.327 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:17 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:57:16.763465+0000 mgr.smithi055.szwngw (mgr.14164) 233 : cluster [DBG] pgmap v221: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:57:18.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:17 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:57:17.663176+0000 mon.smithi055 (mon.0) 293 : audit [DBG] from='client.? 172.21.15.137:0/1744937782' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:57:19.046 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:57:19.046 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:57:20.204 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:20.203+0000 7f7e2759e700 1 -- 172.21.15.137:0/1010819064 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7e280f51a0 msgr2=0x7f7e280f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:57:20.205 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:20.203+0000 7f7e2759e700 1 --2- 172.21.15.137:0/1010819064 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7e280f51a0 0x7f7e280f55c0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7f7e18009fb0 tx=0x7f7e18009b30).stop 2023-11-15T01:57:20.205 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:20.203+0000 7f7e2759e700 1 -- 172.21.15.137:0/1010819064 shutdown_connections 2023-11-15T01:57:20.205 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:20.203+0000 7f7e2759e700 1 --2- 172.21.15.137:0/1010819064 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7e280f51a0 0x7f7e280f55c0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:57:20.205 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:20.203+0000 7f7e2759e700 1 -- 172.21.15.137:0/1010819064 >> 172.21.15.137:0/1010819064 conn(0x7f7e280f09a0 msgr2=0x7f7e280f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:57:20.205 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:20.203+0000 7f7e2759e700 1 -- 172.21.15.137:0/1010819064 shutdown_connections 2023-11-15T01:57:20.206 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:20.203+0000 7f7e2759e700 1 -- 172.21.15.137:0/1010819064 wait complete. 2023-11-15T01:57:20.206 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:20.203+0000 7f7e2759e700 1 Processor -- start 2023-11-15T01:57:20.206 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:20.203+0000 7f7e2759e700 1 -- start start 2023-11-15T01:57:20.206 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:20.204+0000 7f7e2759e700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7e280f51a0 0x7f7e281053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:57:20.206 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:20.204+0000 7f7e2759e700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e281058a0 con 0x7f7e280f51a0 2023-11-15T01:57:20.207 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:20.204+0000 7f7e2659c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7e280f51a0 0x7f7e281053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:57:20.207 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:20.204+0000 7f7e2659c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7e280f51a0 0x7f7e281053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:57174/0 (socket says 172.21.15.137:57174) 2023-11-15T01:57:20.207 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:20.204+0000 7f7e2659c700 1 -- 172.21.15.137:0/1576583749 learned_addr learned my addr 172.21.15.137:0/1576583749 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:57:20.209 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:20.204+0000 7f7e2659c700 1 -- 172.21.15.137:0/1576583749 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e18005040 con 0x7f7e280f51a0 2023-11-15T01:57:20.209 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:20.204+0000 7f7e2659c700 1 --2- 172.21.15.137:0/1576583749 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7e280f51a0 0x7f7e281053b0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7f7e18013510 tx=0x7f7e1800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:57:20.209 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:20.204+0000 7f7e16ffd700 1 -- 172.21.15.137:0/1576583749 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e18009320 con 0x7f7e280f51a0 2023-11-15T01:57:20.209 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:20.204+0000 7f7e2759e700 1 -- 172.21.15.137:0/1576583749 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e28105aa0 con 0x7f7e280f51a0 2023-11-15T01:57:20.209 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:20.205+0000 7f7e16ffd700 1 -- 172.21.15.137:0/1576583749 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7e18013f00 con 0x7f7e280f51a0 2023-11-15T01:57:20.210 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:20.205+0000 7f7e16ffd700 1 -- 172.21.15.137:0/1576583749 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e18014dc0 con 0x7f7e280f51a0 2023-11-15T01:57:20.210 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:20.205+0000 7f7e2759e700 1 -- 172.21.15.137:0/1576583749 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e28105f30 con 0x7f7e280f51a0 2023-11-15T01:57:20.210 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:20.205+0000 7f7e2759e700 1 -- 172.21.15.137:0/1576583749 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e28045ad0 con 0x7f7e280f51a0 2023-11-15T01:57:20.210 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:20.205+0000 7f7e16ffd700 1 -- 172.21.15.137:0/1576583749 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7e180095f0 con 0x7f7e280f51a0 2023-11-15T01:57:20.210 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:20.206+0000 7f7e16ffd700 1 --2- 172.21.15.137:0/1576583749 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f7e1002ff60 0x7f7e10032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:57:20.211 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:20.206+0000 7f7e16ffd700 1 -- 172.21.15.137:0/1576583749 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7e180410b0 con 0x7f7e280f51a0 2023-11-15T01:57:20.211 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:20.206+0000 7f7e25d9b700 1 --2- 172.21.15.137:0/1576583749 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f7e1002ff60 0x7f7e10032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:57:20.211 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:20.206+0000 7f7e25d9b700 1 --2- 172.21.15.137:0/1576583749 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f7e1002ff60 0x7f7e10032420 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f7e1c0096a0 tx=0x7f7e1c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:57:20.211 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:20.210+0000 7f7e16ffd700 1 -- 172.21.15.137:0/1576583749 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7e18014420 con 0x7f7e280f51a0 2023-11-15T01:57:20.327 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:19 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:57:18.763889+0000 mgr.smithi055.szwngw (mgr.14164) 234 : cluster [DBG] pgmap v222: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:57:20.417 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:20.417+0000 7f7e2759e700 1 -- 172.21.15.137:0/1576583749 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7e28106de0 con 0x7f7e280f51a0 2023-11-15T01:57:20.418 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:20.417+0000 7f7e16ffd700 1 -- 172.21.15.137:0/1576583749 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7e18010230 con 0x7f7e280f51a0 2023-11-15T01:57:20.419 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:57:20.419 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:57:20.420 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:20.419+0000 7f7e2759e700 1 -- 172.21.15.137:0/1576583749 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f7e1002ff60 msgr2=0x7f7e10032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:57:20.420 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:20.419+0000 7f7e2759e700 1 --2- 172.21.15.137:0/1576583749 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f7e1002ff60 0x7f7e10032420 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f7e1c0096a0 tx=0x7f7e1c006b40).stop 2023-11-15T01:57:20.420 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:20.419+0000 7f7e2759e700 1 -- 172.21.15.137:0/1576583749 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7e280f51a0 msgr2=0x7f7e281053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:57:20.420 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:20.419+0000 7f7e2759e700 1 --2- 172.21.15.137:0/1576583749 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7e280f51a0 0x7f7e281053b0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7f7e18013510 tx=0x7f7e1800b040).stop 2023-11-15T01:57:20.421 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:20.419+0000 7f7e2759e700 1 -- 172.21.15.137:0/1576583749 shutdown_connections 2023-11-15T01:57:20.421 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:20.419+0000 7f7e2759e700 1 --2- 172.21.15.137:0/1576583749 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f7e1002ff60 0x7f7e10032420 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:57:20.421 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:20.419+0000 7f7e2759e700 1 --2- 172.21.15.137:0/1576583749 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7e280f51a0 0x7f7e281053b0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:57:20.421 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:20.419+0000 7f7e2759e700 1 -- 172.21.15.137:0/1576583749 >> 172.21.15.137:0/1576583749 conn(0x7f7e280f09a0 msgr2=0x7f7e280f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:57:20.421 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:20.419+0000 7f7e2759e700 1 -- 172.21.15.137:0/1576583749 shutdown_connections 2023-11-15T01:57:20.422 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:20.419+0000 7f7e2759e700 1 -- 172.21.15.137:0/1576583749 wait complete. 2023-11-15T01:57:20.422 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:57:21.327 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:20 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:57:20.418470+0000 mon.smithi055 (mon.0) 294 : audit [DBG] from='client.? 172.21.15.137:0/1576583749' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:57:21.811 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:57:21.812 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:57:22.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:21 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:57:20.764324+0000 mgr.smithi055.szwngw (mgr.14164) 235 : cluster [DBG] pgmap v223: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:57:23.135 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:23.133+0000 7f47e8963700 1 -- 172.21.15.137:0/3047121768 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f47e40f5410 msgr2=0x7f47e40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:57:23.135 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:23.133+0000 7f47e8963700 1 --2- 172.21.15.137:0/3047121768 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f47e40f5410 0x7f47e40f5830 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7f47cc009fb0 tx=0x7f47cc009b30).stop 2023-11-15T01:57:23.136 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:23.133+0000 7f47e8963700 1 -- 172.21.15.137:0/3047121768 shutdown_connections 2023-11-15T01:57:23.136 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:23.133+0000 7f47e8963700 1 --2- 172.21.15.137:0/3047121768 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f47e40f5410 0x7f47e40f5830 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:57:23.136 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:23.133+0000 7f47e8963700 1 -- 172.21.15.137:0/3047121768 >> 172.21.15.137:0/3047121768 conn(0x7f47e40f0b70 msgr2=0x7f47e40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:57:23.136 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:23.133+0000 7f47e8963700 1 -- 172.21.15.137:0/3047121768 shutdown_connections 2023-11-15T01:57:23.136 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:23.133+0000 7f47e8963700 1 -- 172.21.15.137:0/3047121768 wait complete. 2023-11-15T01:57:23.136 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:23.133+0000 7f47e8963700 1 Processor -- start 2023-11-15T01:57:23.137 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:23.133+0000 7f47e8963700 1 -- start start 2023-11-15T01:57:23.137 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:23.133+0000 7f47e8963700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f47e40f5410 0x7f47e410b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:57:23.137 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:23.133+0000 7f47e8963700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47e4107240 con 0x7f47e40f5410 2023-11-15T01:57:23.137 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:23.134+0000 7f47e259c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f47e40f5410 0x7f47e410b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:57:23.137 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:23.134+0000 7f47e259c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f47e40f5410 0x7f47e410b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:57190/0 (socket says 172.21.15.137:57190) 2023-11-15T01:57:23.138 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:23.134+0000 7f47e259c700 1 -- 172.21.15.137:0/4172688416 learned_addr learned my addr 172.21.15.137:0/4172688416 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:57:23.138 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:23.134+0000 7f47e259c700 1 -- 172.21.15.137:0/4172688416 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47cc005040 con 0x7f47e40f5410 2023-11-15T01:57:23.138 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:23.134+0000 7f47e259c700 1 --2- 172.21.15.137:0/4172688416 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f47e40f5410 0x7f47e410b850 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7f47cc014000 tx=0x7f47cc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:57:23.138 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:23.134+0000 7f47daffd700 1 -- 172.21.15.137:0/4172688416 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f47cc0095f0 con 0x7f47e40f5410 2023-11-15T01:57:23.138 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:23.134+0000 7f47e8963700 1 -- 172.21.15.137:0/4172688416 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47e4060220 con 0x7f47e40f5410 2023-11-15T01:57:23.139 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:23.134+0000 7f47e8963700 1 -- 172.21.15.137:0/4172688416 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47e410bd40 con 0x7f47e40f5410 2023-11-15T01:57:23.139 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:23.134+0000 7f47daffd700 1 -- 172.21.15.137:0/4172688416 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f47cc007e70 con 0x7f47e40f5410 2023-11-15T01:57:23.139 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:23.134+0000 7f47daffd700 1 -- 172.21.15.137:0/4172688416 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f47cc014ce0 con 0x7f47e40f5410 2023-11-15T01:57:23.140 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:23.135+0000 7f47e8963700 1 -- 172.21.15.137:0/4172688416 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47c8004fa0 con 0x7f47e40f5410 2023-11-15T01:57:23.140 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:23.135+0000 7f47daffd700 1 -- 172.21.15.137:0/4172688416 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f47cc025070 con 0x7f47e40f5410 2023-11-15T01:57:23.140 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:23.135+0000 7f47daffd700 1 --2- 172.21.15.137:0/4172688416 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f47d002ff60 0x7f47d0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:57:23.141 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:23.135+0000 7f47daffd700 1 -- 172.21.15.137:0/4172688416 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f47cc040860 con 0x7f47e40f5410 2023-11-15T01:57:23.141 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:23.136+0000 7f47e1d9b700 1 --2- 172.21.15.137:0/4172688416 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f47d002ff60 0x7f47d0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:57:23.141 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:23.136+0000 7f47e1d9b700 1 --2- 172.21.15.137:0/4172688416 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f47d002ff60 0x7f47d0032420 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f47d40096a0 tx=0x7f47d4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:57:23.142 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:23.140+0000 7f47daffd700 1 -- 172.21.15.137:0/4172688416 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f47cc0079c0 con 0x7f47e40f5410 2023-11-15T01:57:23.345 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:23.345+0000 7f47e8963700 1 -- 172.21.15.137:0/4172688416 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f47c8005670 con 0x7f47e40f5410 2023-11-15T01:57:23.346 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:23.345+0000 7f47daffd700 1 -- 172.21.15.137:0/4172688416 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f47cc025300 con 0x7f47e40f5410 2023-11-15T01:57:23.346 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:57:23.346 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:57:23.348 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:23.347+0000 7f47e8963700 1 -- 172.21.15.137:0/4172688416 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f47d002ff60 msgr2=0x7f47d0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:57:23.348 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:23.347+0000 7f47e8963700 1 --2- 172.21.15.137:0/4172688416 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f47d002ff60 0x7f47d0032420 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f47d40096a0 tx=0x7f47d4006b40).stop 2023-11-15T01:57:23.348 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:23.347+0000 7f47e8963700 1 -- 172.21.15.137:0/4172688416 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f47e40f5410 msgr2=0x7f47e410b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:57:23.349 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:23.347+0000 7f47e8963700 1 --2- 172.21.15.137:0/4172688416 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f47e40f5410 0x7f47e410b850 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7f47cc014000 tx=0x7f47cc00b040).stop 2023-11-15T01:57:23.349 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:23.347+0000 7f47e8963700 1 -- 172.21.15.137:0/4172688416 shutdown_connections 2023-11-15T01:57:23.349 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:23.347+0000 7f47e8963700 1 --2- 172.21.15.137:0/4172688416 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f47d002ff60 0x7f47d0032420 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:57:23.349 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:23.347+0000 7f47e8963700 1 --2- 172.21.15.137:0/4172688416 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f47e40f5410 0x7f47e410b850 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:57:23.349 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:23.347+0000 7f47e8963700 1 -- 172.21.15.137:0/4172688416 >> 172.21.15.137:0/4172688416 conn(0x7f47e40f0b70 msgr2=0x7f47e40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:57:23.350 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:23.347+0000 7f47e8963700 1 -- 172.21.15.137:0/4172688416 shutdown_connections 2023-11-15T01:57:23.350 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:23.347+0000 7f47e8963700 1 -- 172.21.15.137:0/4172688416 wait complete. 2023-11-15T01:57:23.350 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:57:24.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:23 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:57:22.764790+0000 mgr.smithi055.szwngw (mgr.14164) 236 : cluster [DBG] pgmap v224: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:57:24.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:23 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:57:23.346490+0000 mon.smithi055 (mon.0) 295 : audit [DBG] from='client.? 172.21.15.137:0/4172688416' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:57:24.730 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:57:24.731 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:57:25.922 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:25.920+0000 7fd975f5f700 1 -- 172.21.15.137:0/2259969506 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd970060220 msgr2=0x7fd9700f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:57:25.922 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:25.920+0000 7fd975f5f700 1 --2- 172.21.15.137:0/2259969506 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd970060220 0x7fd9700f9150 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7fd960009fb0 tx=0x7fd960009b30).stop 2023-11-15T01:57:25.922 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:25.920+0000 7fd975f5f700 1 -- 172.21.15.137:0/2259969506 shutdown_connections 2023-11-15T01:57:25.923 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:25.920+0000 7fd975f5f700 1 --2- 172.21.15.137:0/2259969506 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd970060220 0x7fd9700f9150 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:57:25.923 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:25.920+0000 7fd975f5f700 1 -- 172.21.15.137:0/2259969506 >> 172.21.15.137:0/2259969506 conn(0x7fd9700f0b70 msgr2=0x7fd9700f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:57:25.923 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:25.920+0000 7fd975f5f700 1 -- 172.21.15.137:0/2259969506 shutdown_connections 2023-11-15T01:57:25.923 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:25.920+0000 7fd975f5f700 1 -- 172.21.15.137:0/2259969506 wait complete. 2023-11-15T01:57:25.923 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:25.920+0000 7fd975f5f700 1 Processor -- start 2023-11-15T01:57:25.924 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:25.921+0000 7fd975f5f700 1 -- start start 2023-11-15T01:57:25.924 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:25.921+0000 7fd975f5f700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd970060220 0x7fd9701066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:57:25.924 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:25.921+0000 7fd975f5f700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd970106bd0 con 0x7fd970060220 2023-11-15T01:57:25.924 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:25.921+0000 7fd96f7fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd970060220 0x7fd9701066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:57:25.924 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:25.921+0000 7fd96f7fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd970060220 0x7fd9701066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:57196/0 (socket says 172.21.15.137:57196) 2023-11-15T01:57:25.925 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:25.921+0000 7fd96f7fe700 1 -- 172.21.15.137:0/1608600715 learned_addr learned my addr 172.21.15.137:0/1608600715 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:57:25.925 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:25.921+0000 7fd96f7fe700 1 -- 172.21.15.137:0/1608600715 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd960005040 con 0x7fd970060220 2023-11-15T01:57:25.925 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:25.922+0000 7fd96f7fe700 1 --2- 172.21.15.137:0/1608600715 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd970060220 0x7fd9701066e0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7fd960013510 tx=0x7fd96000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:57:25.926 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:25.922+0000 7fd967fff700 1 -- 172.21.15.137:0/1608600715 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd960009320 con 0x7fd970060220 2023-11-15T01:57:25.926 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:25.922+0000 7fd975f5f700 1 -- 172.21.15.137:0/1608600715 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd970106dd0 con 0x7fd970060220 2023-11-15T01:57:25.927 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:25.922+0000 7fd967fff700 1 -- 172.21.15.137:0/1608600715 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd960013f00 con 0x7fd970060220 2023-11-15T01:57:25.927 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:25.922+0000 7fd967fff700 1 -- 172.21.15.137:0/1608600715 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd960014dc0 con 0x7fd970060220 2023-11-15T01:57:25.927 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:25.922+0000 7fd975f5f700 1 -- 172.21.15.137:0/1608600715 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd970107260 con 0x7fd970060220 2023-11-15T01:57:25.927 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:25.923+0000 7fd975f5f700 1 -- 172.21.15.137:0/1608600715 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd970045ad0 con 0x7fd970060220 2023-11-15T01:57:25.927 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:25.923+0000 7fd967fff700 1 -- 172.21.15.137:0/1608600715 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd9600095f0 con 0x7fd970060220 2023-11-15T01:57:25.928 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:25.923+0000 7fd967fff700 1 --2- 172.21.15.137:0/1608600715 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fd95c02ff60 0x7fd95c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:57:25.928 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:25.923+0000 7fd967fff700 1 -- 172.21.15.137:0/1608600715 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd960041080 con 0x7fd970060220 2023-11-15T01:57:25.928 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:25.923+0000 7fd96effd700 1 --2- 172.21.15.137:0/1608600715 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fd95c02ff60 0x7fd95c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:57:25.928 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:25.924+0000 7fd96effd700 1 --2- 172.21.15.137:0/1608600715 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fd95c02ff60 0x7fd95c032420 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7fd9580096a0 tx=0x7fd958006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:57:25.929 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:25.928+0000 7fd967fff700 1 -- 172.21.15.137:0/1608600715 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd96001c790 con 0x7fd970060220 2023-11-15T01:57:26.134 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:26.133+0000 7fd975f5f700 1 -- 172.21.15.137:0/1608600715 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd970108110 con 0x7fd970060220 2023-11-15T01:57:26.134 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:26.134+0000 7fd967fff700 1 -- 172.21.15.137:0/1608600715 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd960010230 con 0x7fd970060220 2023-11-15T01:57:26.135 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:57:26.135 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:57:26.136 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:26.135+0000 7fd975f5f700 1 -- 172.21.15.137:0/1608600715 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fd95c02ff60 msgr2=0x7fd95c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:57:26.136 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:26.135+0000 7fd975f5f700 1 --2- 172.21.15.137:0/1608600715 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fd95c02ff60 0x7fd95c032420 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7fd9580096a0 tx=0x7fd958006b40).stop 2023-11-15T01:57:26.136 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:26.135+0000 7fd975f5f700 1 -- 172.21.15.137:0/1608600715 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd970060220 msgr2=0x7fd9701066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:57:26.137 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:26.135+0000 7fd975f5f700 1 --2- 172.21.15.137:0/1608600715 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd970060220 0x7fd9701066e0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7fd960013510 tx=0x7fd96000b040).stop 2023-11-15T01:57:26.137 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:26.135+0000 7fd975f5f700 1 -- 172.21.15.137:0/1608600715 shutdown_connections 2023-11-15T01:57:26.137 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:26.135+0000 7fd975f5f700 1 --2- 172.21.15.137:0/1608600715 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fd95c02ff60 0x7fd95c032420 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:57:26.137 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:26.135+0000 7fd975f5f700 1 --2- 172.21.15.137:0/1608600715 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd970060220 0x7fd9701066e0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:57:26.138 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:26.135+0000 7fd975f5f700 1 -- 172.21.15.137:0/1608600715 >> 172.21.15.137:0/1608600715 conn(0x7fd9700f0b70 msgr2=0x7fd9700f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:57:26.138 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:26.135+0000 7fd975f5f700 1 -- 172.21.15.137:0/1608600715 shutdown_connections 2023-11-15T01:57:26.138 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:26.135+0000 7fd975f5f700 1 -- 172.21.15.137:0/1608600715 wait complete. 2023-11-15T01:57:26.139 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:57:26.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:25 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:57:24.765071+0000 mgr.smithi055.szwngw (mgr.14164) 237 : cluster [DBG] pgmap v225: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:57:27.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:26 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:57:26.134868+0000 mon.smithi055 (mon.0) 296 : audit [DBG] from='client.? 172.21.15.137:0/1608600715' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:57:27.520 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:57:27.521 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:57:28.260 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:27 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:57:26.765402+0000 mgr.smithi055.szwngw (mgr.14164) 238 : cluster [DBG] pgmap v226: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:57:28.669 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:28.668+0000 7fe875c1e700 1 -- 172.21.15.137:0/3505323046 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe8700f5410 msgr2=0x7fe8700f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:57:28.669 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:28.668+0000 7fe875c1e700 1 --2- 172.21.15.137:0/3505323046 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe8700f5410 0x7fe8700f5830 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7fe858009fb0 tx=0x7fe858009b30).stop 2023-11-15T01:57:28.670 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:28.668+0000 7fe875c1e700 1 -- 172.21.15.137:0/3505323046 shutdown_connections 2023-11-15T01:57:28.670 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:28.668+0000 7fe875c1e700 1 --2- 172.21.15.137:0/3505323046 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe8700f5410 0x7fe8700f5830 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:57:28.670 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:28.668+0000 7fe875c1e700 1 -- 172.21.15.137:0/3505323046 >> 172.21.15.137:0/3505323046 conn(0x7fe8700f0b70 msgr2=0x7fe8700f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:57:28.670 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:28.668+0000 7fe875c1e700 1 -- 172.21.15.137:0/3505323046 shutdown_connections 2023-11-15T01:57:28.671 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:28.669+0000 7fe875c1e700 1 -- 172.21.15.137:0/3505323046 wait complete. 2023-11-15T01:57:28.673 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:28.669+0000 7fe875c1e700 1 Processor -- start 2023-11-15T01:57:28.673 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:28.669+0000 7fe875c1e700 1 -- start start 2023-11-15T01:57:28.673 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:28.669+0000 7fe875c1e700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe8700f5410 0x7fe870109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:57:28.673 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:28.669+0000 7fe875c1e700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe870060220 con 0x7fe8700f5410 2023-11-15T01:57:28.673 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:28.670+0000 7fe86f7fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe8700f5410 0x7fe870109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:57:28.674 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:28.670+0000 7fe86f7fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe8700f5410 0x7fe870109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:52756/0 (socket says 172.21.15.137:52756) 2023-11-15T01:57:28.674 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:28.670+0000 7fe86f7fe700 1 -- 172.21.15.137:0/3324485057 learned_addr learned my addr 172.21.15.137:0/3324485057 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:57:28.674 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:28.670+0000 7fe86f7fe700 1 -- 172.21.15.137:0/3324485057 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe858005040 con 0x7fe8700f5410 2023-11-15T01:57:28.674 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:28.670+0000 7fe86f7fe700 1 --2- 172.21.15.137:0/3324485057 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe8700f5410 0x7fe870109450 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7fe858014000 tx=0x7fe85800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:57:28.674 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:28.670+0000 7fe867fff700 1 -- 172.21.15.137:0/3324485057 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe8580095f0 con 0x7fe8700f5410 2023-11-15T01:57:28.675 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:28.670+0000 7fe875c1e700 1 -- 172.21.15.137:0/3324485057 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe870060420 con 0x7fe8700f5410 2023-11-15T01:57:28.675 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:28.670+0000 7fe867fff700 1 -- 172.21.15.137:0/3324485057 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe858007e70 con 0x7fe8700f5410 2023-11-15T01:57:28.675 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:28.670+0000 7fe867fff700 1 -- 172.21.15.137:0/3324485057 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe858014ea0 con 0x7fe8700f5410 2023-11-15T01:57:28.676 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:28.670+0000 7fe875c1e700 1 -- 172.21.15.137:0/3324485057 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe870109b00 con 0x7fe8700f5410 2023-11-15T01:57:28.676 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:28.671+0000 7fe875c1e700 1 -- 172.21.15.137:0/3324485057 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe870059100 con 0x7fe8700f5410 2023-11-15T01:57:28.676 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:28.671+0000 7fe867fff700 1 -- 172.21.15.137:0/3324485057 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe858025070 con 0x7fe8700f5410 2023-11-15T01:57:28.676 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:28.671+0000 7fe867fff700 1 --2- 172.21.15.137:0/3324485057 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fe85c02ff60 0x7fe85c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:57:28.676 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:28.671+0000 7fe867fff700 1 -- 172.21.15.137:0/3324485057 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe858040920 con 0x7fe8700f5410 2023-11-15T01:57:28.677 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:28.672+0000 7fe86effd700 1 --2- 172.21.15.137:0/3324485057 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fe85c02ff60 0x7fe85c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:57:28.677 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:28.672+0000 7fe86effd700 1 --2- 172.21.15.137:0/3324485057 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fe85c02ff60 0x7fe85c032420 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7fe8600096a0 tx=0x7fe860006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:57:28.677 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:28.676+0000 7fe867fff700 1 -- 172.21.15.137:0/3324485057 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe85800f4e0 con 0x7fe8700f5410 2023-11-15T01:57:28.881 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:28.880+0000 7fe875c1e700 1 -- 172.21.15.137:0/3324485057 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe870045ad0 con 0x7fe8700f5410 2023-11-15T01:57:28.882 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:28.881+0000 7fe867fff700 1 -- 172.21.15.137:0/3324485057 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe85800f300 con 0x7fe8700f5410 2023-11-15T01:57:28.882 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:57:28.882 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:57:28.883 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:28.883+0000 7fe875c1e700 1 -- 172.21.15.137:0/3324485057 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fe85c02ff60 msgr2=0x7fe85c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:57:28.883 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:28.883+0000 7fe875c1e700 1 --2- 172.21.15.137:0/3324485057 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fe85c02ff60 0x7fe85c032420 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7fe8600096a0 tx=0x7fe860006b40).stop 2023-11-15T01:57:28.884 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:28.883+0000 7fe875c1e700 1 -- 172.21.15.137:0/3324485057 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe8700f5410 msgr2=0x7fe870109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:57:28.884 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:28.883+0000 7fe875c1e700 1 --2- 172.21.15.137:0/3324485057 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe8700f5410 0x7fe870109450 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7fe858014000 tx=0x7fe85800b040).stop 2023-11-15T01:57:28.884 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:28.883+0000 7fe875c1e700 1 -- 172.21.15.137:0/3324485057 shutdown_connections 2023-11-15T01:57:28.884 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:28.883+0000 7fe875c1e700 1 --2- 172.21.15.137:0/3324485057 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fe85c02ff60 0x7fe85c032420 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:57:28.884 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:28.883+0000 7fe875c1e700 1 --2- 172.21.15.137:0/3324485057 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe8700f5410 0x7fe870109450 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:57:28.885 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:28.883+0000 7fe875c1e700 1 -- 172.21.15.137:0/3324485057 >> 172.21.15.137:0/3324485057 conn(0x7fe8700f0b70 msgr2=0x7fe8700f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:57:28.885 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:28.883+0000 7fe875c1e700 1 -- 172.21.15.137:0/3324485057 shutdown_connections 2023-11-15T01:57:28.885 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:28.883+0000 7fe875c1e700 1 -- 172.21.15.137:0/3324485057 wait complete. 2023-11-15T01:57:28.885 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:57:30.258 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:57:30.258 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:57:30.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:29 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:57:28.765731+0000 mgr.smithi055.szwngw (mgr.14164) 239 : cluster [DBG] pgmap v227: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:57:30.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:29 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:57:28.882282+0000 mon.smithi055 (mon.0) 297 : audit [DBG] from='client.? 172.21.15.137:0/3324485057' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:57:31.403 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:31.401+0000 7f7fd75dd700 1 -- 172.21.15.137:0/896970862 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7fd00f6510 msgr2=0x7f7fd00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:57:31.403 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:31.401+0000 7f7fd75dd700 1 --2- 172.21.15.137:0/896970862 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7fd00f6510 0x7f7fd00f6930 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7f7fc4009fb0 tx=0x7f7fc4009b30).stop 2023-11-15T01:57:31.403 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:31.401+0000 7f7fd75dd700 1 -- 172.21.15.137:0/896970862 shutdown_connections 2023-11-15T01:57:31.403 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:31.401+0000 7f7fd75dd700 1 --2- 172.21.15.137:0/896970862 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7fd00f6510 0x7f7fd00f6930 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:57:31.404 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:31.401+0000 7f7fd75dd700 1 -- 172.21.15.137:0/896970862 >> 172.21.15.137:0/896970862 conn(0x7f7fd00f1c70 msgr2=0x7f7fd00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:57:31.404 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:31.401+0000 7f7fd75dd700 1 -- 172.21.15.137:0/896970862 shutdown_connections 2023-11-15T01:57:31.404 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:31.401+0000 7f7fd75dd700 1 -- 172.21.15.137:0/896970862 wait complete. 2023-11-15T01:57:31.404 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:31.402+0000 7f7fd75dd700 1 Processor -- start 2023-11-15T01:57:31.404 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:31.402+0000 7f7fd75dd700 1 -- start start 2023-11-15T01:57:31.405 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:31.402+0000 7f7fd75dd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7fd00f6510 0x7f7fd006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:57:31.405 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:31.402+0000 7f7fd75dd700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fd0069e20 con 0x7f7fd00f6510 2023-11-15T01:57:31.405 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:31.402+0000 7f7fd5379700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7fd00f6510 0x7f7fd006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:57:31.405 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:31.402+0000 7f7fd5379700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7fd00f6510 0x7f7fd006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:52766/0 (socket says 172.21.15.137:52766) 2023-11-15T01:57:31.405 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:31.402+0000 7f7fd5379700 1 -- 172.21.15.137:0/4110822124 learned_addr learned my addr 172.21.15.137:0/4110822124 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:57:31.406 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:31.402+0000 7f7fd5379700 1 -- 172.21.15.137:0/4110822124 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fc4005040 con 0x7f7fd00f6510 2023-11-15T01:57:31.407 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:31.403+0000 7f7fd5379700 1 --2- 172.21.15.137:0/4110822124 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7fd00f6510 0x7f7fd006b9a0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f7fc4013510 tx=0x7f7fc400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:57:31.407 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:31.403+0000 7f7fc1ffb700 1 -- 172.21.15.137:0/4110822124 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7fc4009320 con 0x7f7fd00f6510 2023-11-15T01:57:31.407 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:31.403+0000 7f7fd75dd700 1 -- 172.21.15.137:0/4110822124 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fd0069fc0 con 0x7f7fd00f6510 2023-11-15T01:57:31.408 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:31.403+0000 7f7fc1ffb700 1 -- 172.21.15.137:0/4110822124 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7fc4013f00 con 0x7f7fd00f6510 2023-11-15T01:57:31.408 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:31.403+0000 7f7fc1ffb700 1 -- 172.21.15.137:0/4110822124 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7fc4014d90 con 0x7f7fd00f6510 2023-11-15T01:57:31.408 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:31.403+0000 7f7fd75dd700 1 -- 172.21.15.137:0/4110822124 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fd006a450 con 0x7f7fd00f6510 2023-11-15T01:57:31.408 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:31.404+0000 7f7fd75dd700 1 -- 172.21.15.137:0/4110822124 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fd0059100 con 0x7f7fd00f6510 2023-11-15T01:57:31.408 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:31.404+0000 7f7fc1ffb700 1 -- 172.21.15.137:0/4110822124 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7fc40095f0 con 0x7f7fd00f6510 2023-11-15T01:57:31.409 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:31.404+0000 7f7fc1ffb700 1 --2- 172.21.15.137:0/4110822124 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f7fbc02ff60 0x7f7fbc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:57:31.409 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:31.404+0000 7f7fc1ffb700 1 -- 172.21.15.137:0/4110822124 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7fc4040fc0 con 0x7f7fd00f6510 2023-11-15T01:57:31.409 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:31.404+0000 7f7fd4b78700 1 --2- 172.21.15.137:0/4110822124 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f7fbc02ff60 0x7f7fbc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:57:31.409 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:31.405+0000 7f7fd4b78700 1 --2- 172.21.15.137:0/4110822124 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f7fbc02ff60 0x7f7fbc032420 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f7fcc0096a0 tx=0x7f7fcc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:57:31.410 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:31.409+0000 7f7fc1ffb700 1 -- 172.21.15.137:0/4110822124 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7fc4014420 con 0x7f7fd00f6510 2023-11-15T01:57:31.616 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:31.616+0000 7f7fd75dd700 1 -- 172.21.15.137:0/4110822124 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7fd0045ad0 con 0x7f7fd00f6510 2023-11-15T01:57:31.617 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:31.616+0000 7f7fc1ffb700 1 -- 172.21.15.137:0/4110822124 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7fc4010230 con 0x7f7fd00f6510 2023-11-15T01:57:31.617 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:57:31.618 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:57:31.619 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:31.618+0000 7f7fd75dd700 1 -- 172.21.15.137:0/4110822124 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f7fbc02ff60 msgr2=0x7f7fbc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:57:31.619 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:31.618+0000 7f7fd75dd700 1 --2- 172.21.15.137:0/4110822124 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f7fbc02ff60 0x7f7fbc032420 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f7fcc0096a0 tx=0x7f7fcc006b40).stop 2023-11-15T01:57:31.619 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:31.618+0000 7f7fd75dd700 1 -- 172.21.15.137:0/4110822124 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7fd00f6510 msgr2=0x7f7fd006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:57:31.619 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:31.618+0000 7f7fd75dd700 1 --2- 172.21.15.137:0/4110822124 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7fd00f6510 0x7f7fd006b9a0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f7fc4013510 tx=0x7f7fc400b040).stop 2023-11-15T01:57:31.620 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:31.618+0000 7f7fd75dd700 1 -- 172.21.15.137:0/4110822124 shutdown_connections 2023-11-15T01:57:31.620 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:31.618+0000 7f7fd75dd700 1 --2- 172.21.15.137:0/4110822124 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f7fbc02ff60 0x7f7fbc032420 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:57:31.620 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:31.618+0000 7f7fd75dd700 1 --2- 172.21.15.137:0/4110822124 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7fd00f6510 0x7f7fd006b9a0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:57:31.620 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:31.618+0000 7f7fd75dd700 1 -- 172.21.15.137:0/4110822124 >> 172.21.15.137:0/4110822124 conn(0x7f7fd00f1c70 msgr2=0x7f7fd00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:57:31.620 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:31.618+0000 7f7fd75dd700 1 -- 172.21.15.137:0/4110822124 shutdown_connections 2023-11-15T01:57:31.621 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:31.618+0000 7f7fd75dd700 1 -- 172.21.15.137:0/4110822124 wait complete. 2023-11-15T01:57:31.621 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:57:31.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:31 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:57:30.766133+0000 mgr.smithi055.szwngw (mgr.14164) 240 : cluster [DBG] pgmap v228: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:57:32.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:32 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:57:31.617576+0000 mon.smithi055 (mon.0) 298 : audit [DBG] from='client.? 172.21.15.137:0/4110822124' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:57:33.002 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:57:33.003 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:57:33.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:33 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:57:32.766593+0000 mgr.smithi055.szwngw (mgr.14164) 241 : cluster [DBG] pgmap v229: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:57:34.130 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:34.129+0000 7fdf22a0b700 1 -- 172.21.15.137:0/1324654776 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdf1c0f6510 msgr2=0x7fdf1c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:57:34.130 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:34.129+0000 7fdf22a0b700 1 --2- 172.21.15.137:0/1324654776 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdf1c0f6510 0x7fdf1c0f6930 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7fdf04009fb0 tx=0x7fdf04009b30).stop 2023-11-15T01:57:34.131 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:34.129+0000 7fdf22a0b700 1 -- 172.21.15.137:0/1324654776 shutdown_connections 2023-11-15T01:57:34.131 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:34.129+0000 7fdf22a0b700 1 --2- 172.21.15.137:0/1324654776 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdf1c0f6510 0x7fdf1c0f6930 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:57:34.131 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:34.129+0000 7fdf22a0b700 1 -- 172.21.15.137:0/1324654776 >> 172.21.15.137:0/1324654776 conn(0x7fdf1c0f1c70 msgr2=0x7fdf1c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:57:34.131 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:34.129+0000 7fdf22a0b700 1 -- 172.21.15.137:0/1324654776 shutdown_connections 2023-11-15T01:57:34.131 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:34.129+0000 7fdf22a0b700 1 -- 172.21.15.137:0/1324654776 wait complete. 2023-11-15T01:57:34.132 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:34.129+0000 7fdf22a0b700 1 Processor -- start 2023-11-15T01:57:34.132 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:34.130+0000 7fdf22a0b700 1 -- start start 2023-11-15T01:57:34.133 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:34.130+0000 7fdf22a0b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdf1c0f6510 0x7fdf1c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:57:34.134 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:34.130+0000 7fdf22a0b700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf1c106c20 con 0x7fdf1c0f6510 2023-11-15T01:57:34.134 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:34.130+0000 7fdf1bfff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdf1c0f6510 0x7fdf1c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:57:34.134 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:34.130+0000 7fdf1bfff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdf1c0f6510 0x7fdf1c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:52786/0 (socket says 172.21.15.137:52786) 2023-11-15T01:57:34.134 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:34.130+0000 7fdf1bfff700 1 -- 172.21.15.137:0/3183837222 learned_addr learned my addr 172.21.15.137:0/3183837222 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:57:34.135 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:34.131+0000 7fdf1bfff700 1 -- 172.21.15.137:0/3183837222 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf04005040 con 0x7fdf1c0f6510 2023-11-15T01:57:34.135 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:34.131+0000 7fdf1bfff700 1 --2- 172.21.15.137:0/3183837222 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdf1c0f6510 0x7fdf1c106730 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7fdf04013510 tx=0x7fdf0400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:57:34.135 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:34.131+0000 7fdf18ff9700 1 -- 172.21.15.137:0/3183837222 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf04009320 con 0x7fdf1c0f6510 2023-11-15T01:57:34.135 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:34.131+0000 7fdf22a0b700 1 -- 172.21.15.137:0/3183837222 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf1c106e20 con 0x7fdf1c0f6510 2023-11-15T01:57:34.135 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:34.131+0000 7fdf18ff9700 1 -- 172.21.15.137:0/3183837222 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdf04013f00 con 0x7fdf1c0f6510 2023-11-15T01:57:34.136 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:34.131+0000 7fdf18ff9700 1 -- 172.21.15.137:0/3183837222 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf04014d90 con 0x7fdf1c0f6510 2023-11-15T01:57:34.137 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:34.131+0000 7fdf22a0b700 1 -- 172.21.15.137:0/3183837222 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf1c1072b0 con 0x7fdf1c0f6510 2023-11-15T01:57:34.137 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:34.132+0000 7fdf18ff9700 1 -- 172.21.15.137:0/3183837222 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdf040095f0 con 0x7fdf1c0f6510 2023-11-15T01:57:34.138 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:34.132+0000 7fdf22a0b700 1 -- 172.21.15.137:0/3183837222 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf1c045ad0 con 0x7fdf1c0f6510 2023-11-15T01:57:34.138 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:34.132+0000 7fdf18ff9700 1 --2- 172.21.15.137:0/3183837222 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fdf0802ff60 0x7fdf08032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:57:34.138 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:34.132+0000 7fdf18ff9700 1 -- 172.21.15.137:0/3183837222 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdf04041080 con 0x7fdf1c0f6510 2023-11-15T01:57:34.138 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:34.132+0000 7fdf1b7fe700 1 --2- 172.21.15.137:0/3183837222 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fdf0802ff60 0x7fdf08032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:57:34.138 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:34.133+0000 7fdf1b7fe700 1 --2- 172.21.15.137:0/3183837222 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fdf0802ff60 0x7fdf08032420 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fdf0c0096a0 tx=0x7fdf0c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:57:34.139 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:34.137+0000 7fdf18ff9700 1 -- 172.21.15.137:0/3183837222 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdf04014420 con 0x7fdf1c0f6510 2023-11-15T01:57:34.342 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:34.342+0000 7fdf22a0b700 1 -- 172.21.15.137:0/3183837222 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdf1c0fbf60 con 0x7fdf1c0f6510 2023-11-15T01:57:34.343 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:34.343+0000 7fdf18ff9700 1 -- 172.21.15.137:0/3183837222 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdf04010230 con 0x7fdf1c0f6510 2023-11-15T01:57:34.344 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:57:34.344 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:57:34.345 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:34.344+0000 7fdf22a0b700 1 -- 172.21.15.137:0/3183837222 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fdf0802ff60 msgr2=0x7fdf08032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:57:34.345 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:34.344+0000 7fdf22a0b700 1 --2- 172.21.15.137:0/3183837222 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fdf0802ff60 0x7fdf08032420 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fdf0c0096a0 tx=0x7fdf0c006b40).stop 2023-11-15T01:57:34.345 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:34.344+0000 7fdf22a0b700 1 -- 172.21.15.137:0/3183837222 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdf1c0f6510 msgr2=0x7fdf1c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:57:34.346 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:34.344+0000 7fdf22a0b700 1 --2- 172.21.15.137:0/3183837222 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdf1c0f6510 0x7fdf1c106730 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7fdf04013510 tx=0x7fdf0400b040).stop 2023-11-15T01:57:34.346 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:34.344+0000 7fdf22a0b700 1 -- 172.21.15.137:0/3183837222 shutdown_connections 2023-11-15T01:57:34.346 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:34.344+0000 7fdf22a0b700 1 --2- 172.21.15.137:0/3183837222 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fdf0802ff60 0x7fdf08032420 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:57:34.346 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:34.344+0000 7fdf22a0b700 1 --2- 172.21.15.137:0/3183837222 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdf1c0f6510 0x7fdf1c106730 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:57:34.346 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:34.344+0000 7fdf22a0b700 1 -- 172.21.15.137:0/3183837222 >> 172.21.15.137:0/3183837222 conn(0x7fdf1c0f1c70 msgr2=0x7fdf1c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:57:34.347 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:34.344+0000 7fdf22a0b700 1 -- 172.21.15.137:0/3183837222 shutdown_connections 2023-11-15T01:57:34.347 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:34.344+0000 7fdf22a0b700 1 -- 172.21.15.137:0/3183837222 wait complete. 2023-11-15T01:57:34.347 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:57:34.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:34 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:57:34.343767+0000 mon.smithi055 (mon.0) 299 : audit [DBG] from='client.? 172.21.15.137:0/3183837222' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:57:35.709 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:57:35.710 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:57:35.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:35 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:57:34.766953+0000 mgr.smithi055.szwngw (mgr.14164) 242 : cluster [DBG] pgmap v230: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:57:36.878 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:36.876+0000 7ff723f39700 1 -- 172.21.15.137:0/1652297658 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff71c060280 msgr2=0x7ff71c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:57:36.878 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:36.876+0000 7ff723f39700 1 --2- 172.21.15.137:0/1652297658 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff71c060280 0x7ff71c0f90f0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7ff70c009fb0 tx=0x7ff70c009b30).stop 2023-11-15T01:57:36.879 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:36.876+0000 7ff723f39700 1 -- 172.21.15.137:0/1652297658 shutdown_connections 2023-11-15T01:57:36.879 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:36.876+0000 7ff723f39700 1 --2- 172.21.15.137:0/1652297658 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff71c060280 0x7ff71c0f90f0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:57:36.879 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:36.876+0000 7ff723f39700 1 -- 172.21.15.137:0/1652297658 >> 172.21.15.137:0/1652297658 conn(0x7ff71c0f0b70 msgr2=0x7ff71c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:57:36.879 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:36.877+0000 7ff723f39700 1 -- 172.21.15.137:0/1652297658 shutdown_connections 2023-11-15T01:57:36.879 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:36.877+0000 7ff723f39700 1 -- 172.21.15.137:0/1652297658 wait complete. 2023-11-15T01:57:36.880 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:36.877+0000 7ff723f39700 1 Processor -- start 2023-11-15T01:57:36.880 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:36.877+0000 7ff723f39700 1 -- start start 2023-11-15T01:57:36.880 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:36.877+0000 7ff723f39700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff71c060280 0x7ff71c106690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:57:36.880 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:36.877+0000 7ff723f39700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff71c106b80 con 0x7ff71c060280 2023-11-15T01:57:36.880 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:36.878+0000 7ff721cd5700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff71c060280 0x7ff71c106690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:57:36.881 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:36.878+0000 7ff721cd5700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff71c060280 0x7ff71c106690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:52806/0 (socket says 172.21.15.137:52806) 2023-11-15T01:57:36.881 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:36.878+0000 7ff721cd5700 1 -- 172.21.15.137:0/1554731301 learned_addr learned my addr 172.21.15.137:0/1554731301 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:57:36.882 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:36.878+0000 7ff721cd5700 1 -- 172.21.15.137:0/1554731301 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff70c005040 con 0x7ff71c060280 2023-11-15T01:57:36.883 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:36.878+0000 7ff721cd5700 1 --2- 172.21.15.137:0/1554731301 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff71c060280 0x7ff71c106690 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7ff70c013510 tx=0x7ff70c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:57:36.883 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:36.878+0000 7ff7127fc700 1 -- 172.21.15.137:0/1554731301 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff70c009320 con 0x7ff71c060280 2023-11-15T01:57:36.883 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:36.878+0000 7ff723f39700 1 -- 172.21.15.137:0/1554731301 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff71c106d80 con 0x7ff71c060280 2023-11-15T01:57:36.883 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:36.878+0000 7ff7127fc700 1 -- 172.21.15.137:0/1554731301 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff70c013f00 con 0x7ff71c060280 2023-11-15T01:57:36.883 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:36.878+0000 7ff7127fc700 1 -- 172.21.15.137:0/1554731301 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff70c014dc0 con 0x7ff71c060280 2023-11-15T01:57:36.884 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:36.878+0000 7ff723f39700 1 -- 172.21.15.137:0/1554731301 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff71c107210 con 0x7ff71c060280 2023-11-15T01:57:36.884 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:36.879+0000 7ff723f39700 1 -- 172.21.15.137:0/1554731301 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff71c045ad0 con 0x7ff71c060280 2023-11-15T01:57:36.884 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:36.879+0000 7ff7127fc700 1 -- 172.21.15.137:0/1554731301 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff70c0095f0 con 0x7ff71c060280 2023-11-15T01:57:36.884 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:36.879+0000 7ff7127fc700 1 --2- 172.21.15.137:0/1554731301 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7ff70802ff60 0x7ff708032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:57:36.884 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:36.879+0000 7ff7127fc700 1 -- 172.21.15.137:0/1554731301 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff70c041100 con 0x7ff71c060280 2023-11-15T01:57:36.885 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:36.880+0000 7ff7214d4700 1 --2- 172.21.15.137:0/1554731301 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7ff70802ff60 0x7ff708032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:57:36.885 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:36.880+0000 7ff7214d4700 1 --2- 172.21.15.137:0/1554731301 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7ff70802ff60 0x7ff708032420 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7ff7180096a0 tx=0x7ff718006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:57:36.885 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:36.884+0000 7ff7127fc700 1 -- 172.21.15.137:0/1554731301 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff70c014420 con 0x7ff71c060280 2023-11-15T01:57:37.097 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:37.096+0000 7ff723f39700 1 -- 172.21.15.137:0/1554731301 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff71c1080c0 con 0x7ff71c060280 2023-11-15T01:57:37.098 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:37.097+0000 7ff7127fc700 1 -- 172.21.15.137:0/1554731301 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff70c010230 con 0x7ff71c060280 2023-11-15T01:57:37.098 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:57:37.098 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:57:37.099 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:37.098+0000 7ff723f39700 1 -- 172.21.15.137:0/1554731301 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7ff70802ff60 msgr2=0x7ff708032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:57:37.099 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:37.098+0000 7ff723f39700 1 --2- 172.21.15.137:0/1554731301 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7ff70802ff60 0x7ff708032420 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7ff7180096a0 tx=0x7ff718006b40).stop 2023-11-15T01:57:37.100 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:37.098+0000 7ff723f39700 1 -- 172.21.15.137:0/1554731301 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff71c060280 msgr2=0x7ff71c106690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:57:37.100 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:37.098+0000 7ff723f39700 1 --2- 172.21.15.137:0/1554731301 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff71c060280 0x7ff71c106690 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7ff70c013510 tx=0x7ff70c00b040).stop 2023-11-15T01:57:37.100 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:37.099+0000 7ff723f39700 1 -- 172.21.15.137:0/1554731301 shutdown_connections 2023-11-15T01:57:37.100 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:37.099+0000 7ff723f39700 1 --2- 172.21.15.137:0/1554731301 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7ff70802ff60 0x7ff708032420 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:57:37.100 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:37.099+0000 7ff723f39700 1 --2- 172.21.15.137:0/1554731301 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff71c060280 0x7ff71c106690 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:57:37.100 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:37.099+0000 7ff723f39700 1 -- 172.21.15.137:0/1554731301 >> 172.21.15.137:0/1554731301 conn(0x7ff71c0f0b70 msgr2=0x7ff71c0f1600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:57:37.101 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:37.099+0000 7ff723f39700 1 -- 172.21.15.137:0/1554731301 shutdown_connections 2023-11-15T01:57:37.101 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:37.099+0000 7ff723f39700 1 -- 172.21.15.137:0/1554731301 wait complete. 2023-11-15T01:57:37.101 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:57:38.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:37 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:57:36.767314+0000 mgr.smithi055.szwngw (mgr.14164) 243 : cluster [DBG] pgmap v231: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:57:38.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:37 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:57:37.098076+0000 mon.smithi055 (mon.0) 300 : audit [DBG] from='client.? 172.21.15.137:0/1554731301' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:57:38.491 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:57:38.491 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:57:39.611 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:39.609+0000 7f8db6d7d700 1 -- 172.21.15.137:0/2428457364 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8db00f62a0 msgr2=0x7f8db00f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:57:39.611 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:39.609+0000 7f8db6d7d700 1 --2- 172.21.15.137:0/2428457364 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8db00f62a0 0x7f8db00f66c0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7f8da0009fb0 tx=0x7f8da0009b30).stop 2023-11-15T01:57:39.612 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:39.610+0000 7f8db6d7d700 1 -- 172.21.15.137:0/2428457364 shutdown_connections 2023-11-15T01:57:39.612 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:39.610+0000 7f8db6d7d700 1 --2- 172.21.15.137:0/2428457364 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8db00f62a0 0x7f8db00f66c0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:57:39.612 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:39.610+0000 7f8db6d7d700 1 -- 172.21.15.137:0/2428457364 >> 172.21.15.137:0/2428457364 conn(0x7f8db00f1aa0 msgr2=0x7f8db00f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:57:39.612 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:39.610+0000 7f8db6d7d700 1 -- 172.21.15.137:0/2428457364 shutdown_connections 2023-11-15T01:57:39.612 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:39.610+0000 7f8db6d7d700 1 -- 172.21.15.137:0/2428457364 wait complete. 2023-11-15T01:57:39.613 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:39.610+0000 7f8db6d7d700 1 Processor -- start 2023-11-15T01:57:39.613 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:39.610+0000 7f8db6d7d700 1 -- start start 2023-11-15T01:57:39.613 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:39.610+0000 7f8db6d7d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8db00f62a0 0x7f8db01064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:57:39.613 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:39.610+0000 7f8db6d7d700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8db01069a0 con 0x7f8db00f62a0 2023-11-15T01:57:39.613 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:39.611+0000 7f8db5d7b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8db00f62a0 0x7f8db01064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:57:39.614 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:39.611+0000 7f8db5d7b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8db00f62a0 0x7f8db01064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:46096/0 (socket says 172.21.15.137:46096) 2023-11-15T01:57:39.614 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:39.611+0000 7f8db5d7b700 1 -- 172.21.15.137:0/2428973735 learned_addr learned my addr 172.21.15.137:0/2428973735 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:57:39.615 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:39.611+0000 7f8db5d7b700 1 -- 172.21.15.137:0/2428973735 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8da0005040 con 0x7f8db00f62a0 2023-11-15T01:57:39.615 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:39.611+0000 7f8db5d7b700 1 --2- 172.21.15.137:0/2428973735 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8db00f62a0 0x7f8db01064b0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f8da0013510 tx=0x7f8da000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:57:39.616 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:39.611+0000 7f8da67fc700 1 -- 172.21.15.137:0/2428973735 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8da0009320 con 0x7f8db00f62a0 2023-11-15T01:57:39.616 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:39.611+0000 7f8db6d7d700 1 -- 172.21.15.137:0/2428973735 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8db0106ba0 con 0x7f8db00f62a0 2023-11-15T01:57:39.616 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:39.611+0000 7f8da67fc700 1 -- 172.21.15.137:0/2428973735 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8da0013f00 con 0x7f8db00f62a0 2023-11-15T01:57:39.616 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:39.611+0000 7f8da67fc700 1 -- 172.21.15.137:0/2428973735 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8da0014dc0 con 0x7f8db00f62a0 2023-11-15T01:57:39.616 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:39.611+0000 7f8db6d7d700 1 -- 172.21.15.137:0/2428973735 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8db0107030 con 0x7f8db00f62a0 2023-11-15T01:57:39.617 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:39.612+0000 7f8da67fc700 1 -- 172.21.15.137:0/2428973735 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8da00095f0 con 0x7f8db00f62a0 2023-11-15T01:57:39.617 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:39.612+0000 7f8db6d7d700 1 -- 172.21.15.137:0/2428973735 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8db0045ad0 con 0x7f8db00f62a0 2023-11-15T01:57:39.617 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:39.612+0000 7f8da67fc700 1 --2- 172.21.15.137:0/2428973735 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f8d9c02ff60 0x7f8d9c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:57:39.617 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:39.612+0000 7f8da67fc700 1 -- 172.21.15.137:0/2428973735 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8da00410f0 con 0x7f8db00f62a0 2023-11-15T01:57:39.617 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:39.612+0000 7f8db557a700 1 --2- 172.21.15.137:0/2428973735 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f8d9c02ff60 0x7f8d9c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:57:39.617 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:39.613+0000 7f8db557a700 1 --2- 172.21.15.137:0/2428973735 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f8d9c02ff60 0x7f8d9c032420 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f8dac0096a0 tx=0x7f8dac006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:57:39.618 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:39.617+0000 7f8da67fc700 1 -- 172.21.15.137:0/2428973735 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8da0014420 con 0x7f8db00f62a0 2023-11-15T01:57:39.823 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:39.823+0000 7f8db6d7d700 1 -- 172.21.15.137:0/2428973735 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8db0107ee0 con 0x7f8db00f62a0 2023-11-15T01:57:39.825 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:39.825+0000 7f8da67fc700 1 -- 172.21.15.137:0/2428973735 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8da0010230 con 0x7f8db00f62a0 2023-11-15T01:57:39.826 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:57:39.826 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:57:39.827 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:39.826+0000 7f8db6d7d700 1 -- 172.21.15.137:0/2428973735 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f8d9c02ff60 msgr2=0x7f8d9c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:57:39.827 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:39.826+0000 7f8db6d7d700 1 --2- 172.21.15.137:0/2428973735 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f8d9c02ff60 0x7f8d9c032420 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f8dac0096a0 tx=0x7f8dac006b40).stop 2023-11-15T01:57:39.828 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:39.826+0000 7f8db6d7d700 1 -- 172.21.15.137:0/2428973735 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8db00f62a0 msgr2=0x7f8db01064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:57:39.828 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:39.826+0000 7f8db6d7d700 1 --2- 172.21.15.137:0/2428973735 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8db00f62a0 0x7f8db01064b0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f8da0013510 tx=0x7f8da000b040).stop 2023-11-15T01:57:39.828 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:39.826+0000 7f8db6d7d700 1 -- 172.21.15.137:0/2428973735 shutdown_connections 2023-11-15T01:57:39.828 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:39.826+0000 7f8db6d7d700 1 --2- 172.21.15.137:0/2428973735 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f8d9c02ff60 0x7f8d9c032420 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:57:39.828 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:39.826+0000 7f8db6d7d700 1 --2- 172.21.15.137:0/2428973735 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8db00f62a0 0x7f8db01064b0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:57:39.829 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:39.826+0000 7f8db6d7d700 1 -- 172.21.15.137:0/2428973735 >> 172.21.15.137:0/2428973735 conn(0x7f8db00f1aa0 msgr2=0x7f8db00f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:57:39.829 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:39.826+0000 7f8db6d7d700 1 -- 172.21.15.137:0/2428973735 shutdown_connections 2023-11-15T01:57:39.829 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:39.826+0000 7f8db6d7d700 1 -- 172.21.15.137:0/2428973735 wait complete. 2023-11-15T01:57:39.829 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:57:40.077 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:39 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:57:38.767730+0000 mgr.smithi055.szwngw (mgr.14164) 244 : cluster [DBG] pgmap v232: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:57:41.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:40 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:57:39.825859+0000 mon.smithi055 (mon.0) 301 : audit [DBG] from='client.? 172.21.15.137:0/2428973735' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:57:41.215 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:57:41.215 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:57:42.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:41 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:57:40.767993+0000 mgr.smithi055.szwngw (mgr.14164) 245 : cluster [DBG] pgmap v233: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:57:42.362 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:42.361+0000 7fc2fba0c700 1 -- 172.21.15.137:0/4239879113 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc2f40f5410 msgr2=0x7fc2f40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:57:42.362 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:42.361+0000 7fc2fba0c700 1 --2- 172.21.15.137:0/4239879113 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc2f40f5410 0x7fc2f40f5830 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7fc2e4009fb0 tx=0x7fc2e4009b30).stop 2023-11-15T01:57:42.363 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:42.361+0000 7fc2fba0c700 1 -- 172.21.15.137:0/4239879113 shutdown_connections 2023-11-15T01:57:42.363 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:42.361+0000 7fc2fba0c700 1 --2- 172.21.15.137:0/4239879113 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc2f40f5410 0x7fc2f40f5830 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:57:42.363 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:42.361+0000 7fc2fba0c700 1 -- 172.21.15.137:0/4239879113 >> 172.21.15.137:0/4239879113 conn(0x7fc2f40f0b70 msgr2=0x7fc2f40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:57:42.363 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:42.361+0000 7fc2fba0c700 1 -- 172.21.15.137:0/4239879113 shutdown_connections 2023-11-15T01:57:42.364 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:42.361+0000 7fc2fba0c700 1 -- 172.21.15.137:0/4239879113 wait complete. 2023-11-15T01:57:42.366 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:42.362+0000 7fc2fba0c700 1 Processor -- start 2023-11-15T01:57:42.366 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:42.362+0000 7fc2fba0c700 1 -- start start 2023-11-15T01:57:42.366 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:42.362+0000 7fc2fba0c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc2f40f5410 0x7fc2f4105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:57:42.367 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:42.362+0000 7fc2fba0c700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f4105b20 con 0x7fc2f40f5410 2023-11-15T01:57:42.367 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:42.362+0000 7fc2f97a8700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc2f40f5410 0x7fc2f4105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:57:42.367 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:42.362+0000 7fc2f97a8700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc2f40f5410 0x7fc2f4105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:46110/0 (socket says 172.21.15.137:46110) 2023-11-15T01:57:42.368 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:42.362+0000 7fc2f97a8700 1 -- 172.21.15.137:0/384598928 learned_addr learned my addr 172.21.15.137:0/384598928 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:57:42.368 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:42.363+0000 7fc2f97a8700 1 -- 172.21.15.137:0/384598928 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2e4005040 con 0x7fc2f40f5410 2023-11-15T01:57:42.368 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:42.363+0000 7fc2f97a8700 1 --2- 172.21.15.137:0/384598928 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc2f40f5410 0x7fc2f4105630 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7fc2e4014000 tx=0x7fc2e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:57:42.368 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:42.363+0000 7fc2e9ffb700 1 -- 172.21.15.137:0/384598928 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc2e40095f0 con 0x7fc2f40f5410 2023-11-15T01:57:42.368 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:42.363+0000 7fc2fba0c700 1 -- 172.21.15.137:0/384598928 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2f4105d20 con 0x7fc2f40f5410 2023-11-15T01:57:42.369 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:42.363+0000 7fc2e9ffb700 1 -- 172.21.15.137:0/384598928 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc2e4007e70 con 0x7fc2f40f5410 2023-11-15T01:57:42.369 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:42.363+0000 7fc2e9ffb700 1 -- 172.21.15.137:0/384598928 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc2e4014ea0 con 0x7fc2f40f5410 2023-11-15T01:57:42.369 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:42.363+0000 7fc2fba0c700 1 -- 172.21.15.137:0/384598928 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2f41061b0 con 0x7fc2f40f5410 2023-11-15T01:57:42.369 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:42.364+0000 7fc2fba0c700 1 -- 172.21.15.137:0/384598928 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2f4045ad0 con 0x7fc2f40f5410 2023-11-15T01:57:42.370 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:42.364+0000 7fc2e9ffb700 1 -- 172.21.15.137:0/384598928 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc2e4025070 con 0x7fc2f40f5410 2023-11-15T01:57:42.370 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:42.364+0000 7fc2e9ffb700 1 --2- 172.21.15.137:0/384598928 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fc2e002ff60 0x7fc2e0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:57:42.370 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:42.364+0000 7fc2e9ffb700 1 -- 172.21.15.137:0/384598928 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc2e4040940 con 0x7fc2f40f5410 2023-11-15T01:57:42.370 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:42.365+0000 7fc2f8fa7700 1 --2- 172.21.15.137:0/384598928 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fc2e002ff60 0x7fc2e0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:57:42.370 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:42.365+0000 7fc2f8fa7700 1 --2- 172.21.15.137:0/384598928 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fc2e002ff60 0x7fc2e0032420 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7fc2f00096a0 tx=0x7fc2f0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:57:42.371 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:42.369+0000 7fc2e9ffb700 1 -- 172.21.15.137:0/384598928 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc2e40079c0 con 0x7fc2f40f5410 2023-11-15T01:57:42.576 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:42.575+0000 7fc2fba0c700 1 -- 172.21.15.137:0/384598928 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc2f40fae60 con 0x7fc2f40f5410 2023-11-15T01:57:42.576 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:42.576+0000 7fc2e9ffb700 1 -- 172.21.15.137:0/384598928 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc2e400f300 con 0x7fc2f40f5410 2023-11-15T01:57:42.577 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:57:42.577 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:57:42.578 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:42.577+0000 7fc2fba0c700 1 -- 172.21.15.137:0/384598928 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fc2e002ff60 msgr2=0x7fc2e0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:57:42.578 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:42.577+0000 7fc2fba0c700 1 --2- 172.21.15.137:0/384598928 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fc2e002ff60 0x7fc2e0032420 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7fc2f00096a0 tx=0x7fc2f0006b40).stop 2023-11-15T01:57:42.579 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:42.577+0000 7fc2fba0c700 1 -- 172.21.15.137:0/384598928 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc2f40f5410 msgr2=0x7fc2f4105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:57:42.579 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:42.577+0000 7fc2fba0c700 1 --2- 172.21.15.137:0/384598928 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc2f40f5410 0x7fc2f4105630 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7fc2e4014000 tx=0x7fc2e400b040).stop 2023-11-15T01:57:42.579 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:42.577+0000 7fc2fba0c700 1 -- 172.21.15.137:0/384598928 shutdown_connections 2023-11-15T01:57:42.579 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:42.577+0000 7fc2fba0c700 1 --2- 172.21.15.137:0/384598928 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fc2e002ff60 0x7fc2e0032420 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:57:42.579 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:42.577+0000 7fc2fba0c700 1 --2- 172.21.15.137:0/384598928 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc2f40f5410 0x7fc2f4105630 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:57:42.580 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:42.577+0000 7fc2fba0c700 1 -- 172.21.15.137:0/384598928 >> 172.21.15.137:0/384598928 conn(0x7fc2f40f0b70 msgr2=0x7fc2f40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:57:42.580 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:42.578+0000 7fc2fba0c700 1 -- 172.21.15.137:0/384598928 shutdown_connections 2023-11-15T01:57:42.580 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:42.578+0000 7fc2fba0c700 1 -- 172.21.15.137:0/384598928 wait complete. 2023-11-15T01:57:42.581 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:57:43.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:42 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:57:42.576732+0000 mon.smithi055 (mon.0) 302 : audit [DBG] from='client.? 172.21.15.137:0/384598928' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:57:43.961 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:57:43.962 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:57:44.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:43 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:57:42.768345+0000 mgr.smithi055.szwngw (mgr.14164) 246 : cluster [DBG] pgmap v234: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:57:44.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:43 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:57:42.855661+0000 mon.smithi055 (mon.0) 303 : audit [INF] from='mgr.14164 172.21.15.55:0/59463371' entity='mgr.smithi055.szwngw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi055.szwngw/mirror_snapshot_schedule"}]: dispatch 2023-11-15T01:57:44.329 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:43 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:57:42.857914+0000 mon.smithi055 (mon.0) 304 : audit [INF] from='mgr.14164 172.21.15.55:0/59463371' entity='mgr.smithi055.szwngw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi055.szwngw/trash_purge_schedule"}]: dispatch 2023-11-15T01:57:45.069 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:45.066+0000 7f191347a700 1 -- 172.21.15.137:0/2427509627 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f190c0f6510 msgr2=0x7f190c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:57:45.069 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:45.066+0000 7f191347a700 1 --2- 172.21.15.137:0/2427509627 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f190c0f6510 0x7f190c0f6930 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7f18fc009fb0 tx=0x7f18fc009b30).stop 2023-11-15T01:57:45.069 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:45.066+0000 7f191347a700 1 -- 172.21.15.137:0/2427509627 shutdown_connections 2023-11-15T01:57:45.069 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:45.066+0000 7f191347a700 1 --2- 172.21.15.137:0/2427509627 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f190c0f6510 0x7f190c0f6930 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:57:45.070 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:45.066+0000 7f191347a700 1 -- 172.21.15.137:0/2427509627 >> 172.21.15.137:0/2427509627 conn(0x7f190c0f1c70 msgr2=0x7f190c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:57:45.070 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:45.067+0000 7f191347a700 1 -- 172.21.15.137:0/2427509627 shutdown_connections 2023-11-15T01:57:45.070 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:45.067+0000 7f191347a700 1 -- 172.21.15.137:0/2427509627 wait complete. 2023-11-15T01:57:45.070 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:45.067+0000 7f191347a700 1 Processor -- start 2023-11-15T01:57:45.070 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:45.067+0000 7f191347a700 1 -- start start 2023-11-15T01:57:45.071 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:45.067+0000 7f191347a700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f190c0f6510 0x7f190c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:57:45.071 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:45.067+0000 7f191347a700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f190c106c20 con 0x7f190c0f6510 2023-11-15T01:57:45.071 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:45.067+0000 7f1911216700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f190c0f6510 0x7f190c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:57:45.071 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:45.068+0000 7f1911216700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f190c0f6510 0x7f190c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:46138/0 (socket says 172.21.15.137:46138) 2023-11-15T01:57:45.071 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:45.068+0000 7f1911216700 1 -- 172.21.15.137:0/2807293707 learned_addr learned my addr 172.21.15.137:0/2807293707 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:57:45.072 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:45.068+0000 7f1911216700 1 -- 172.21.15.137:0/2807293707 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18fc005040 con 0x7f190c0f6510 2023-11-15T01:57:45.072 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:45.068+0000 7f1911216700 1 --2- 172.21.15.137:0/2807293707 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f190c0f6510 0x7f190c106730 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7f18fc013510 tx=0x7f18fc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:57:45.072 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:45.068+0000 7f1901ffb700 1 -- 172.21.15.137:0/2807293707 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f18fc009320 con 0x7f190c0f6510 2023-11-15T01:57:45.072 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:45.068+0000 7f191347a700 1 -- 172.21.15.137:0/2807293707 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f190c106e20 con 0x7f190c0f6510 2023-11-15T01:57:45.072 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:45.068+0000 7f1901ffb700 1 -- 172.21.15.137:0/2807293707 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f18fc013f00 con 0x7f190c0f6510 2023-11-15T01:57:45.073 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:45.068+0000 7f1901ffb700 1 -- 172.21.15.137:0/2807293707 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f18fc014d90 con 0x7f190c0f6510 2023-11-15T01:57:45.073 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:45.068+0000 7f191347a700 1 -- 172.21.15.137:0/2807293707 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f190c1072b0 con 0x7f190c0f6510 2023-11-15T01:57:45.074 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:45.069+0000 7f1901ffb700 1 -- 172.21.15.137:0/2807293707 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f18fc0095f0 con 0x7f190c0f6510 2023-11-15T01:57:45.074 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:45.069+0000 7f191347a700 1 -- 172.21.15.137:0/2807293707 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f190c045ad0 con 0x7f190c0f6510 2023-11-15T01:57:45.074 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:45.069+0000 7f1901ffb700 1 --2- 172.21.15.137:0/2807293707 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f18f802ff60 0x7f18f8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:57:45.074 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:45.069+0000 7f1901ffb700 1 -- 172.21.15.137:0/2807293707 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f18fc0410a0 con 0x7f190c0f6510 2023-11-15T01:57:45.075 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:45.070+0000 7f1910a15700 1 --2- 172.21.15.137:0/2807293707 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f18f802ff60 0x7f18f8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:57:45.075 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:45.070+0000 7f1910a15700 1 --2- 172.21.15.137:0/2807293707 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f18f802ff60 0x7f18f8032420 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f19080096a0 tx=0x7f1908006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:57:45.075 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:45.074+0000 7f1901ffb700 1 -- 172.21.15.137:0/2807293707 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f18fc014420 con 0x7f190c0f6510 2023-11-15T01:57:45.280 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:45.280+0000 7f191347a700 1 -- 172.21.15.137:0/2807293707 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f190c0fbf60 con 0x7f190c0f6510 2023-11-15T01:57:45.281 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:45.280+0000 7f1901ffb700 1 -- 172.21.15.137:0/2807293707 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f18fc010230 con 0x7f190c0f6510 2023-11-15T01:57:45.282 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:57:45.282 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:57:45.283 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:45.282+0000 7f191347a700 1 -- 172.21.15.137:0/2807293707 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f18f802ff60 msgr2=0x7f18f8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:57:45.283 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:45.282+0000 7f191347a700 1 --2- 172.21.15.137:0/2807293707 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f18f802ff60 0x7f18f8032420 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f19080096a0 tx=0x7f1908006b40).stop 2023-11-15T01:57:45.284 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:45.282+0000 7f191347a700 1 -- 172.21.15.137:0/2807293707 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f190c0f6510 msgr2=0x7f190c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:57:45.284 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:45.282+0000 7f191347a700 1 --2- 172.21.15.137:0/2807293707 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f190c0f6510 0x7f190c106730 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7f18fc013510 tx=0x7f18fc00b040).stop 2023-11-15T01:57:45.284 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:45.282+0000 7f191347a700 1 -- 172.21.15.137:0/2807293707 shutdown_connections 2023-11-15T01:57:45.284 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:45.282+0000 7f191347a700 1 --2- 172.21.15.137:0/2807293707 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f18f802ff60 0x7f18f8032420 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:57:45.284 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:45.282+0000 7f191347a700 1 --2- 172.21.15.137:0/2807293707 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f190c0f6510 0x7f190c106730 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:57:45.285 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:45.282+0000 7f191347a700 1 -- 172.21.15.137:0/2807293707 >> 172.21.15.137:0/2807293707 conn(0x7f190c0f1c70 msgr2=0x7f190c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:57:45.285 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:45.282+0000 7f191347a700 1 -- 172.21.15.137:0/2807293707 shutdown_connections 2023-11-15T01:57:45.285 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:45.282+0000 7f191347a700 1 -- 172.21.15.137:0/2807293707 wait complete. 2023-11-15T01:57:45.285 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:57:46.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:45 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cephadm 2023-11-15T01:57:44.369940+0000 mgr.smithi055.szwngw (mgr.14164) 247 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi055 ... 2023-11-15T01:57:46.329 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:45 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: Verifying port 9100 ... 2023-11-15T01:57:46.329 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:45 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: Non-zero exit code 1 from systemctl start ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7@node-exporter.smithi055 2023-11-15T01:57:46.329 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:45 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: systemctl: stderr Job for ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7@node-exporter.smithi055.service failed because a timeout was exceeded. 2023-11-15T01:57:46.329 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:45 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: systemctl: stderr See "systemctl status ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7@node-exporter.smithi055.service" and "journalctl -xe" for details. 2023-11-15T01:57:46.329 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:45 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: Traceback (most recent call last): 2023-11-15T01:57:46.330 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:45 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: File "/var/lib/ceph/0cb4fc0c-8359-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-11-15T01:57:46.330 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:45 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: main() 2023-11-15T01:57:46.330 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:45 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: File "/var/lib/ceph/0cb4fc0c-8359-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-11-15T01:57:46.330 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:45 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: r = ctx.func(ctx) 2023-11-15T01:57:46.330 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:45 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: File "/var/lib/ceph/0cb4fc0c-8359-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-11-15T01:57:46.331 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:45 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: return func(ctx) 2023-11-15T01:57:46.331 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:45 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: File "/var/lib/ceph/0cb4fc0c-8359-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2023-11-15T01:57:46.331 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:45 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: ports=daemon_ports) 2023-11-15T01:57:46.331 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:45 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: File "/var/lib/ceph/0cb4fc0c-8359-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2023-11-15T01:57:46.331 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:45 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: c, osd_fsid=osd_fsid, ports=ports) 2023-11-15T01:57:46.332 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:45 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: File "/var/lib/ceph/0cb4fc0c-8359-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2023-11-15T01:57:46.332 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:45 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2023-11-15T01:57:46.332 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:45 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: File "/var/lib/ceph/0cb4fc0c-8359-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2023-11-15T01:57:46.332 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:45 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2023-11-15T01:57:46.333 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:45 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: RuntimeError: Failed command: systemctl start ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7@node-exporter.smithi055 2023-11-15T01:57:46.333 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:45 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: Traceback (most recent call last): 2023-11-15T01:57:46.333 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:45 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2023-11-15T01:57:46.333 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:45 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: yield (conn, connr) 2023-11-15T01:57:46.333 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:45 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2023-11-15T01:57:46.333 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:45 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: code, '\n'.join(err))) 2023-11-15T01:57:46.334 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:45 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi055 ... 2023-11-15T01:57:46.334 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:45 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: Verifying port 9100 ... 2023-11-15T01:57:46.334 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:45 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: Non-zero exit code 1 from systemctl start ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7@node-exporter.smithi055 2023-11-15T01:57:46.334 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:45 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: systemctl: stderr Job for ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7@node-exporter.smithi055.service failed because a timeout was exceeded. 2023-11-15T01:57:46.335 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:45 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: systemctl: stderr See "systemctl status ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7@node-exporter.smithi055.service" and "journalctl -xe" for details. 2023-11-15T01:57:46.335 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:45 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: Traceback (most recent call last): 2023-11-15T01:57:46.335 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:45 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: File "/var/lib/ceph/0cb4fc0c-8359-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-11-15T01:57:46.335 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:45 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: main() 2023-11-15T01:57:46.335 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:45 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: File "/var/lib/ceph/0cb4fc0c-8359-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-11-15T01:57:46.335 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:45 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: r = ctx.func(ctx) 2023-11-15T01:57:46.336 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:45 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: File "/var/lib/ceph/0cb4fc0c-8359-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-11-15T01:57:46.336 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:45 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: return func(ctx) 2023-11-15T01:57:46.336 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:45 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: File "/var/lib/ceph/0cb4fc0c-8359-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2023-11-15T01:57:46.336 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:45 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: ports=daemon_ports) 2023-11-15T01:57:46.336 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:45 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: File "/var/lib/ceph/0cb4fc0c-8359-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2023-11-15T01:57:46.337 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:45 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: c, osd_fsid=osd_fsid, ports=ports) 2023-11-15T01:57:46.337 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:45 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: File "/var/lib/ceph/0cb4fc0c-8359-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2023-11-15T01:57:46.337 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:45 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2023-11-15T01:57:46.337 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:45 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: File "/var/lib/ceph/0cb4fc0c-8359-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2023-11-15T01:57:46.337 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:45 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2023-11-15T01:57:46.338 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:45 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: RuntimeError: Failed command: systemctl start ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7@node-exporter.smithi055 2023-11-15T01:57:46.338 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:45 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cephadm 2023-11-15T01:57:44.375820+0000 mgr.smithi055.szwngw (mgr.14164) 248 : cephadm [INF] Deploying daemon prometheus.smithi055 on smithi055 2023-11-15T01:57:46.338 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:45 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:57:44.768784+0000 mgr.smithi055.szwngw (mgr.14164) 249 : cluster [DBG] pgmap v235: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:57:46.338 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:45 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:57:45.281611+0000 mon.smithi055 (mon.0) 305 : audit [DBG] from='client.? 172.21.15.137:0/2807293707' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:57:46.664 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:57:46.665 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:57:47.792 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:47.790+0000 7fb6a8fbb700 1 -- 172.21.15.137:0/2860836580 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb6a4060280 msgr2=0x7fb6a40f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:57:47.792 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:47.790+0000 7fb6a8fbb700 1 --2- 172.21.15.137:0/2860836580 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb6a4060280 0x7fb6a40f90f0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7fb694009fb0 tx=0x7fb694009b30).stop 2023-11-15T01:57:47.792 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:47.790+0000 7fb6a8fbb700 1 -- 172.21.15.137:0/2860836580 shutdown_connections 2023-11-15T01:57:47.792 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:47.790+0000 7fb6a8fbb700 1 --2- 172.21.15.137:0/2860836580 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb6a4060280 0x7fb6a40f90f0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:57:47.793 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:47.790+0000 7fb6a8fbb700 1 -- 172.21.15.137:0/2860836580 >> 172.21.15.137:0/2860836580 conn(0x7fb6a40f0b70 msgr2=0x7fb6a40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:57:47.793 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:47.791+0000 7fb6a8fbb700 1 -- 172.21.15.137:0/2860836580 shutdown_connections 2023-11-15T01:57:47.793 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:47.791+0000 7fb6a8fbb700 1 -- 172.21.15.137:0/2860836580 wait complete. 2023-11-15T01:57:47.793 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:47.791+0000 7fb6a8fbb700 1 Processor -- start 2023-11-15T01:57:47.793 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:47.791+0000 7fb6a8fbb700 1 -- start start 2023-11-15T01:57:47.795 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:47.791+0000 7fb6a8fbb700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb6a4060280 0x7fb6a41066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:57:47.795 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:47.791+0000 7fb6a8fbb700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a4106bd0 con 0x7fb6a4060280 2023-11-15T01:57:47.795 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:47.792+0000 7fb6a259c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb6a4060280 0x7fb6a41066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:57:47.796 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:47.792+0000 7fb6a259c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb6a4060280 0x7fb6a41066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:46162/0 (socket says 172.21.15.137:46162) 2023-11-15T01:57:47.796 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:47.792+0000 7fb6a259c700 1 -- 172.21.15.137:0/477135939 learned_addr learned my addr 172.21.15.137:0/477135939 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:57:47.796 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:47.792+0000 7fb6a259c700 1 -- 172.21.15.137:0/477135939 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb694005040 con 0x7fb6a4060280 2023-11-15T01:57:47.796 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:47.792+0000 7fb6a259c700 1 --2- 172.21.15.137:0/477135939 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb6a4060280 0x7fb6a41066e0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7fb694013510 tx=0x7fb69400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:57:47.796 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:47.792+0000 7fb692ffd700 1 -- 172.21.15.137:0/477135939 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb694009320 con 0x7fb6a4060280 2023-11-15T01:57:47.797 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:47.792+0000 7fb6a8fbb700 1 -- 172.21.15.137:0/477135939 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6a4106dd0 con 0x7fb6a4060280 2023-11-15T01:57:47.797 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:47.792+0000 7fb692ffd700 1 -- 172.21.15.137:0/477135939 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb694013f00 con 0x7fb6a4060280 2023-11-15T01:57:47.797 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:47.793+0000 7fb692ffd700 1 -- 172.21.15.137:0/477135939 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb694014dc0 con 0x7fb6a4060280 2023-11-15T01:57:47.797 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:47.793+0000 7fb6a8fbb700 1 -- 172.21.15.137:0/477135939 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6a4107260 con 0x7fb6a4060280 2023-11-15T01:57:47.797 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:47.793+0000 7fb692ffd700 1 -- 172.21.15.137:0/477135939 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb6940095f0 con 0x7fb6a4060280 2023-11-15T01:57:47.797 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:47.793+0000 7fb6a8fbb700 1 -- 172.21.15.137:0/477135939 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6a4045ad0 con 0x7fb6a4060280 2023-11-15T01:57:47.798 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:47.793+0000 7fb692ffd700 1 --2- 172.21.15.137:0/477135939 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fb68c02ff60 0x7fb68c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:57:47.798 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:47.793+0000 7fb692ffd700 1 -- 172.21.15.137:0/477135939 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb694041030 con 0x7fb6a4060280 2023-11-15T01:57:47.798 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:47.794+0000 7fb6a1d9b700 1 --2- 172.21.15.137:0/477135939 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fb68c02ff60 0x7fb68c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:57:47.798 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:47.794+0000 7fb6a1d9b700 1 --2- 172.21.15.137:0/477135939 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fb68c02ff60 0x7fb68c032420 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7fb6980096a0 tx=0x7fb698006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:57:47.799 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:47.798+0000 7fb692ffd700 1 -- 172.21.15.137:0/477135939 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb694014420 con 0x7fb6a4060280 2023-11-15T01:57:48.008 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:48.007+0000 7fb6a8fbb700 1 -- 172.21.15.137:0/477135939 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb6a4108110 con 0x7fb6a4060280 2023-11-15T01:57:48.008 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:48.007+0000 7fb692ffd700 1 -- 172.21.15.137:0/477135939 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb694010230 con 0x7fb6a4060280 2023-11-15T01:57:48.009 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:57:48.009 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:57:48.011 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:48.009+0000 7fb6a8fbb700 1 -- 172.21.15.137:0/477135939 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fb68c02ff60 msgr2=0x7fb68c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:57:48.011 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:48.009+0000 7fb6a8fbb700 1 --2- 172.21.15.137:0/477135939 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fb68c02ff60 0x7fb68c032420 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7fb6980096a0 tx=0x7fb698006b40).stop 2023-11-15T01:57:48.011 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:48.009+0000 7fb6a8fbb700 1 -- 172.21.15.137:0/477135939 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb6a4060280 msgr2=0x7fb6a41066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:57:48.011 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:48.009+0000 7fb6a8fbb700 1 --2- 172.21.15.137:0/477135939 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb6a4060280 0x7fb6a41066e0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7fb694013510 tx=0x7fb69400b040).stop 2023-11-15T01:57:48.012 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:48.009+0000 7fb6a8fbb700 1 -- 172.21.15.137:0/477135939 shutdown_connections 2023-11-15T01:57:48.012 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:48.009+0000 7fb6a8fbb700 1 --2- 172.21.15.137:0/477135939 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fb68c02ff60 0x7fb68c032420 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:57:48.012 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:48.009+0000 7fb6a8fbb700 1 --2- 172.21.15.137:0/477135939 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb6a4060280 0x7fb6a41066e0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:57:48.013 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:48.009+0000 7fb6a8fbb700 1 -- 172.21.15.137:0/477135939 >> 172.21.15.137:0/477135939 conn(0x7fb6a40f0b70 msgr2=0x7fb6a40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:57:48.013 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:48.009+0000 7fb6a8fbb700 1 -- 172.21.15.137:0/477135939 shutdown_connections 2023-11-15T01:57:48.013 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:48.009+0000 7fb6a8fbb700 1 -- 172.21.15.137:0/477135939 wait complete. 2023-11-15T01:57:48.014 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:57:48.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:47 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:57:46.769073+0000 mgr.smithi055.szwngw (mgr.14164) 250 : cluster [DBG] pgmap v236: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:57:49.327 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:48 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:57:48.008635+0000 mon.smithi055 (mon.0) 306 : audit [DBG] from='client.? 172.21.15.137:0/477135939' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:57:49.382 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:57:49.383 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:57:50.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:49 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:57:48.769366+0000 mgr.smithi055.szwngw (mgr.14164) 251 : cluster [DBG] pgmap v237: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:57:50.525 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:50.522+0000 7f0257231700 1 -- 172.21.15.137:0/1800961915 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0250060280 msgr2=0x7f02500f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:57:50.526 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:50.522+0000 7f0257231700 1 --2- 172.21.15.137:0/1800961915 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0250060280 0x7f02500f90f0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7f0244009fb0 tx=0x7f0244009b30).stop 2023-11-15T01:57:50.526 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:50.522+0000 7f0257231700 1 -- 172.21.15.137:0/1800961915 shutdown_connections 2023-11-15T01:57:50.527 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:50.522+0000 7f0257231700 1 --2- 172.21.15.137:0/1800961915 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0250060280 0x7f02500f90f0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:57:50.527 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:50.522+0000 7f0257231700 1 -- 172.21.15.137:0/1800961915 >> 172.21.15.137:0/1800961915 conn(0x7f02500f0b70 msgr2=0x7f02500f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:57:50.527 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:50.522+0000 7f0257231700 1 -- 172.21.15.137:0/1800961915 shutdown_connections 2023-11-15T01:57:50.528 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:50.522+0000 7f0257231700 1 -- 172.21.15.137:0/1800961915 wait complete. 2023-11-15T01:57:50.528 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:50.523+0000 7f0257231700 1 Processor -- start 2023-11-15T01:57:50.528 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:50.523+0000 7f0257231700 1 -- start start 2023-11-15T01:57:50.529 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:50.523+0000 7f0257231700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0250060280 0x7f02501066c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:57:50.529 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:50.523+0000 7f0257231700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0250106bb0 con 0x7f0250060280 2023-11-15T01:57:50.529 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:50.524+0000 7f0254fcd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0250060280 0x7f02501066c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:57:50.530 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:50.524+0000 7f0254fcd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0250060280 0x7f02501066c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:36070/0 (socket says 172.21.15.137:36070) 2023-11-15T01:57:50.531 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:50.524+0000 7f0254fcd700 1 -- 172.21.15.137:0/1097207623 learned_addr learned my addr 172.21.15.137:0/1097207623 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:57:50.531 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:50.524+0000 7f0254fcd700 1 -- 172.21.15.137:0/1097207623 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0244005040 con 0x7f0250060280 2023-11-15T01:57:50.532 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:50.524+0000 7f0254fcd700 1 --2- 172.21.15.137:0/1097207623 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0250060280 0x7f02501066c0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7f0244013510 tx=0x7f024400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:57:50.532 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:50.524+0000 7f024d7fa700 1 -- 172.21.15.137:0/1097207623 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0244009320 con 0x7f0250060280 2023-11-15T01:57:50.532 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:50.524+0000 7f0257231700 1 -- 172.21.15.137:0/1097207623 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0250106db0 con 0x7f0250060280 2023-11-15T01:57:50.533 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:50.524+0000 7f024d7fa700 1 -- 172.21.15.137:0/1097207623 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0244013f00 con 0x7f0250060280 2023-11-15T01:57:50.533 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:50.524+0000 7f024d7fa700 1 -- 172.21.15.137:0/1097207623 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0244014dc0 con 0x7f0250060280 2023-11-15T01:57:50.533 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:50.524+0000 7f0257231700 1 -- 172.21.15.137:0/1097207623 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0250107240 con 0x7f0250060280 2023-11-15T01:57:50.535 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:50.525+0000 7f024d7fa700 1 -- 172.21.15.137:0/1097207623 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f024401d3d0 con 0x7f0250060280 2023-11-15T01:57:50.536 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:50.525+0000 7f0257231700 1 -- 172.21.15.137:0/1097207623 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0250045ad0 con 0x7f0250060280 2023-11-15T01:57:50.536 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:50.525+0000 7f024d7fa700 1 --2- 172.21.15.137:0/1097207623 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f024002ff60 0x7f0240032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:57:50.537 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:50.525+0000 7f024d7fa700 1 -- 172.21.15.137:0/1097207623 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f02440410f0 con 0x7f0250060280 2023-11-15T01:57:50.537 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:50.526+0000 7f024ffff700 1 --2- 172.21.15.137:0/1097207623 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f024002ff60 0x7f0240032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:57:50.537 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:50.526+0000 7f024ffff700 1 --2- 172.21.15.137:0/1097207623 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f024002ff60 0x7f0240032420 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f023c0096a0 tx=0x7f023c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:57:50.538 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:50.530+0000 7f024d7fa700 1 -- 172.21.15.137:0/1097207623 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f02440126f0 con 0x7f0250060280 2023-11-15T01:57:50.740 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:50.739+0000 7f0257231700 1 -- 172.21.15.137:0/1097207623 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f02501080f0 con 0x7f0250060280 2023-11-15T01:57:50.740 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:50.740+0000 7f024d7fa700 1 -- 172.21.15.137:0/1097207623 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0244012330 con 0x7f0250060280 2023-11-15T01:57:50.741 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:57:50.741 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:57:50.742 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:50.741+0000 7f0257231700 1 -- 172.21.15.137:0/1097207623 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f024002ff60 msgr2=0x7f0240032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:57:50.742 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:50.741+0000 7f0257231700 1 --2- 172.21.15.137:0/1097207623 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f024002ff60 0x7f0240032420 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f023c0096a0 tx=0x7f023c006b40).stop 2023-11-15T01:57:50.742 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:50.741+0000 7f0257231700 1 -- 172.21.15.137:0/1097207623 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0250060280 msgr2=0x7f02501066c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:57:50.743 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:50.741+0000 7f0257231700 1 --2- 172.21.15.137:0/1097207623 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0250060280 0x7f02501066c0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7f0244013510 tx=0x7f024400b040).stop 2023-11-15T01:57:50.743 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:50.741+0000 7f0257231700 1 -- 172.21.15.137:0/1097207623 shutdown_connections 2023-11-15T01:57:50.743 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:50.741+0000 7f0257231700 1 --2- 172.21.15.137:0/1097207623 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f024002ff60 0x7f0240032420 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:57:50.743 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:50.741+0000 7f0257231700 1 --2- 172.21.15.137:0/1097207623 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0250060280 0x7f02501066c0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:57:50.743 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:50.741+0000 7f0257231700 1 -- 172.21.15.137:0/1097207623 >> 172.21.15.137:0/1097207623 conn(0x7f02500f0b70 msgr2=0x7f02500f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:57:50.744 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:50.741+0000 7f0257231700 1 -- 172.21.15.137:0/1097207623 shutdown_connections 2023-11-15T01:57:50.744 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:50.741+0000 7f0257231700 1 -- 172.21.15.137:0/1097207623 wait complete. 2023-11-15T01:57:50.744 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:57:51.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:51 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:57:50.740934+0000 mon.smithi055 (mon.0) 307 : audit [DBG] from='client.? 172.21.15.137:0/1097207623' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:57:51.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:51 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:57:50.769663+0000 mgr.smithi055.szwngw (mgr.14164) 252 : cluster [DBG] pgmap v238: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:57:52.123 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:57:52.124 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:57:53.312 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:53.309+0000 7f63069e9700 1 -- 172.21.15.137:0/2416703833 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f63000f5410 msgr2=0x7f63000f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:57:53.312 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:53.309+0000 7f63069e9700 1 --2- 172.21.15.137:0/2416703833 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f63000f5410 0x7f63000f5830 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7f62f0009fb0 tx=0x7f62f0009b30).stop 2023-11-15T01:57:53.312 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:53.309+0000 7f63069e9700 1 -- 172.21.15.137:0/2416703833 shutdown_connections 2023-11-15T01:57:53.312 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:53.309+0000 7f63069e9700 1 --2- 172.21.15.137:0/2416703833 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f63000f5410 0x7f63000f5830 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:57:53.313 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:53.309+0000 7f63069e9700 1 -- 172.21.15.137:0/2416703833 >> 172.21.15.137:0/2416703833 conn(0x7f63000f0b70 msgr2=0x7f63000f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:57:53.313 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:53.309+0000 7f63069e9700 1 -- 172.21.15.137:0/2416703833 shutdown_connections 2023-11-15T01:57:53.313 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:53.309+0000 7f63069e9700 1 -- 172.21.15.137:0/2416703833 wait complete. 2023-11-15T01:57:53.313 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:53.310+0000 7f63069e9700 1 Processor -- start 2023-11-15T01:57:53.314 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:53.310+0000 7f63069e9700 1 -- start start 2023-11-15T01:57:53.314 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:53.310+0000 7f63069e9700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f63000f5410 0x7f6300106710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:57:53.314 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:53.310+0000 7f63069e9700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6300106c00 con 0x7f63000f5410 2023-11-15T01:57:53.314 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:53.310+0000 7f62fffff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f63000f5410 0x7f6300106710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:57:53.314 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:53.310+0000 7f62fffff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f63000f5410 0x7f6300106710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:36084/0 (socket says 172.21.15.137:36084) 2023-11-15T01:57:53.315 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:53.310+0000 7f62fffff700 1 -- 172.21.15.137:0/1912056628 learned_addr learned my addr 172.21.15.137:0/1912056628 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:57:53.315 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:53.311+0000 7f62fffff700 1 -- 172.21.15.137:0/1912056628 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62f0005040 con 0x7f63000f5410 2023-11-15T01:57:53.315 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:53.311+0000 7f62fffff700 1 --2- 172.21.15.137:0/1912056628 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f63000f5410 0x7f6300106710 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7f62f0013510 tx=0x7f62f000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:57:53.315 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:53.311+0000 7f62fcff9700 1 -- 172.21.15.137:0/1912056628 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f62f0009320 con 0x7f63000f5410 2023-11-15T01:57:53.315 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:53.311+0000 7f63069e9700 1 -- 172.21.15.137:0/1912056628 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6300106e00 con 0x7f63000f5410 2023-11-15T01:57:53.316 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:53.311+0000 7f62fcff9700 1 -- 172.21.15.137:0/1912056628 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f62f0013f00 con 0x7f63000f5410 2023-11-15T01:57:53.316 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:53.311+0000 7f62fcff9700 1 -- 172.21.15.137:0/1912056628 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f62f0014d90 con 0x7f63000f5410 2023-11-15T01:57:53.316 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:53.311+0000 7f63069e9700 1 -- 172.21.15.137:0/1912056628 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6300107290 con 0x7f63000f5410 2023-11-15T01:57:53.317 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:53.312+0000 7f62fcff9700 1 -- 172.21.15.137:0/1912056628 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f62f00095f0 con 0x7f63000f5410 2023-11-15T01:57:53.317 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:53.312+0000 7f63069e9700 1 -- 172.21.15.137:0/1912056628 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6300045ad0 con 0x7f63000f5410 2023-11-15T01:57:53.317 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:53.312+0000 7f62fcff9700 1 --2- 172.21.15.137:0/1912056628 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f62e802ff60 0x7f62e8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:57:53.318 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:53.312+0000 7f62fcff9700 1 -- 172.21.15.137:0/1912056628 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f62f0041000 con 0x7f63000f5410 2023-11-15T01:57:53.318 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:53.312+0000 7f62ff7fe700 1 --2- 172.21.15.137:0/1912056628 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f62e802ff60 0x7f62e8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:57:53.318 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:53.313+0000 7f62ff7fe700 1 --2- 172.21.15.137:0/1912056628 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f62e802ff60 0x7f62e8032420 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f62f40096a0 tx=0x7f62f4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:57:53.318 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:53.317+0000 7f62fcff9700 1 -- 172.21.15.137:0/1912056628 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f62f0014420 con 0x7f63000f5410 2023-11-15T01:57:53.527 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:53.526+0000 7f63069e9700 1 -- 172.21.15.137:0/1912056628 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6300108140 con 0x7f63000f5410 2023-11-15T01:57:53.527 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:53.527+0000 7f62fcff9700 1 -- 172.21.15.137:0/1912056628 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f62f001d7f0 con 0x7f63000f5410 2023-11-15T01:57:53.528 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:57:53.528 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:57:53.529 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:53.528+0000 7f63069e9700 1 -- 172.21.15.137:0/1912056628 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f62e802ff60 msgr2=0x7f62e8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:57:53.529 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:53.528+0000 7f63069e9700 1 --2- 172.21.15.137:0/1912056628 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f62e802ff60 0x7f62e8032420 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f62f40096a0 tx=0x7f62f4006b40).stop 2023-11-15T01:57:53.529 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:53.528+0000 7f63069e9700 1 -- 172.21.15.137:0/1912056628 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f63000f5410 msgr2=0x7f6300106710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:57:53.530 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:53.528+0000 7f63069e9700 1 --2- 172.21.15.137:0/1912056628 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f63000f5410 0x7f6300106710 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7f62f0013510 tx=0x7f62f000b040).stop 2023-11-15T01:57:53.530 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:53.529+0000 7f63069e9700 1 -- 172.21.15.137:0/1912056628 shutdown_connections 2023-11-15T01:57:53.530 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:53.529+0000 7f63069e9700 1 --2- 172.21.15.137:0/1912056628 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f62e802ff60 0x7f62e8032420 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:57:53.530 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:53.529+0000 7f63069e9700 1 --2- 172.21.15.137:0/1912056628 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f63000f5410 0x7f6300106710 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:57:53.530 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:53.529+0000 7f63069e9700 1 -- 172.21.15.137:0/1912056628 >> 172.21.15.137:0/1912056628 conn(0x7f63000f0b70 msgr2=0x7f630005fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:57:53.531 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:53.529+0000 7f63069e9700 1 -- 172.21.15.137:0/1912056628 shutdown_connections 2023-11-15T01:57:53.531 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:53.529+0000 7f63069e9700 1 -- 172.21.15.137:0/1912056628 wait complete. 2023-11-15T01:57:53.531 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:57:54.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:53 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:57:52.769965+0000 mgr.smithi055.szwngw (mgr.14164) 253 : cluster [DBG] pgmap v239: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:57:54.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:53 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:57:53.528006+0000 mon.smithi055 (mon.0) 308 : audit [DBG] from='client.? 172.21.15.137:0/1912056628' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:57:54.937 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:57:54.938 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:57:56.128 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:56.125+0000 7fd5bec9b700 1 -- 172.21.15.137:0/1118139794 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd5b80f5410 msgr2=0x7fd5b80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:57:56.128 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:56.125+0000 7fd5bec9b700 1 --2- 172.21.15.137:0/1118139794 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd5b80f5410 0x7fd5b80f5830 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7fd5ac009fb0 tx=0x7fd5ac009b30).stop 2023-11-15T01:57:56.129 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:56.125+0000 7fd5bec9b700 1 -- 172.21.15.137:0/1118139794 shutdown_connections 2023-11-15T01:57:56.129 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:56.125+0000 7fd5bec9b700 1 --2- 172.21.15.137:0/1118139794 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd5b80f5410 0x7fd5b80f5830 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:57:56.129 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:56.125+0000 7fd5bec9b700 1 -- 172.21.15.137:0/1118139794 >> 172.21.15.137:0/1118139794 conn(0x7fd5b80f0b70 msgr2=0x7fd5b80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:57:56.129 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:56.125+0000 7fd5bec9b700 1 -- 172.21.15.137:0/1118139794 shutdown_connections 2023-11-15T01:57:56.129 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:56.125+0000 7fd5bec9b700 1 -- 172.21.15.137:0/1118139794 wait complete. 2023-11-15T01:57:56.130 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:56.126+0000 7fd5bec9b700 1 Processor -- start 2023-11-15T01:57:56.130 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:56.126+0000 7fd5bec9b700 1 -- start start 2023-11-15T01:57:56.130 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:56.126+0000 7fd5bec9b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd5b80f5410 0x7fd5b8109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:57:56.130 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:56.126+0000 7fd5bec9b700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5b8060240 con 0x7fd5b80f5410 2023-11-15T01:57:56.130 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:56.126+0000 7fd5bca37700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd5b80f5410 0x7fd5b8109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:57:56.131 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:56.127+0000 7fd5bca37700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd5b80f5410 0x7fd5b8109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:36116/0 (socket says 172.21.15.137:36116) 2023-11-15T01:57:56.131 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:56.127+0000 7fd5bca37700 1 -- 172.21.15.137:0/1637068906 learned_addr learned my addr 172.21.15.137:0/1637068906 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:57:56.132 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:56.127+0000 7fd5bca37700 1 -- 172.21.15.137:0/1637068906 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5ac005040 con 0x7fd5b80f5410 2023-11-15T01:57:56.132 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:56.127+0000 7fd5bca37700 1 --2- 172.21.15.137:0/1637068906 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd5b80f5410 0x7fd5b8109430 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7fd5ac013510 tx=0x7fd5ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:57:56.133 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:56.127+0000 7fd5b57fa700 1 -- 172.21.15.137:0/1637068906 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd5ac009320 con 0x7fd5b80f5410 2023-11-15T01:57:56.133 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:56.127+0000 7fd5bec9b700 1 -- 172.21.15.137:0/1637068906 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5b8060440 con 0x7fd5b80f5410 2023-11-15T01:57:56.133 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:56.127+0000 7fd5b57fa700 1 -- 172.21.15.137:0/1637068906 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd5ac013f00 con 0x7fd5b80f5410 2023-11-15T01:57:56.133 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:56.127+0000 7fd5b57fa700 1 -- 172.21.15.137:0/1637068906 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd5ac014d90 con 0x7fd5b80f5410 2023-11-15T01:57:56.133 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:56.127+0000 7fd5bec9b700 1 -- 172.21.15.137:0/1637068906 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5b8109ae0 con 0x7fd5b80f5410 2023-11-15T01:57:56.134 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:56.128+0000 7fd5bec9b700 1 -- 172.21.15.137:0/1637068906 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5b8045ad0 con 0x7fd5b80f5410 2023-11-15T01:57:56.134 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:56.128+0000 7fd5b57fa700 1 -- 172.21.15.137:0/1637068906 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd5ac0095f0 con 0x7fd5b80f5410 2023-11-15T01:57:56.134 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:56.128+0000 7fd5b57fa700 1 --2- 172.21.15.137:0/1637068906 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fd5a802ff60 0x7fd5a8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:57:56.134 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:56.128+0000 7fd5b57fa700 1 -- 172.21.15.137:0/1637068906 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd5ac040fe0 con 0x7fd5b80f5410 2023-11-15T01:57:56.134 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:56.129+0000 7fd5b7fff700 1 --2- 172.21.15.137:0/1637068906 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fd5a802ff60 0x7fd5a8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:57:56.135 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:56.129+0000 7fd5b7fff700 1 --2- 172.21.15.137:0/1637068906 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fd5a802ff60 0x7fd5a8032420 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7fd5a40096a0 tx=0x7fd5a4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:57:56.135 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:56.133+0000 7fd5b57fa700 1 -- 172.21.15.137:0/1637068906 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd5ac010230 con 0x7fd5b80f5410 2023-11-15T01:57:56.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:55 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:57:54.770258+0000 mgr.smithi055.szwngw (mgr.14164) 254 : cluster [DBG] pgmap v240: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:57:56.341 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:56.339+0000 7fd5bec9b700 1 -- 172.21.15.137:0/1637068906 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd5b80fae60 con 0x7fd5b80f5410 2023-11-15T01:57:56.342 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:56.340+0000 7fd5b57fa700 1 -- 172.21.15.137:0/1637068906 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd5ac010230 con 0x7fd5b80f5410 2023-11-15T01:57:56.342 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:57:56.342 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:57:56.344 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:56.342+0000 7fd5bec9b700 1 -- 172.21.15.137:0/1637068906 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fd5a802ff60 msgr2=0x7fd5a8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:57:56.345 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:56.342+0000 7fd5bec9b700 1 --2- 172.21.15.137:0/1637068906 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fd5a802ff60 0x7fd5a8032420 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7fd5a40096a0 tx=0x7fd5a4006b40).stop 2023-11-15T01:57:56.345 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:56.342+0000 7fd5bec9b700 1 -- 172.21.15.137:0/1637068906 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd5b80f5410 msgr2=0x7fd5b8109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:57:56.345 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:56.342+0000 7fd5bec9b700 1 --2- 172.21.15.137:0/1637068906 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd5b80f5410 0x7fd5b8109430 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7fd5ac013510 tx=0x7fd5ac00b040).stop 2023-11-15T01:57:56.345 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:56.342+0000 7fd5bec9b700 1 -- 172.21.15.137:0/1637068906 shutdown_connections 2023-11-15T01:57:56.345 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:56.342+0000 7fd5bec9b700 1 --2- 172.21.15.137:0/1637068906 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fd5a802ff60 0x7fd5a8032420 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:57:56.346 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:56.342+0000 7fd5bec9b700 1 --2- 172.21.15.137:0/1637068906 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd5b80f5410 0x7fd5b8109430 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:57:56.346 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:56.342+0000 7fd5bec9b700 1 -- 172.21.15.137:0/1637068906 >> 172.21.15.137:0/1637068906 conn(0x7fd5b80f0b70 msgr2=0x7fd5b80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:57:56.346 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:56.342+0000 7fd5bec9b700 1 -- 172.21.15.137:0/1637068906 shutdown_connections 2023-11-15T01:57:56.346 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:56.342+0000 7fd5bec9b700 1 -- 172.21.15.137:0/1637068906 wait complete. 2023-11-15T01:57:56.346 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:57:57.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:56 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:57:56.342276+0000 mon.smithi055 (mon.0) 309 : audit [DBG] from='client.? 172.21.15.137:0/1637068906' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:57:57.742 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:57:57.743 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:57:58.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:57 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:57:56.770473+0000 mgr.smithi055.szwngw (mgr.14164) 255 : cluster [DBG] pgmap v241: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:57:58.890 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:58.888+0000 7fbefb089700 1 -- 172.21.15.137:0/783972082 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbef40f6510 msgr2=0x7fbef40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:57:58.891 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:58.888+0000 7fbefb089700 1 --2- 172.21.15.137:0/783972082 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbef40f6510 0x7fbef40f6930 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7fbee8009fb0 tx=0x7fbee8009b30).stop 2023-11-15T01:57:58.891 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:58.888+0000 7fbefb089700 1 -- 172.21.15.137:0/783972082 shutdown_connections 2023-11-15T01:57:58.891 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:58.888+0000 7fbefb089700 1 --2- 172.21.15.137:0/783972082 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbef40f6510 0x7fbef40f6930 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:57:58.891 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:58.888+0000 7fbefb089700 1 -- 172.21.15.137:0/783972082 >> 172.21.15.137:0/783972082 conn(0x7fbef40f1c70 msgr2=0x7fbef40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:57:58.892 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:58.888+0000 7fbefb089700 1 -- 172.21.15.137:0/783972082 shutdown_connections 2023-11-15T01:57:58.892 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:58.888+0000 7fbefb089700 1 -- 172.21.15.137:0/783972082 wait complete. 2023-11-15T01:57:58.892 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:58.889+0000 7fbefb089700 1 Processor -- start 2023-11-15T01:57:58.892 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:58.889+0000 7fbefb089700 1 -- start start 2023-11-15T01:57:58.892 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:58.889+0000 7fbefb089700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbef40f6510 0x7fbef4106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:57:58.893 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:58.889+0000 7fbefb089700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbef4106c20 con 0x7fbef40f6510 2023-11-15T01:57:58.893 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:58.889+0000 7fbef8e25700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbef40f6510 0x7fbef4106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:57:58.893 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:58.889+0000 7fbef8e25700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbef40f6510 0x7fbef4106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:57714/0 (socket says 172.21.15.137:57714) 2023-11-15T01:57:58.893 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:58.889+0000 7fbef8e25700 1 -- 172.21.15.137:0/119116172 learned_addr learned my addr 172.21.15.137:0/119116172 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:57:58.895 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:58.889+0000 7fbef8e25700 1 -- 172.21.15.137:0/119116172 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbee8005040 con 0x7fbef40f6510 2023-11-15T01:57:58.895 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:58.890+0000 7fbef8e25700 1 --2- 172.21.15.137:0/119116172 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbef40f6510 0x7fbef4106730 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7fbee8014000 tx=0x7fbee800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:57:58.895 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:58.890+0000 7fbef17fa700 1 -- 172.21.15.137:0/119116172 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbee80095f0 con 0x7fbef40f6510 2023-11-15T01:57:58.895 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:58.890+0000 7fbef17fa700 1 -- 172.21.15.137:0/119116172 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbee8007e70 con 0x7fbef40f6510 2023-11-15T01:57:58.896 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:58.890+0000 7fbefb089700 1 -- 172.21.15.137:0/119116172 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbef4106e20 con 0x7fbef40f6510 2023-11-15T01:57:58.896 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:58.890+0000 7fbef17fa700 1 -- 172.21.15.137:0/119116172 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbee8014ea0 con 0x7fbef40f6510 2023-11-15T01:57:58.896 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:58.890+0000 7fbefb089700 1 -- 172.21.15.137:0/119116172 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbef41072b0 con 0x7fbef40f6510 2023-11-15T01:57:58.896 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:58.891+0000 7fbef17fa700 1 -- 172.21.15.137:0/119116172 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbee8025070 con 0x7fbef40f6510 2023-11-15T01:57:58.896 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:58.891+0000 7fbefb089700 1 -- 172.21.15.137:0/119116172 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbef4045ad0 con 0x7fbef40f6510 2023-11-15T01:57:58.897 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:58.891+0000 7fbef17fa700 1 --2- 172.21.15.137:0/119116172 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fbee402ff60 0x7fbee4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:57:58.897 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:58.891+0000 7fbef17fa700 1 -- 172.21.15.137:0/119116172 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbee8040a40 con 0x7fbef40f6510 2023-11-15T01:57:58.897 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:58.891+0000 7fbef3fff700 1 --2- 172.21.15.137:0/119116172 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fbee402ff60 0x7fbee4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:57:58.897 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:58.891+0000 7fbef3fff700 1 --2- 172.21.15.137:0/119116172 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fbee402ff60 0x7fbee4032420 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7fbee00096a0 tx=0x7fbee0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:57:58.898 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:58.896+0000 7fbef17fa700 1 -- 172.21.15.137:0/119116172 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbee80079c0 con 0x7fbef40f6510 2023-11-15T01:57:59.101 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:59.099+0000 7fbefb089700 1 -- 172.21.15.137:0/119116172 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbef40fbf60 con 0x7fbef40f6510 2023-11-15T01:57:59.101 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:59.101+0000 7fbef17fa700 1 -- 172.21.15.137:0/119116172 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbee800f300 con 0x7fbef40f6510 2023-11-15T01:57:59.102 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:57:59.102 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:57:59.103 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:59.101+0000 7fbefb089700 1 -- 172.21.15.137:0/119116172 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fbee402ff60 msgr2=0x7fbee4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:57:59.103 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:59.101+0000 7fbefb089700 1 --2- 172.21.15.137:0/119116172 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fbee402ff60 0x7fbee4032420 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7fbee00096a0 tx=0x7fbee0006b40).stop 2023-11-15T01:57:59.103 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:59.101+0000 7fbefb089700 1 -- 172.21.15.137:0/119116172 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbef40f6510 msgr2=0x7fbef4106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:57:59.104 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:59.101+0000 7fbefb089700 1 --2- 172.21.15.137:0/119116172 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbef40f6510 0x7fbef4106730 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7fbee8014000 tx=0x7fbee800b040).stop 2023-11-15T01:57:59.104 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:59.101+0000 7fbefb089700 1 -- 172.21.15.137:0/119116172 shutdown_connections 2023-11-15T01:57:59.104 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:59.101+0000 7fbefb089700 1 --2- 172.21.15.137:0/119116172 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fbee402ff60 0x7fbee4032420 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:57:59.104 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:59.101+0000 7fbefb089700 1 --2- 172.21.15.137:0/119116172 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbef40f6510 0x7fbef4106730 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:57:59.104 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:59.101+0000 7fbefb089700 1 -- 172.21.15.137:0/119116172 >> 172.21.15.137:0/119116172 conn(0x7fbef40f1c70 msgr2=0x7fbef40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:57:59.105 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:59.102+0000 7fbefb089700 1 -- 172.21.15.137:0/119116172 shutdown_connections 2023-11-15T01:57:59.105 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:57:59.102+0000 7fbefb089700 1 -- 172.21.15.137:0/119116172 wait complete. 2023-11-15T01:57:59.105 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:58:00.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:59 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:57:58.770831+0000 mgr.smithi055.szwngw (mgr.14164) 256 : cluster [DBG] pgmap v242: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:58:00.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:57:59 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:57:59.102024+0000 mon.smithi055 (mon.0) 310 : audit [DBG] from='client.? 172.21.15.137:0/119116172' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:58:00.493 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:58:00.494 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:58:01.665 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:01.662+0000 7f1e09b4d700 1 -- 172.21.15.137:0/2699190488 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1e040f5410 msgr2=0x7f1e040f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:58:01.665 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:01.662+0000 7f1e09b4d700 1 --2- 172.21.15.137:0/2699190488 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1e040f5410 0x7f1e040f5830 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7f1df4009fb0 tx=0x7f1df4009b30).stop 2023-11-15T01:58:01.666 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:01.663+0000 7f1e09b4d700 1 -- 172.21.15.137:0/2699190488 shutdown_connections 2023-11-15T01:58:01.666 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:01.663+0000 7f1e09b4d700 1 --2- 172.21.15.137:0/2699190488 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1e040f5410 0x7f1e040f5830 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:58:01.666 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:01.663+0000 7f1e09b4d700 1 -- 172.21.15.137:0/2699190488 >> 172.21.15.137:0/2699190488 conn(0x7f1e040f0b70 msgr2=0x7f1e040f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:58:01.666 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:01.663+0000 7f1e09b4d700 1 -- 172.21.15.137:0/2699190488 shutdown_connections 2023-11-15T01:58:01.666 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:01.663+0000 7f1e09b4d700 1 -- 172.21.15.137:0/2699190488 wait complete. 2023-11-15T01:58:01.667 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:01.663+0000 7f1e09b4d700 1 Processor -- start 2023-11-15T01:58:01.667 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:01.663+0000 7f1e09b4d700 1 -- start start 2023-11-15T01:58:01.667 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:01.663+0000 7f1e09b4d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1e040f5410 0x7f1e04109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:58:01.667 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:01.663+0000 7f1e09b4d700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e04060220 con 0x7f1e040f5410 2023-11-15T01:58:01.667 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:01.664+0000 7f1e037fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1e040f5410 0x7f1e04109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:58:01.668 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:01.664+0000 7f1e037fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1e040f5410 0x7f1e04109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:57738/0 (socket says 172.21.15.137:57738) 2023-11-15T01:58:01.668 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:01.664+0000 7f1e037fe700 1 -- 172.21.15.137:0/3902787477 learned_addr learned my addr 172.21.15.137:0/3902787477 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:58:01.669 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:01.664+0000 7f1e037fe700 1 -- 172.21.15.137:0/3902787477 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1df4005040 con 0x7f1e040f5410 2023-11-15T01:58:01.669 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:01.664+0000 7f1e037fe700 1 --2- 172.21.15.137:0/3902787477 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1e040f5410 0x7f1e04109450 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7f1df4013510 tx=0x7f1df400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:58:01.670 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:01.664+0000 7f1debfff700 1 -- 172.21.15.137:0/3902787477 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1df4009320 con 0x7f1e040f5410 2023-11-15T01:58:01.670 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:01.664+0000 7f1e09b4d700 1 -- 172.21.15.137:0/3902787477 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e04060420 con 0x7f1e040f5410 2023-11-15T01:58:01.670 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:01.664+0000 7f1debfff700 1 -- 172.21.15.137:0/3902787477 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1df4013f00 con 0x7f1e040f5410 2023-11-15T01:58:01.670 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:01.664+0000 7f1debfff700 1 -- 172.21.15.137:0/3902787477 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1df4014d90 con 0x7f1e040f5410 2023-11-15T01:58:01.671 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:01.664+0000 7f1e09b4d700 1 -- 172.21.15.137:0/3902787477 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e04109b00 con 0x7f1e040f5410 2023-11-15T01:58:01.671 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:01.665+0000 7f1debfff700 1 -- 172.21.15.137:0/3902787477 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1df40095f0 con 0x7f1e040f5410 2023-11-15T01:58:01.671 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:01.665+0000 7f1e09b4d700 1 -- 172.21.15.137:0/3902787477 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e04059100 con 0x7f1e040f5410 2023-11-15T01:58:01.671 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:01.665+0000 7f1debfff700 1 --2- 172.21.15.137:0/3902787477 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f1dec02ff60 0x7f1dec032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:58:01.671 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:01.665+0000 7f1debfff700 1 -- 172.21.15.137:0/3902787477 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1df4040fe0 con 0x7f1e040f5410 2023-11-15T01:58:01.671 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:01.666+0000 7f1e02ffd700 1 --2- 172.21.15.137:0/3902787477 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f1dec02ff60 0x7f1dec032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:58:01.672 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:01.666+0000 7f1e02ffd700 1 --2- 172.21.15.137:0/3902787477 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f1dec02ff60 0x7f1dec032420 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f1df80096a0 tx=0x7f1df8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:58:01.672 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:01.670+0000 7f1debfff700 1 -- 172.21.15.137:0/3902787477 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1df401d7f0 con 0x7f1e040f5410 2023-11-15T01:58:01.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:58:01 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:58:00.771144+0000 mgr.smithi055.szwngw (mgr.14164) 257 : cluster [DBG] pgmap v243: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:58:01.879 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:01.878+0000 7f1e09b4d700 1 -- 172.21.15.137:0/3902787477 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1e04045ad0 con 0x7f1e040f5410 2023-11-15T01:58:01.880 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:01.879+0000 7f1debfff700 1 -- 172.21.15.137:0/3902787477 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1df4010230 con 0x7f1e040f5410 2023-11-15T01:58:01.880 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:58:01.881 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:58:01.882 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:01.880+0000 7f1e09b4d700 1 -- 172.21.15.137:0/3902787477 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f1dec02ff60 msgr2=0x7f1dec032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:58:01.882 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:01.880+0000 7f1e09b4d700 1 --2- 172.21.15.137:0/3902787477 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f1dec02ff60 0x7f1dec032420 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f1df80096a0 tx=0x7f1df8006b40).stop 2023-11-15T01:58:01.882 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:01.880+0000 7f1e09b4d700 1 -- 172.21.15.137:0/3902787477 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1e040f5410 msgr2=0x7f1e04109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:58:01.883 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:01.880+0000 7f1e09b4d700 1 --2- 172.21.15.137:0/3902787477 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1e040f5410 0x7f1e04109450 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7f1df4013510 tx=0x7f1df400b040).stop 2023-11-15T01:58:01.883 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:01.880+0000 7f1e09b4d700 1 -- 172.21.15.137:0/3902787477 shutdown_connections 2023-11-15T01:58:01.883 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:01.880+0000 7f1e09b4d700 1 --2- 172.21.15.137:0/3902787477 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f1dec02ff60 0x7f1dec032420 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:58:01.883 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:01.880+0000 7f1e09b4d700 1 --2- 172.21.15.137:0/3902787477 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1e040f5410 0x7f1e04109450 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:58:01.883 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:01.880+0000 7f1e09b4d700 1 -- 172.21.15.137:0/3902787477 >> 172.21.15.137:0/3902787477 conn(0x7f1e040f0b70 msgr2=0x7f1e040f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:58:01.884 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:01.880+0000 7f1e09b4d700 1 -- 172.21.15.137:0/3902787477 shutdown_connections 2023-11-15T01:58:01.884 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:01.880+0000 7f1e09b4d700 1 -- 172.21.15.137:0/3902787477 wait complete. 2023-11-15T01:58:01.884 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:58:02.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:58:02 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:58:01.880638+0000 mon.smithi055 (mon.0) 311 : audit [DBG] from='client.? 172.21.15.137:0/3902787477' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:58:03.261 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:58:03.261 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:58:03.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:58:03 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:58:02.771560+0000 mgr.smithi055.szwngw (mgr.14164) 258 : cluster [DBG] pgmap v244: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:58:04.365 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:04.364+0000 7fbe6f9c6700 1 -- 172.21.15.137:0/1551637392 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbe680f6510 msgr2=0x7fbe680f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:58:04.366 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:04.364+0000 7fbe6f9c6700 1 --2- 172.21.15.137:0/1551637392 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbe680f6510 0x7fbe680f6930 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7fbe64009fb0 tx=0x7fbe64009b30).stop 2023-11-15T01:58:04.366 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:04.364+0000 7fbe6f9c6700 1 -- 172.21.15.137:0/1551637392 shutdown_connections 2023-11-15T01:58:04.366 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:04.364+0000 7fbe6f9c6700 1 --2- 172.21.15.137:0/1551637392 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbe680f6510 0x7fbe680f6930 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:58:04.366 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:04.364+0000 7fbe6f9c6700 1 -- 172.21.15.137:0/1551637392 >> 172.21.15.137:0/1551637392 conn(0x7fbe680f1c70 msgr2=0x7fbe680f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:58:04.367 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:04.364+0000 7fbe6f9c6700 1 -- 172.21.15.137:0/1551637392 shutdown_connections 2023-11-15T01:58:04.367 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:04.364+0000 7fbe6f9c6700 1 -- 172.21.15.137:0/1551637392 wait complete. 2023-11-15T01:58:04.367 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:04.364+0000 7fbe6f9c6700 1 Processor -- start 2023-11-15T01:58:04.367 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:04.364+0000 7fbe6f9c6700 1 -- start start 2023-11-15T01:58:04.367 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:04.364+0000 7fbe6f9c6700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbe680f6510 0x7fbe68106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:58:04.368 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:04.364+0000 7fbe6f9c6700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe68106c20 con 0x7fbe680f6510 2023-11-15T01:58:04.368 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:04.365+0000 7fbe6d762700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbe680f6510 0x7fbe68106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:58:04.368 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:04.365+0000 7fbe6d762700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbe680f6510 0x7fbe68106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:57748/0 (socket says 172.21.15.137:57748) 2023-11-15T01:58:04.368 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:04.365+0000 7fbe6d762700 1 -- 172.21.15.137:0/2020403013 learned_addr learned my addr 172.21.15.137:0/2020403013 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:58:04.368 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:04.365+0000 7fbe6d762700 1 -- 172.21.15.137:0/2020403013 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe64005040 con 0x7fbe680f6510 2023-11-15T01:58:04.369 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:04.365+0000 7fbe6d762700 1 --2- 172.21.15.137:0/2020403013 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbe680f6510 0x7fbe68106730 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7fbe64013510 tx=0x7fbe6400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:58:04.370 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:04.365+0000 7fbe5dffb700 1 -- 172.21.15.137:0/2020403013 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe64009320 con 0x7fbe680f6510 2023-11-15T01:58:04.370 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:04.365+0000 7fbe6f9c6700 1 -- 172.21.15.137:0/2020403013 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe68106e20 con 0x7fbe680f6510 2023-11-15T01:58:04.370 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:04.365+0000 7fbe5dffb700 1 -- 172.21.15.137:0/2020403013 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbe64013f00 con 0x7fbe680f6510 2023-11-15T01:58:04.371 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:04.365+0000 7fbe5dffb700 1 -- 172.21.15.137:0/2020403013 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe64014d90 con 0x7fbe680f6510 2023-11-15T01:58:04.371 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:04.365+0000 7fbe6f9c6700 1 -- 172.21.15.137:0/2020403013 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe681072b0 con 0x7fbe680f6510 2023-11-15T01:58:04.371 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:04.366+0000 7fbe5dffb700 1 -- 172.21.15.137:0/2020403013 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbe64014420 con 0x7fbe680f6510 2023-11-15T01:58:04.371 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:04.366+0000 7fbe6f9c6700 1 -- 172.21.15.137:0/2020403013 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe68045ad0 con 0x7fbe680f6510 2023-11-15T01:58:04.371 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:04.367+0000 7fbe5dffb700 1 --2- 172.21.15.137:0/2020403013 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fbe5402ff60 0x7fbe54032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:58:04.372 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:04.367+0000 7fbe5dffb700 1 -- 172.21.15.137:0/2020403013 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbe64041000 con 0x7fbe680f6510 2023-11-15T01:58:04.372 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:04.367+0000 7fbe6cf61700 1 --2- 172.21.15.137:0/2020403013 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fbe5402ff60 0x7fbe54032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:58:04.372 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:04.367+0000 7fbe6cf61700 1 --2- 172.21.15.137:0/2020403013 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fbe5402ff60 0x7fbe54032420 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7fbe580096a0 tx=0x7fbe58006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:58:04.372 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:04.371+0000 7fbe5dffb700 1 -- 172.21.15.137:0/2020403013 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbe640095f0 con 0x7fbe680f6510 2023-11-15T01:58:04.578 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:04.578+0000 7fbe6f9c6700 1 -- 172.21.15.137:0/2020403013 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbe680fbf60 con 0x7fbe680f6510 2023-11-15T01:58:04.579 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:04.578+0000 7fbe5dffb700 1 -- 172.21.15.137:0/2020403013 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbe6401db70 con 0x7fbe680f6510 2023-11-15T01:58:04.579 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:58:04.580 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:58:04.580 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:04.580+0000 7fbe6f9c6700 1 -- 172.21.15.137:0/2020403013 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fbe5402ff60 msgr2=0x7fbe54032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:58:04.581 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:04.580+0000 7fbe6f9c6700 1 --2- 172.21.15.137:0/2020403013 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fbe5402ff60 0x7fbe54032420 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7fbe580096a0 tx=0x7fbe58006b40).stop 2023-11-15T01:58:04.581 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:04.580+0000 7fbe6f9c6700 1 -- 172.21.15.137:0/2020403013 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbe680f6510 msgr2=0x7fbe68106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:58:04.581 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:04.580+0000 7fbe6f9c6700 1 --2- 172.21.15.137:0/2020403013 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbe680f6510 0x7fbe68106730 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7fbe64013510 tx=0x7fbe6400b040).stop 2023-11-15T01:58:04.581 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:04.580+0000 7fbe6f9c6700 1 -- 172.21.15.137:0/2020403013 shutdown_connections 2023-11-15T01:58:04.582 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:04.580+0000 7fbe6f9c6700 1 --2- 172.21.15.137:0/2020403013 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fbe5402ff60 0x7fbe54032420 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:58:04.582 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:04.580+0000 7fbe6f9c6700 1 --2- 172.21.15.137:0/2020403013 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbe680f6510 0x7fbe68106730 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:58:04.582 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:04.580+0000 7fbe6f9c6700 1 -- 172.21.15.137:0/2020403013 >> 172.21.15.137:0/2020403013 conn(0x7fbe680f1c70 msgr2=0x7fbe680f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:58:04.582 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:04.580+0000 7fbe6f9c6700 1 -- 172.21.15.137:0/2020403013 shutdown_connections 2023-11-15T01:58:04.582 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:04.580+0000 7fbe6f9c6700 1 -- 172.21.15.137:0/2020403013 wait complete. 2023-11-15T01:58:04.583 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:58:05.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:58:04 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:58:04.579483+0000 mon.smithi055 (mon.0) 312 : audit [DBG] from='client.? 172.21.15.137:0/2020403013' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:58:05.956 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:58:05.956 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:58:06.078 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:58:05 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:58:04.771995+0000 mgr.smithi055.szwngw (mgr.14164) 259 : cluster [DBG] pgmap v245: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:58:07.116 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:07.115+0000 7f19704a5700 1 -- 172.21.15.137:0/1993196391 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f19680f5410 msgr2=0x7f19680f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:58:07.116 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:07.115+0000 7f19704a5700 1 --2- 172.21.15.137:0/1993196391 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f19680f5410 0x7f19680f5830 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7f1958009fb0 tx=0x7f1958009b30).stop 2023-11-15T01:58:07.116 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:07.115+0000 7f19704a5700 1 -- 172.21.15.137:0/1993196391 shutdown_connections 2023-11-15T01:58:07.117 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:07.115+0000 7f19704a5700 1 --2- 172.21.15.137:0/1993196391 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f19680f5410 0x7f19680f5830 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:58:07.117 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:07.115+0000 7f19704a5700 1 -- 172.21.15.137:0/1993196391 >> 172.21.15.137:0/1993196391 conn(0x7f19680f0b70 msgr2=0x7f19680f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:58:07.117 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:07.115+0000 7f19704a5700 1 -- 172.21.15.137:0/1993196391 shutdown_connections 2023-11-15T01:58:07.117 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:07.115+0000 7f19704a5700 1 -- 172.21.15.137:0/1993196391 wait complete. 2023-11-15T01:58:07.119 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:07.116+0000 7f19704a5700 1 Processor -- start 2023-11-15T01:58:07.120 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:07.116+0000 7f19704a5700 1 -- start start 2023-11-15T01:58:07.120 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:07.116+0000 7f19704a5700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f19680f5410 0x7f1968108910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:58:07.120 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:07.116+0000 7f19704a5700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1968108e00 con 0x7f19680f5410 2023-11-15T01:58:07.120 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:07.116+0000 7f196e241700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f19680f5410 0x7f1968108910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:58:07.121 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:07.116+0000 7f196e241700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f19680f5410 0x7f1968108910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:57776/0 (socket says 172.21.15.137:57776) 2023-11-15T01:58:07.121 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:07.116+0000 7f196e241700 1 -- 172.21.15.137:0/1997793673 learned_addr learned my addr 172.21.15.137:0/1997793673 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:58:07.121 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:07.116+0000 7f196e241700 1 -- 172.21.15.137:0/1997793673 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1958005040 con 0x7f19680f5410 2023-11-15T01:58:07.121 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:07.116+0000 7f196e241700 1 --2- 172.21.15.137:0/1997793673 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f19680f5410 0x7f1968108910 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7f1958013510 tx=0x7f195800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:58:07.121 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:07.117+0000 7f195effd700 1 -- 172.21.15.137:0/1997793673 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1958009320 con 0x7f19680f5410 2023-11-15T01:58:07.122 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:07.117+0000 7f195effd700 1 -- 172.21.15.137:0/1997793673 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1958013f00 con 0x7f19680f5410 2023-11-15T01:58:07.122 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:07.117+0000 7f19704a5700 1 -- 172.21.15.137:0/1997793673 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1968109000 con 0x7f19680f5410 2023-11-15T01:58:07.122 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:07.117+0000 7f195effd700 1 -- 172.21.15.137:0/1997793673 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1958014d90 con 0x7f19680f5410 2023-11-15T01:58:07.122 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:07.117+0000 7f19704a5700 1 -- 172.21.15.137:0/1997793673 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1968109490 con 0x7f19680f5410 2023-11-15T01:58:07.123 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:07.117+0000 7f195effd700 1 -- 172.21.15.137:0/1997793673 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f19580095f0 con 0x7f19680f5410 2023-11-15T01:58:07.123 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:07.118+0000 7f19704a5700 1 -- 172.21.15.137:0/1997793673 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1968045ad0 con 0x7f19680f5410 2023-11-15T01:58:07.123 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:07.118+0000 7f195effd700 1 --2- 172.21.15.137:0/1997793673 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f195402ff60 0x7f1954032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:58:07.123 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:07.118+0000 7f195effd700 1 -- 172.21.15.137:0/1997793673 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f19580410a0 con 0x7f19680f5410 2023-11-15T01:58:07.123 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:07.118+0000 7f196da40700 1 --2- 172.21.15.137:0/1997793673 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f195402ff60 0x7f1954032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:58:07.124 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:07.118+0000 7f196da40700 1 --2- 172.21.15.137:0/1997793673 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f195402ff60 0x7f1954032420 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f19640096a0 tx=0x7f1964006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:58:07.124 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:07.123+0000 7f195effd700 1 -- 172.21.15.137:0/1997793673 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f195801d7f0 con 0x7f19680f5410 2023-11-15T01:58:07.331 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:07.331+0000 7f19704a5700 1 -- 172.21.15.137:0/1997793673 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f19680fae60 con 0x7f19680f5410 2023-11-15T01:58:07.333 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:07.331+0000 7f195effd700 1 -- 172.21.15.137:0/1997793673 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1958010230 con 0x7f19680f5410 2023-11-15T01:58:07.334 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:58:07.334 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:58:07.335 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:07.333+0000 7f19704a5700 1 -- 172.21.15.137:0/1997793673 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f195402ff60 msgr2=0x7f1954032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:58:07.335 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:07.333+0000 7f19704a5700 1 --2- 172.21.15.137:0/1997793673 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f195402ff60 0x7f1954032420 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f19640096a0 tx=0x7f1964006b40).stop 2023-11-15T01:58:07.335 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:07.333+0000 7f19704a5700 1 -- 172.21.15.137:0/1997793673 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f19680f5410 msgr2=0x7f1968108910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:58:07.336 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:07.333+0000 7f19704a5700 1 --2- 172.21.15.137:0/1997793673 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f19680f5410 0x7f1968108910 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7f1958013510 tx=0x7f195800b040).stop 2023-11-15T01:58:07.336 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:07.333+0000 7f19704a5700 1 -- 172.21.15.137:0/1997793673 shutdown_connections 2023-11-15T01:58:07.336 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:07.333+0000 7f19704a5700 1 --2- 172.21.15.137:0/1997793673 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f195402ff60 0x7f1954032420 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:58:07.336 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:07.333+0000 7f19704a5700 1 --2- 172.21.15.137:0/1997793673 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f19680f5410 0x7f1968108910 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:58:07.337 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:07.333+0000 7f19704a5700 1 -- 172.21.15.137:0/1997793673 >> 172.21.15.137:0/1997793673 conn(0x7f19680f0b70 msgr2=0x7f196805fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:58:07.337 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:07.333+0000 7f19704a5700 1 -- 172.21.15.137:0/1997793673 shutdown_connections 2023-11-15T01:58:07.337 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:07.333+0000 7f19704a5700 1 -- 172.21.15.137:0/1997793673 wait complete. 2023-11-15T01:58:07.337 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:58:08.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:58:07 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:58:06.772363+0000 mgr.smithi055.szwngw (mgr.14164) 260 : cluster [DBG] pgmap v246: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:58:08.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:58:07 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:58:07.332540+0000 mon.smithi055 (mon.0) 313 : audit [DBG] from='client.? 172.21.15.137:0/1997793673' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:58:08.707 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:58:08.708 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:58:09.811 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:09.808+0000 7fa6efd8b700 1 -- 172.21.15.137:0/376198359 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa6e80f6510 msgr2=0x7fa6e80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:58:09.811 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:09.808+0000 7fa6efd8b700 1 --2- 172.21.15.137:0/376198359 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa6e80f6510 0x7fa6e80f6930 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7fa6d8009fb0 tx=0x7fa6d8009b30).stop 2023-11-15T01:58:09.811 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:09.808+0000 7fa6efd8b700 1 -- 172.21.15.137:0/376198359 shutdown_connections 2023-11-15T01:58:09.811 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:09.808+0000 7fa6efd8b700 1 --2- 172.21.15.137:0/376198359 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa6e80f6510 0x7fa6e80f6930 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:58:09.812 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:09.808+0000 7fa6efd8b700 1 -- 172.21.15.137:0/376198359 >> 172.21.15.137:0/376198359 conn(0x7fa6e80f1c70 msgr2=0x7fa6e80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:58:09.812 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:09.808+0000 7fa6efd8b700 1 -- 172.21.15.137:0/376198359 shutdown_connections 2023-11-15T01:58:09.812 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:09.808+0000 7fa6efd8b700 1 -- 172.21.15.137:0/376198359 wait complete. 2023-11-15T01:58:09.812 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:09.809+0000 7fa6efd8b700 1 Processor -- start 2023-11-15T01:58:09.812 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:09.809+0000 7fa6efd8b700 1 -- start start 2023-11-15T01:58:09.813 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:09.809+0000 7fa6efd8b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa6e80f6510 0x7fa6e8106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:58:09.813 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:09.809+0000 7fa6efd8b700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e8106c20 con 0x7fa6e80f6510 2023-11-15T01:58:09.813 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:09.809+0000 7fa6edb27700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa6e80f6510 0x7fa6e8106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:58:09.813 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:09.809+0000 7fa6edb27700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa6e80f6510 0x7fa6e8106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:33344/0 (socket says 172.21.15.137:33344) 2023-11-15T01:58:09.813 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:09.809+0000 7fa6edb27700 1 -- 172.21.15.137:0/1562951155 learned_addr learned my addr 172.21.15.137:0/1562951155 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:58:09.814 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:09.810+0000 7fa6edb27700 1 -- 172.21.15.137:0/1562951155 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6d8005040 con 0x7fa6e80f6510 2023-11-15T01:58:09.814 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:09.810+0000 7fa6edb27700 1 --2- 172.21.15.137:0/1562951155 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa6e80f6510 0x7fa6e8106730 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7fa6d8013510 tx=0x7fa6d800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:58:09.814 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:09.810+0000 7fa6de7fc700 1 -- 172.21.15.137:0/1562951155 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa6d8009320 con 0x7fa6e80f6510 2023-11-15T01:58:09.814 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:09.810+0000 7fa6de7fc700 1 -- 172.21.15.137:0/1562951155 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa6d8013f00 con 0x7fa6e80f6510 2023-11-15T01:58:09.814 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:09.810+0000 7fa6efd8b700 1 -- 172.21.15.137:0/1562951155 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6e8106e20 con 0x7fa6e80f6510 2023-11-15T01:58:09.814 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:09.810+0000 7fa6de7fc700 1 -- 172.21.15.137:0/1562951155 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa6d8014d90 con 0x7fa6e80f6510 2023-11-15T01:58:09.815 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:09.810+0000 7fa6efd8b700 1 -- 172.21.15.137:0/1562951155 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6e81072b0 con 0x7fa6e80f6510 2023-11-15T01:58:09.816 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:09.811+0000 7fa6efd8b700 1 -- 172.21.15.137:0/1562951155 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6e8045ad0 con 0x7fa6e80f6510 2023-11-15T01:58:09.816 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:09.811+0000 7fa6de7fc700 1 -- 172.21.15.137:0/1562951155 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa6d80095f0 con 0x7fa6e80f6510 2023-11-15T01:58:09.816 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:09.811+0000 7fa6de7fc700 1 --2- 172.21.15.137:0/1562951155 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fa6d402ff60 0x7fa6d4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:58:09.816 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:09.811+0000 7fa6de7fc700 1 -- 172.21.15.137:0/1562951155 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa6d8041000 con 0x7fa6e80f6510 2023-11-15T01:58:09.816 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:09.811+0000 7fa6ed326700 1 --2- 172.21.15.137:0/1562951155 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fa6d402ff60 0x7fa6d4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:58:09.817 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:09.812+0000 7fa6ed326700 1 --2- 172.21.15.137:0/1562951155 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fa6d402ff60 0x7fa6d4032420 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7fa6e40096a0 tx=0x7fa6e4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:58:09.817 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:09.816+0000 7fa6de7fc700 1 -- 172.21.15.137:0/1562951155 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa6d8014420 con 0x7fa6e80f6510 2023-11-15T01:58:10.025 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:10.024+0000 7fa6efd8b700 1 -- 172.21.15.137:0/1562951155 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa6e80fbf60 con 0x7fa6e80f6510 2023-11-15T01:58:10.026 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:10.025+0000 7fa6de7fc700 1 -- 172.21.15.137:0/1562951155 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa6d8010230 con 0x7fa6e80f6510 2023-11-15T01:58:10.026 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:58:10.026 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:58:10.027 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:10.027+0000 7fa6efd8b700 1 -- 172.21.15.137:0/1562951155 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fa6d402ff60 msgr2=0x7fa6d4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:58:10.028 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:10.027+0000 7fa6efd8b700 1 --2- 172.21.15.137:0/1562951155 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fa6d402ff60 0x7fa6d4032420 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7fa6e40096a0 tx=0x7fa6e4006b40).stop 2023-11-15T01:58:10.028 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:10.027+0000 7fa6efd8b700 1 -- 172.21.15.137:0/1562951155 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa6e80f6510 msgr2=0x7fa6e8106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:58:10.028 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:10.027+0000 7fa6efd8b700 1 --2- 172.21.15.137:0/1562951155 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa6e80f6510 0x7fa6e8106730 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7fa6d8013510 tx=0x7fa6d800b040).stop 2023-11-15T01:58:10.028 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:10.027+0000 7fa6efd8b700 1 -- 172.21.15.137:0/1562951155 shutdown_connections 2023-11-15T01:58:10.028 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:10.027+0000 7fa6efd8b700 1 --2- 172.21.15.137:0/1562951155 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fa6d402ff60 0x7fa6d4032420 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:58:10.029 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:10.027+0000 7fa6efd8b700 1 --2- 172.21.15.137:0/1562951155 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa6e80f6510 0x7fa6e8106730 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:58:10.029 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:10.027+0000 7fa6efd8b700 1 -- 172.21.15.137:0/1562951155 >> 172.21.15.137:0/1562951155 conn(0x7fa6e80f1c70 msgr2=0x7fa6e80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:58:10.029 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:10.027+0000 7fa6efd8b700 1 -- 172.21.15.137:0/1562951155 shutdown_connections 2023-11-15T01:58:10.029 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:10.027+0000 7fa6efd8b700 1 -- 172.21.15.137:0/1562951155 wait complete. 2023-11-15T01:58:10.030 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:58:10.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:58:09 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:58:08.772848+0000 mgr.smithi055.szwngw (mgr.14164) 261 : cluster [DBG] pgmap v247: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:58:11.327 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:58:10 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:58:10.026277+0000 mon.smithi055 (mon.0) 314 : audit [DBG] from='client.? 172.21.15.137:0/1562951155' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:58:11.421 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:58:11.422 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:58:12.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:58:11 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:58:10.773160+0000 mgr.smithi055.szwngw (mgr.14164) 262 : cluster [DBG] pgmap v248: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:58:12.604 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:12.603+0000 7fae11ece700 1 -- 172.21.15.137:0/3137440718 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fae0c060200 msgr2=0x7fae0c0fb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:58:12.604 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:12.603+0000 7fae11ece700 1 --2- 172.21.15.137:0/3137440718 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fae0c060200 0x7fae0c0fb570 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7fadfc009960 tx=0x7fadfc004d10).stop 2023-11-15T01:58:12.605 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:12.603+0000 7fae11ece700 1 -- 172.21.15.137:0/3137440718 shutdown_connections 2023-11-15T01:58:12.605 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:12.603+0000 7fae11ece700 1 --2- 172.21.15.137:0/3137440718 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fae0c060200 0x7fae0c0fb570 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:58:12.605 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:12.603+0000 7fae11ece700 1 -- 172.21.15.137:0/3137440718 >> 172.21.15.137:0/3137440718 conn(0x7fae0c0f0b70 msgr2=0x7fae0c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:58:12.605 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:12.604+0000 7fae11ece700 1 -- 172.21.15.137:0/3137440718 shutdown_connections 2023-11-15T01:58:12.606 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:12.604+0000 7fae11ece700 1 -- 172.21.15.137:0/3137440718 wait complete. 2023-11-15T01:58:12.608 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:12.604+0000 7fae11ece700 1 Processor -- start 2023-11-15T01:58:12.608 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:12.604+0000 7fae11ece700 1 -- start start 2023-11-15T01:58:12.609 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:12.604+0000 7fae11ece700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fae0c060200 0x7fae0c1088d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:58:12.609 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:12.604+0000 7fae11ece700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae0c108dc0 con 0x7fae0c060200 2023-11-15T01:58:12.609 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:12.605+0000 7fae0b7fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fae0c060200 0x7fae0c1088d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:58:12.609 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:12.605+0000 7fae0b7fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fae0c060200 0x7fae0c1088d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:33372/0 (socket says 172.21.15.137:33372) 2023-11-15T01:58:12.610 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:12.605+0000 7fae0b7fe700 1 -- 172.21.15.137:0/2798104911 learned_addr learned my addr 172.21.15.137:0/2798104911 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:58:12.610 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:12.605+0000 7fae0b7fe700 1 -- 172.21.15.137:0/2798104911 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadfc009640 con 0x7fae0c060200 2023-11-15T01:58:12.610 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:12.605+0000 7fae0b7fe700 1 --2- 172.21.15.137:0/2798104911 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fae0c060200 0x7fae0c1088d0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7fadfc007bd0 tx=0x7fadfc004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:58:12.610 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:12.605+0000 7fae03fff700 1 -- 172.21.15.137:0/2798104911 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fadfc018070 con 0x7fae0c060200 2023-11-15T01:58:12.610 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:12.605+0000 7fae11ece700 1 -- 172.21.15.137:0/2798104911 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae0c108fc0 con 0x7fae0c060200 2023-11-15T01:58:12.611 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:12.605+0000 7fae03fff700 1 -- 172.21.15.137:0/2798104911 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fadfc00b870 con 0x7fae0c060200 2023-11-15T01:58:12.611 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:12.605+0000 7fae03fff700 1 -- 172.21.15.137:0/2798104911 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fadfc01d420 con 0x7fae0c060200 2023-11-15T01:58:12.611 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:12.605+0000 7fae11ece700 1 -- 172.21.15.137:0/2798104911 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae0c109450 con 0x7fae0c060200 2023-11-15T01:58:12.611 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:12.606+0000 7fae03fff700 1 -- 172.21.15.137:0/2798104911 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fadfc024060 con 0x7fae0c060200 2023-11-15T01:58:12.611 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:12.606+0000 7fae11ece700 1 -- 172.21.15.137:0/2798104911 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae0c045ad0 con 0x7fae0c060200 2023-11-15T01:58:12.611 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:12.606+0000 7fae03fff700 1 --2- 172.21.15.137:0/2798104911 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fadf802ff60 0x7fadf8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:58:12.612 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:12.606+0000 7fae03fff700 1 -- 172.21.15.137:0/2798104911 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fadfc041170 con 0x7fae0c060200 2023-11-15T01:58:12.612 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:12.606+0000 7fae0affd700 1 --2- 172.21.15.137:0/2798104911 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fadf802ff60 0x7fadf8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:58:12.612 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:12.607+0000 7fae0affd700 1 --2- 172.21.15.137:0/2798104911 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fadf802ff60 0x7fadf8032420 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7fadf40096a0 tx=0x7fadf4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:58:12.613 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:12.611+0000 7fae03fff700 1 -- 172.21.15.137:0/2798104911 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fadfc025520 con 0x7fae0c060200 2023-11-15T01:58:12.816 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:12.815+0000 7fae11ece700 1 -- 172.21.15.137:0/2798104911 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fae0c10a250 con 0x7fae0c060200 2023-11-15T01:58:12.817 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:12.816+0000 7fae03fff700 1 -- 172.21.15.137:0/2798104911 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fadfc025340 con 0x7fae0c060200 2023-11-15T01:58:12.817 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:58:12.818 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:58:12.819 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:12.818+0000 7fae11ece700 1 -- 172.21.15.137:0/2798104911 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fadf802ff60 msgr2=0x7fadf8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:58:12.819 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:12.818+0000 7fae11ece700 1 --2- 172.21.15.137:0/2798104911 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fadf802ff60 0x7fadf8032420 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7fadf40096a0 tx=0x7fadf4006b40).stop 2023-11-15T01:58:12.819 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:12.818+0000 7fae11ece700 1 -- 172.21.15.137:0/2798104911 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fae0c060200 msgr2=0x7fae0c1088d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:58:12.819 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:12.818+0000 7fae11ece700 1 --2- 172.21.15.137:0/2798104911 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fae0c060200 0x7fae0c1088d0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7fadfc007bd0 tx=0x7fadfc004d10).stop 2023-11-15T01:58:12.820 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:12.818+0000 7fae11ece700 1 -- 172.21.15.137:0/2798104911 shutdown_connections 2023-11-15T01:58:12.820 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:12.818+0000 7fae11ece700 1 --2- 172.21.15.137:0/2798104911 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fadf802ff60 0x7fadf8032420 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:58:12.820 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:12.818+0000 7fae11ece700 1 --2- 172.21.15.137:0/2798104911 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fae0c060200 0x7fae0c1088d0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:58:12.820 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:12.818+0000 7fae11ece700 1 -- 172.21.15.137:0/2798104911 >> 172.21.15.137:0/2798104911 conn(0x7fae0c0f0b70 msgr2=0x7fae0c0f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:58:12.820 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:12.818+0000 7fae11ece700 1 -- 172.21.15.137:0/2798104911 shutdown_connections 2023-11-15T01:58:12.821 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:12.818+0000 7fae11ece700 1 -- 172.21.15.137:0/2798104911 wait complete. 2023-11-15T01:58:12.821 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:58:13.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:58:12 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:58:12.817370+0000 mon.smithi055 (mon.0) 315 : audit [DBG] from='client.? 172.21.15.137:0/2798104911' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:58:14.185 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:58:14.186 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:58:14.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:58:13 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:58:12.773512+0000 mgr.smithi055.szwngw (mgr.14164) 263 : cluster [DBG] pgmap v249: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:58:15.339 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:15.338+0000 7fc1fe68b700 1 -- 172.21.15.137:0/3368461297 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc1f80f6510 msgr2=0x7fc1f80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:58:15.340 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:15.338+0000 7fc1fe68b700 1 --2- 172.21.15.137:0/3368461297 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc1f80f6510 0x7fc1f80f6930 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0x7fc1e8009fb0 tx=0x7fc1e8009b30).stop 2023-11-15T01:58:15.340 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:15.338+0000 7fc1fe68b700 1 -- 172.21.15.137:0/3368461297 shutdown_connections 2023-11-15T01:58:15.340 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:15.338+0000 7fc1fe68b700 1 --2- 172.21.15.137:0/3368461297 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc1f80f6510 0x7fc1f80f6930 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:58:15.340 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:15.338+0000 7fc1fe68b700 1 -- 172.21.15.137:0/3368461297 >> 172.21.15.137:0/3368461297 conn(0x7fc1f80f1c70 msgr2=0x7fc1f80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:58:15.340 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:15.338+0000 7fc1fe68b700 1 -- 172.21.15.137:0/3368461297 shutdown_connections 2023-11-15T01:58:15.341 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:15.338+0000 7fc1fe68b700 1 -- 172.21.15.137:0/3368461297 wait complete. 2023-11-15T01:58:15.341 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:15.339+0000 7fc1fe68b700 1 Processor -- start 2023-11-15T01:58:15.341 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:15.339+0000 7fc1fe68b700 1 -- start start 2023-11-15T01:58:15.341 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:15.339+0000 7fc1fe68b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc1f80f6510 0x7fc1f8106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:58:15.341 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:15.339+0000 7fc1fe68b700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1f8106c20 con 0x7fc1f80f6510 2023-11-15T01:58:15.343 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:15.339+0000 7fc1f7fff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc1f80f6510 0x7fc1f8106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:58:15.343 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:15.339+0000 7fc1f7fff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc1f80f6510 0x7fc1f8106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:33386/0 (socket says 172.21.15.137:33386) 2023-11-15T01:58:15.343 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:15.339+0000 7fc1f7fff700 1 -- 172.21.15.137:0/761000661 learned_addr learned my addr 172.21.15.137:0/761000661 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:58:15.344 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:15.339+0000 7fc1f7fff700 1 -- 172.21.15.137:0/761000661 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1e8005040 con 0x7fc1f80f6510 2023-11-15T01:58:15.344 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:15.340+0000 7fc1f7fff700 1 --2- 172.21.15.137:0/761000661 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc1f80f6510 0x7fc1f8106730 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7fc1e8013510 tx=0x7fc1e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:58:15.344 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:15.340+0000 7fc1f4ff9700 1 -- 172.21.15.137:0/761000661 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc1e8009320 con 0x7fc1f80f6510 2023-11-15T01:58:15.344 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:15.340+0000 7fc1fe68b700 1 -- 172.21.15.137:0/761000661 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1f8106e20 con 0x7fc1f80f6510 2023-11-15T01:58:15.344 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:15.340+0000 7fc1f4ff9700 1 -- 172.21.15.137:0/761000661 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc1e8013f00 con 0x7fc1f80f6510 2023-11-15T01:58:15.345 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:15.340+0000 7fc1f4ff9700 1 -- 172.21.15.137:0/761000661 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc1e8014d90 con 0x7fc1f80f6510 2023-11-15T01:58:15.345 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:15.340+0000 7fc1fe68b700 1 -- 172.21.15.137:0/761000661 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1f81072b0 con 0x7fc1f80f6510 2023-11-15T01:58:15.345 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:15.341+0000 7fc1f4ff9700 1 -- 172.21.15.137:0/761000661 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc1e8014420 con 0x7fc1f80f6510 2023-11-15T01:58:15.345 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:15.341+0000 7fc1fe68b700 1 -- 172.21.15.137:0/761000661 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1f8045ad0 con 0x7fc1f80f6510 2023-11-15T01:58:15.345 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:15.341+0000 7fc1f4ff9700 1 --2- 172.21.15.137:0/761000661 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fc1e002ff60 0x7fc1e0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:58:15.346 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:15.341+0000 7fc1f4ff9700 1 -- 172.21.15.137:0/761000661 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc1e8041050 con 0x7fc1f80f6510 2023-11-15T01:58:15.346 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:15.341+0000 7fc1f77fe700 1 --2- 172.21.15.137:0/761000661 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fc1e002ff60 0x7fc1e0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:58:15.346 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:15.341+0000 7fc1f77fe700 1 --2- 172.21.15.137:0/761000661 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fc1e002ff60 0x7fc1e0032420 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7fc1ec0096a0 tx=0x7fc1ec006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:58:15.346 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:15.345+0000 7fc1f4ff9700 1 -- 172.21.15.137:0/761000661 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc1e80095f0 con 0x7fc1f80f6510 2023-11-15T01:58:15.550 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:15.549+0000 7fc1fe68b700 1 -- 172.21.15.137:0/761000661 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc1f80fbf60 con 0x7fc1f80f6510 2023-11-15T01:58:15.550 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:15.550+0000 7fc1f4ff9700 1 -- 172.21.15.137:0/761000661 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc1e801db70 con 0x7fc1f80f6510 2023-11-15T01:58:15.551 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:58:15.551 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:58:15.552 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:15.551+0000 7fc1fe68b700 1 -- 172.21.15.137:0/761000661 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fc1e002ff60 msgr2=0x7fc1e0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:58:15.552 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:15.551+0000 7fc1fe68b700 1 --2- 172.21.15.137:0/761000661 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fc1e002ff60 0x7fc1e0032420 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7fc1ec0096a0 tx=0x7fc1ec006b40).stop 2023-11-15T01:58:15.553 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:15.551+0000 7fc1fe68b700 1 -- 172.21.15.137:0/761000661 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc1f80f6510 msgr2=0x7fc1f8106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:58:15.553 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:15.551+0000 7fc1fe68b700 1 --2- 172.21.15.137:0/761000661 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc1f80f6510 0x7fc1f8106730 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7fc1e8013510 tx=0x7fc1e800b040).stop 2023-11-15T01:58:15.553 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:15.551+0000 7fc1fe68b700 1 -- 172.21.15.137:0/761000661 shutdown_connections 2023-11-15T01:58:15.553 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:15.551+0000 7fc1fe68b700 1 --2- 172.21.15.137:0/761000661 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fc1e002ff60 0x7fc1e0032420 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:58:15.553 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:15.551+0000 7fc1fe68b700 1 --2- 172.21.15.137:0/761000661 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc1f80f6510 0x7fc1f8106730 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:58:15.554 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:15.551+0000 7fc1fe68b700 1 -- 172.21.15.137:0/761000661 >> 172.21.15.137:0/761000661 conn(0x7fc1f80f1c70 msgr2=0x7fc1f80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:58:15.554 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:15.552+0000 7fc1fe68b700 1 -- 172.21.15.137:0/761000661 shutdown_connections 2023-11-15T01:58:15.554 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:15.552+0000 7fc1fe68b700 1 -- 172.21.15.137:0/761000661 wait complete. 2023-11-15T01:58:15.555 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:58:16.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:58:15 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:58:14.773931+0000 mgr.smithi055.szwngw (mgr.14164) 264 : cluster [DBG] pgmap v250: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:58:16.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:58:15 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:58:15.551018+0000 mon.smithi055 (mon.0) 316 : audit [DBG] from='client.? 172.21.15.137:0/761000661' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:58:16.953 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:58:16.953 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:58:18.133 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:18.130+0000 7f572599a700 1 -- 172.21.15.137:0/452033768 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f57200f6510 msgr2=0x7f57200f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:58:18.133 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:18.130+0000 7f572599a700 1 --2- 172.21.15.137:0/452033768 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f57200f6510 0x7f57200f6930 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7f5710009fb0 tx=0x7f5710009b30).stop 2023-11-15T01:58:18.134 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:18.131+0000 7f572599a700 1 -- 172.21.15.137:0/452033768 shutdown_connections 2023-11-15T01:58:18.134 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:18.131+0000 7f572599a700 1 --2- 172.21.15.137:0/452033768 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f57200f6510 0x7f57200f6930 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:58:18.134 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:18.131+0000 7f572599a700 1 -- 172.21.15.137:0/452033768 >> 172.21.15.137:0/452033768 conn(0x7f57200f1c70 msgr2=0x7f57200f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:58:18.134 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:18.131+0000 7f572599a700 1 -- 172.21.15.137:0/452033768 shutdown_connections 2023-11-15T01:58:18.134 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:18.131+0000 7f572599a700 1 -- 172.21.15.137:0/452033768 wait complete. 2023-11-15T01:58:18.135 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:18.131+0000 7f572599a700 1 Processor -- start 2023-11-15T01:58:18.135 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:18.131+0000 7f572599a700 1 -- start start 2023-11-15T01:58:18.135 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:18.131+0000 7f572599a700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f57200f6510 0x7f572006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:58:18.135 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:18.131+0000 7f572599a700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5720069e20 con 0x7f57200f6510 2023-11-15T01:58:18.135 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:18.132+0000 7f571effd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f57200f6510 0x7f572006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:58:18.136 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:18.132+0000 7f571effd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f57200f6510 0x7f572006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:33402/0 (socket says 172.21.15.137:33402) 2023-11-15T01:58:18.136 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:18.132+0000 7f571effd700 1 -- 172.21.15.137:0/4153091240 learned_addr learned my addr 172.21.15.137:0/4153091240 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:58:18.136 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:18.132+0000 7f571effd700 1 -- 172.21.15.137:0/4153091240 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5710005040 con 0x7f57200f6510 2023-11-15T01:58:18.136 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:18.132+0000 7f571effd700 1 --2- 172.21.15.137:0/4153091240 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f57200f6510 0x7f572006b9a0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7f5710013510 tx=0x7f571000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:58:18.137 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:18.132+0000 7f57177fe700 1 -- 172.21.15.137:0/4153091240 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5710009320 con 0x7f57200f6510 2023-11-15T01:58:18.137 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:18.132+0000 7f572599a700 1 -- 172.21.15.137:0/4153091240 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5720069fc0 con 0x7f57200f6510 2023-11-15T01:58:18.137 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:18.132+0000 7f57177fe700 1 -- 172.21.15.137:0/4153091240 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5710013f00 con 0x7f57200f6510 2023-11-15T01:58:18.137 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:18.132+0000 7f57177fe700 1 -- 172.21.15.137:0/4153091240 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5710014d90 con 0x7f57200f6510 2023-11-15T01:58:18.137 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:18.132+0000 7f572599a700 1 -- 172.21.15.137:0/4153091240 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f572006a450 con 0x7f57200f6510 2023-11-15T01:58:18.138 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:18.133+0000 7f57177fe700 1 -- 172.21.15.137:0/4153091240 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f57100263a0 con 0x7f57200f6510 2023-11-15T01:58:18.138 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:18.133+0000 7f572599a700 1 -- 172.21.15.137:0/4153091240 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5720059100 con 0x7f57200f6510 2023-11-15T01:58:18.139 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:18.133+0000 7f57177fe700 1 --2- 172.21.15.137:0/4153091240 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f570c02ff60 0x7f570c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:58:18.139 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:18.133+0000 7f57177fe700 1 -- 172.21.15.137:0/4153091240 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5710041300 con 0x7f57200f6510 2023-11-15T01:58:18.139 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:18.134+0000 7f571e7fc700 1 --2- 172.21.15.137:0/4153091240 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f570c02ff60 0x7f570c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:58:18.139 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:18.134+0000 7f571e7fc700 1 --2- 172.21.15.137:0/4153091240 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f570c02ff60 0x7f570c032420 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f57080096a0 tx=0x7f5708006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:58:18.140 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:18.138+0000 7f57177fe700 1 -- 172.21.15.137:0/4153091240 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f571001cb10 con 0x7f57200f6510 2023-11-15T01:58:18.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:58:17 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:58:16.774257+0000 mgr.smithi055.szwngw (mgr.14164) 265 : cluster [DBG] pgmap v251: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:58:18.341 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:18.341+0000 7f572599a700 1 -- 172.21.15.137:0/4153091240 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5720045ad0 con 0x7f57200f6510 2023-11-15T01:58:18.342 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:18.342+0000 7f57177fe700 1 -- 172.21.15.137:0/4153091240 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f571001cb10 con 0x7f57200f6510 2023-11-15T01:58:18.342 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:58:18.343 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:58:18.344 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:18.343+0000 7f572599a700 1 -- 172.21.15.137:0/4153091240 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f570c02ff60 msgr2=0x7f570c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:58:18.344 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:18.343+0000 7f572599a700 1 --2- 172.21.15.137:0/4153091240 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f570c02ff60 0x7f570c032420 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f57080096a0 tx=0x7f5708006b40).stop 2023-11-15T01:58:18.344 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:18.343+0000 7f572599a700 1 -- 172.21.15.137:0/4153091240 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f57200f6510 msgr2=0x7f572006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:58:18.344 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:18.343+0000 7f572599a700 1 --2- 172.21.15.137:0/4153091240 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f57200f6510 0x7f572006b9a0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7f5710013510 tx=0x7f571000b040).stop 2023-11-15T01:58:18.345 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:18.343+0000 7f572599a700 1 -- 172.21.15.137:0/4153091240 shutdown_connections 2023-11-15T01:58:18.345 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:18.343+0000 7f572599a700 1 --2- 172.21.15.137:0/4153091240 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f570c02ff60 0x7f570c032420 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:58:18.345 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:18.343+0000 7f572599a700 1 --2- 172.21.15.137:0/4153091240 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f57200f6510 0x7f572006b9a0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:58:18.345 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:18.343+0000 7f572599a700 1 -- 172.21.15.137:0/4153091240 >> 172.21.15.137:0/4153091240 conn(0x7f57200f1c70 msgr2=0x7f57200f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:58:18.346 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:18.343+0000 7f572599a700 1 -- 172.21.15.137:0/4153091240 shutdown_connections 2023-11-15T01:58:18.346 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:18.343+0000 7f572599a700 1 -- 172.21.15.137:0/4153091240 wait complete. 2023-11-15T01:58:18.346 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:58:19.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:58:18 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:58:18.342698+0000 mon.smithi055 (mon.0) 317 : audit [DBG] from='client.? 172.21.15.137:0/4153091240' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:58:19.722 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:58:19.722 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:58:20.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:58:19 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:58:18.774547+0000 mgr.smithi055.szwngw (mgr.14164) 266 : cluster [DBG] pgmap v252: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:58:20.803 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:20.801+0000 7fdaaf7fa700 1 -- 172.21.15.137:0/2906205740 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdaa80f5410 msgr2=0x7fdaa80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:58:20.804 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:20.801+0000 7fdaaf7fa700 1 --2- 172.21.15.137:0/2906205740 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdaa80f5410 0x7fdaa80f5830 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7fda98009fb0 tx=0x7fda98009b30).stop 2023-11-15T01:58:20.804 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:20.801+0000 7fdaaf7fa700 1 -- 172.21.15.137:0/2906205740 shutdown_connections 2023-11-15T01:58:20.804 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:20.801+0000 7fdaaf7fa700 1 --2- 172.21.15.137:0/2906205740 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdaa80f5410 0x7fdaa80f5830 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:58:20.804 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:20.801+0000 7fdaaf7fa700 1 -- 172.21.15.137:0/2906205740 >> 172.21.15.137:0/2906205740 conn(0x7fdaa80f0b70 msgr2=0x7fdaa80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:58:20.804 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:20.801+0000 7fdaaf7fa700 1 -- 172.21.15.137:0/2906205740 shutdown_connections 2023-11-15T01:58:20.805 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:20.801+0000 7fdaaf7fa700 1 -- 172.21.15.137:0/2906205740 wait complete. 2023-11-15T01:58:20.805 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:20.802+0000 7fdaaf7fa700 1 Processor -- start 2023-11-15T01:58:20.805 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:20.802+0000 7fdaaf7fa700 1 -- start start 2023-11-15T01:58:20.805 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:20.802+0000 7fdaaf7fa700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdaa80f5410 0x7fdaa8105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:58:20.805 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:20.802+0000 7fdaaf7fa700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaa8105b20 con 0x7fdaa80f5410 2023-11-15T01:58:20.806 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:20.803+0000 7fdaad596700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdaa80f5410 0x7fdaa8105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:58:20.806 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:20.803+0000 7fdaad596700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdaa80f5410 0x7fdaa8105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:45008/0 (socket says 172.21.15.137:45008) 2023-11-15T01:58:20.806 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:20.803+0000 7fdaad596700 1 -- 172.21.15.137:0/57003498 learned_addr learned my addr 172.21.15.137:0/57003498 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:58:20.806 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:20.803+0000 7fdaad596700 1 -- 172.21.15.137:0/57003498 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda98005040 con 0x7fdaa80f5410 2023-11-15T01:58:20.806 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:20.803+0000 7fdaad596700 1 --2- 172.21.15.137:0/57003498 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdaa80f5410 0x7fdaa8105630 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7fda98013510 tx=0x7fda9800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:58:20.808 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:20.803+0000 7fda9dffb700 1 -- 172.21.15.137:0/57003498 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda98009320 con 0x7fdaa80f5410 2023-11-15T01:58:20.808 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:20.803+0000 7fdaaf7fa700 1 -- 172.21.15.137:0/57003498 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdaa8105d20 con 0x7fdaa80f5410 2023-11-15T01:58:20.808 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:20.803+0000 7fda9dffb700 1 -- 172.21.15.137:0/57003498 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fda98013f00 con 0x7fdaa80f5410 2023-11-15T01:58:20.809 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:20.803+0000 7fda9dffb700 1 -- 172.21.15.137:0/57003498 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda98014d90 con 0x7fdaa80f5410 2023-11-15T01:58:20.809 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:20.803+0000 7fdaaf7fa700 1 -- 172.21.15.137:0/57003498 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdaa81061b0 con 0x7fdaa80f5410 2023-11-15T01:58:20.809 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:20.804+0000 7fdaaf7fa700 1 -- 172.21.15.137:0/57003498 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdaa8045ad0 con 0x7fdaa80f5410 2023-11-15T01:58:20.809 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:20.804+0000 7fda9dffb700 1 -- 172.21.15.137:0/57003498 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fda980095f0 con 0x7fdaa80f5410 2023-11-15T01:58:20.809 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:20.805+0000 7fda9dffb700 1 --2- 172.21.15.137:0/57003498 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fda9402ff60 0x7fda94032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:58:20.810 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:20.805+0000 7fda9dffb700 1 -- 172.21.15.137:0/57003498 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fda980410c0 con 0x7fdaa80f5410 2023-11-15T01:58:20.810 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:20.805+0000 7fdaacd95700 1 --2- 172.21.15.137:0/57003498 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fda9402ff60 0x7fda94032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:58:20.810 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:20.805+0000 7fdaacd95700 1 --2- 172.21.15.137:0/57003498 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fda9402ff60 0x7fda94032420 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7fdaa40096a0 tx=0x7fdaa4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:58:20.810 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:20.809+0000 7fda9dffb700 1 -- 172.21.15.137:0/57003498 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fda9801d7f0 con 0x7fdaa80f5410 2023-11-15T01:58:21.018 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:21.017+0000 7fdaaf7fa700 1 -- 172.21.15.137:0/57003498 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdaa80fae60 con 0x7fdaa80f5410 2023-11-15T01:58:21.019 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:21.018+0000 7fda9dffb700 1 -- 172.21.15.137:0/57003498 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fda98010230 con 0x7fdaa80f5410 2023-11-15T01:58:21.019 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:58:21.019 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:58:21.020 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:21.019+0000 7fdaaf7fa700 1 -- 172.21.15.137:0/57003498 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fda9402ff60 msgr2=0x7fda94032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:58:21.021 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:21.019+0000 7fdaaf7fa700 1 --2- 172.21.15.137:0/57003498 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fda9402ff60 0x7fda94032420 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7fdaa40096a0 tx=0x7fdaa4006b40).stop 2023-11-15T01:58:21.021 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:21.020+0000 7fdaaf7fa700 1 -- 172.21.15.137:0/57003498 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdaa80f5410 msgr2=0x7fdaa8105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:58:21.021 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:21.020+0000 7fdaaf7fa700 1 --2- 172.21.15.137:0/57003498 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdaa80f5410 0x7fdaa8105630 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7fda98013510 tx=0x7fda9800b040).stop 2023-11-15T01:58:21.021 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:21.020+0000 7fdaaf7fa700 1 -- 172.21.15.137:0/57003498 shutdown_connections 2023-11-15T01:58:21.021 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:21.020+0000 7fdaaf7fa700 1 --2- 172.21.15.137:0/57003498 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fda9402ff60 0x7fda94032420 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:58:21.021 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:21.020+0000 7fdaaf7fa700 1 --2- 172.21.15.137:0/57003498 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdaa80f5410 0x7fdaa8105630 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:58:21.022 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:21.020+0000 7fdaaf7fa700 1 -- 172.21.15.137:0/57003498 >> 172.21.15.137:0/57003498 conn(0x7fdaa80f0b70 msgr2=0x7fdaa80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:58:21.022 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:21.020+0000 7fdaaf7fa700 1 -- 172.21.15.137:0/57003498 shutdown_connections 2023-11-15T01:58:21.022 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:21.020+0000 7fdaaf7fa700 1 -- 172.21.15.137:0/57003498 wait complete. 2023-11-15T01:58:21.023 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:58:21.827 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:58:21 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:58:20.774843+0000 mgr.smithi055.szwngw (mgr.14164) 267 : cluster [DBG] pgmap v253: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:58:21.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:58:21 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:58:21.019077+0000 mon.smithi055 (mon.0) 318 : audit [DBG] from='client.? 172.21.15.137:0/57003498' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:58:22.394 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:58:22.395 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:58:23.604 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:23.603+0000 7fb0993a8700 1 -- 172.21.15.137:0/2613226336 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb0940f67c0 msgr2=0x7fb0940f6be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:58:23.605 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:23.603+0000 7fb0993a8700 1 --2- 172.21.15.137:0/2613226336 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb0940f67c0 0x7fb0940f6be0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0x7fb084009fb0 tx=0x7fb084009b30).stop 2023-11-15T01:58:23.605 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:23.603+0000 7fb0993a8700 1 -- 172.21.15.137:0/2613226336 shutdown_connections 2023-11-15T01:58:23.605 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:23.603+0000 7fb0993a8700 1 --2- 172.21.15.137:0/2613226336 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb0940f67c0 0x7fb0940f6be0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:58:23.605 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:23.603+0000 7fb0993a8700 1 -- 172.21.15.137:0/2613226336 >> 172.21.15.137:0/2613226336 conn(0x7fb0940f00f0 msgr2=0x7fb0940f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:58:23.605 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:23.604+0000 7fb0993a8700 1 -- 172.21.15.137:0/2613226336 shutdown_connections 2023-11-15T01:58:23.606 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:23.604+0000 7fb0993a8700 1 -- 172.21.15.137:0/2613226336 wait complete. 2023-11-15T01:58:23.608 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:23.604+0000 7fb0993a8700 1 Processor -- start 2023-11-15T01:58:23.608 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:23.604+0000 7fb0993a8700 1 -- start start 2023-11-15T01:58:23.608 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:23.604+0000 7fb0993a8700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb0940f67c0 0x7fb094105470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:58:23.608 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:23.604+0000 7fb0993a8700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb094105960 con 0x7fb0940f67c0 2023-11-15T01:58:23.608 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:23.605+0000 7fb092ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb0940f67c0 0x7fb094105470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:58:23.609 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:23.605+0000 7fb092ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb0940f67c0 0x7fb094105470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:45022/0 (socket says 172.21.15.137:45022) 2023-11-15T01:58:23.609 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:23.605+0000 7fb092ffd700 1 -- 172.21.15.137:0/1201312818 learned_addr learned my addr 172.21.15.137:0/1201312818 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:58:23.609 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:23.605+0000 7fb092ffd700 1 -- 172.21.15.137:0/1201312818 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb084005040 con 0x7fb0940f67c0 2023-11-15T01:58:23.609 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:23.605+0000 7fb092ffd700 1 --2- 172.21.15.137:0/1201312818 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb0940f67c0 0x7fb094105470 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7fb084013510 tx=0x7fb08400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:58:23.610 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:23.605+0000 7fb08b7fe700 1 -- 172.21.15.137:0/1201312818 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb084009320 con 0x7fb0940f67c0 2023-11-15T01:58:23.610 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:23.605+0000 7fb0993a8700 1 -- 172.21.15.137:0/1201312818 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb094105b60 con 0x7fb0940f67c0 2023-11-15T01:58:23.610 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:23.605+0000 7fb08b7fe700 1 -- 172.21.15.137:0/1201312818 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb084013f00 con 0x7fb0940f67c0 2023-11-15T01:58:23.610 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:23.605+0000 7fb08b7fe700 1 -- 172.21.15.137:0/1201312818 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb084014d90 con 0x7fb0940f67c0 2023-11-15T01:58:23.610 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:23.605+0000 7fb0993a8700 1 -- 172.21.15.137:0/1201312818 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb094105ff0 con 0x7fb0940f67c0 2023-11-15T01:58:23.611 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:23.606+0000 7fb08b7fe700 1 -- 172.21.15.137:0/1201312818 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb0840095f0 con 0x7fb0940f67c0 2023-11-15T01:58:23.611 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:23.606+0000 7fb0993a8700 1 -- 172.21.15.137:0/1201312818 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb094045ad0 con 0x7fb0940f67c0 2023-11-15T01:58:23.611 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:23.607+0000 7fb08b7fe700 1 --2- 172.21.15.137:0/1201312818 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fb08002ff60 0x7fb080032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:58:23.611 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:23.607+0000 7fb08b7fe700 1 -- 172.21.15.137:0/1201312818 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb0840410a0 con 0x7fb0940f67c0 2023-11-15T01:58:23.611 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:23.607+0000 7fb08bfff700 1 --2- 172.21.15.137:0/1201312818 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fb08002ff60 0x7fb080032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:58:23.612 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:23.607+0000 7fb08bfff700 1 --2- 172.21.15.137:0/1201312818 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fb08002ff60 0x7fb080032420 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7fb07c0096a0 tx=0x7fb07c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:58:23.612 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:23.612+0000 7fb08b7fe700 1 -- 172.21.15.137:0/1201312818 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb08401d7f0 con 0x7fb0940f67c0 2023-11-15T01:58:23.818 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:23.817+0000 7fb0993a8700 1 -- 172.21.15.137:0/1201312818 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb094106df0 con 0x7fb0940f67c0 2023-11-15T01:58:23.818 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:23.818+0000 7fb08b7fe700 1 -- 172.21.15.137:0/1201312818 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb084010230 con 0x7fb0940f67c0 2023-11-15T01:58:23.819 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:58:23.819 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:58:23.820 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:23.819+0000 7fb0993a8700 1 -- 172.21.15.137:0/1201312818 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fb08002ff60 msgr2=0x7fb080032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:58:23.820 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:23.819+0000 7fb0993a8700 1 --2- 172.21.15.137:0/1201312818 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fb08002ff60 0x7fb080032420 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7fb07c0096a0 tx=0x7fb07c006b40).stop 2023-11-15T01:58:23.821 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:23.819+0000 7fb0993a8700 1 -- 172.21.15.137:0/1201312818 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb0940f67c0 msgr2=0x7fb094105470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:58:23.821 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:23.819+0000 7fb0993a8700 1 --2- 172.21.15.137:0/1201312818 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb0940f67c0 0x7fb094105470 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7fb084013510 tx=0x7fb08400b040).stop 2023-11-15T01:58:23.821 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:23.819+0000 7fb0993a8700 1 -- 172.21.15.137:0/1201312818 shutdown_connections 2023-11-15T01:58:23.821 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:23.819+0000 7fb0993a8700 1 --2- 172.21.15.137:0/1201312818 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fb08002ff60 0x7fb080032420 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:58:23.822 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:23.819+0000 7fb0993a8700 1 --2- 172.21.15.137:0/1201312818 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb0940f67c0 0x7fb094105470 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:58:23.822 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:23.819+0000 7fb0993a8700 1 -- 172.21.15.137:0/1201312818 >> 172.21.15.137:0/1201312818 conn(0x7fb0940f00f0 msgr2=0x7fb0940f0c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:58:23.822 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:23.819+0000 7fb0993a8700 1 -- 172.21.15.137:0/1201312818 shutdown_connections 2023-11-15T01:58:23.822 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:23.819+0000 7fb0993a8700 1 -- 172.21.15.137:0/1201312818 wait complete. 2023-11-15T01:58:23.823 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:58:24.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:58:23 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:58:22.775193+0000 mgr.smithi055.szwngw (mgr.14164) 268 : cluster [DBG] pgmap v254: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:58:24.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:58:23 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:58:23.818751+0000 mon.smithi055 (mon.0) 319 : audit [DBG] from='client.? 172.21.15.137:0/1201312818' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:58:25.193 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:58:25.193 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:58:26.315 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:26.314+0000 7fd3ac823700 1 -- 172.21.15.137:0/1408839475 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd3a80f5410 msgr2=0x7fd3a80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:58:26.316 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:26.314+0000 7fd3ac823700 1 --2- 172.21.15.137:0/1408839475 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd3a80f5410 0x7fd3a80f5830 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7fd390009fb0 tx=0x7fd390009b30).stop 2023-11-15T01:58:26.316 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:26.314+0000 7fd3ac823700 1 -- 172.21.15.137:0/1408839475 shutdown_connections 2023-11-15T01:58:26.316 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:26.314+0000 7fd3ac823700 1 --2- 172.21.15.137:0/1408839475 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd3a80f5410 0x7fd3a80f5830 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:58:26.317 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:26.314+0000 7fd3ac823700 1 -- 172.21.15.137:0/1408839475 >> 172.21.15.137:0/1408839475 conn(0x7fd3a80f0b70 msgr2=0x7fd3a80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:58:26.317 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:26.314+0000 7fd3ac823700 1 -- 172.21.15.137:0/1408839475 shutdown_connections 2023-11-15T01:58:26.317 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:26.314+0000 7fd3ac823700 1 -- 172.21.15.137:0/1408839475 wait complete. 2023-11-15T01:58:26.317 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:26.315+0000 7fd3ac823700 1 Processor -- start 2023-11-15T01:58:26.317 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:26.315+0000 7fd3ac823700 1 -- start start 2023-11-15T01:58:26.319 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:26.315+0000 7fd3ac823700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd3a80f5410 0x7fd3a80fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:58:26.319 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:26.315+0000 7fd3ac823700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a80fb820 con 0x7fd3a80f5410 2023-11-15T01:58:26.319 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:26.315+0000 7fd3a659c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd3a80f5410 0x7fd3a80fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:58:26.320 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:26.315+0000 7fd3a659c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd3a80f5410 0x7fd3a80fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:45050/0 (socket says 172.21.15.137:45050) 2023-11-15T01:58:26.320 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:26.315+0000 7fd3a659c700 1 -- 172.21.15.137:0/976373052 learned_addr learned my addr 172.21.15.137:0/976373052 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:58:26.320 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:26.316+0000 7fd3a659c700 1 -- 172.21.15.137:0/976373052 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd390005040 con 0x7fd3a80f5410 2023-11-15T01:58:26.320 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:26.316+0000 7fd3a659c700 1 --2- 172.21.15.137:0/976373052 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd3a80f5410 0x7fd3a80fd3a0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7fd390013510 tx=0x7fd39000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:58:26.320 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:26.316+0000 7fd39effd700 1 -- 172.21.15.137:0/976373052 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd390009320 con 0x7fd3a80f5410 2023-11-15T01:58:26.321 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:26.316+0000 7fd3ac823700 1 -- 172.21.15.137:0/976373052 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3a80fb9c0 con 0x7fd3a80f5410 2023-11-15T01:58:26.321 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:26.316+0000 7fd39effd700 1 -- 172.21.15.137:0/976373052 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd390013f00 con 0x7fd3a80f5410 2023-11-15T01:58:26.321 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:26.316+0000 7fd39effd700 1 -- 172.21.15.137:0/976373052 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd390014d90 con 0x7fd3a80f5410 2023-11-15T01:58:26.321 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:26.316+0000 7fd3ac823700 1 -- 172.21.15.137:0/976373052 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3a80fbe50 con 0x7fd3a80f5410 2023-11-15T01:58:26.321 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:26.317+0000 7fd3ac823700 1 -- 172.21.15.137:0/976373052 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3a8059100 con 0x7fd3a80f5410 2023-11-15T01:58:26.322 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:26.317+0000 7fd39effd700 1 -- 172.21.15.137:0/976373052 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd3900095f0 con 0x7fd3a80f5410 2023-11-15T01:58:26.322 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:26.317+0000 7fd39effd700 1 --2- 172.21.15.137:0/976373052 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fd39402ff60 0x7fd394032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:58:26.322 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:26.317+0000 7fd39effd700 1 -- 172.21.15.137:0/976373052 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd3900410a0 con 0x7fd3a80f5410 2023-11-15T01:58:26.322 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:26.317+0000 7fd3a5d9b700 1 --2- 172.21.15.137:0/976373052 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fd39402ff60 0x7fd394032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:58:26.323 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:26.318+0000 7fd3a5d9b700 1 --2- 172.21.15.137:0/976373052 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fd39402ff60 0x7fd394032420 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7fd3980096a0 tx=0x7fd398006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:58:26.324 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:26.322+0000 7fd39effd700 1 -- 172.21.15.137:0/976373052 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd39001d7f0 con 0x7fd3a80f5410 2023-11-15T01:58:26.329 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:58:25 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:58:24.775489+0000 mgr.smithi055.szwngw (mgr.14164) 269 : cluster [DBG] pgmap v255: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:58:26.527 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:26.526+0000 7fd3ac823700 1 -- 172.21.15.137:0/976373052 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd3a80fae60 con 0x7fd3a80f5410 2023-11-15T01:58:26.528 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:26.527+0000 7fd39effd700 1 -- 172.21.15.137:0/976373052 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd390010230 con 0x7fd3a80f5410 2023-11-15T01:58:26.528 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:58:26.528 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:58:26.530 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:26.529+0000 7fd3ac823700 1 -- 172.21.15.137:0/976373052 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fd39402ff60 msgr2=0x7fd394032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:58:26.530 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:26.529+0000 7fd3ac823700 1 --2- 172.21.15.137:0/976373052 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fd39402ff60 0x7fd394032420 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7fd3980096a0 tx=0x7fd398006b40).stop 2023-11-15T01:58:26.530 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:26.529+0000 7fd3ac823700 1 -- 172.21.15.137:0/976373052 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd3a80f5410 msgr2=0x7fd3a80fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:58:26.531 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:26.529+0000 7fd3ac823700 1 --2- 172.21.15.137:0/976373052 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd3a80f5410 0x7fd3a80fd3a0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7fd390013510 tx=0x7fd39000b040).stop 2023-11-15T01:58:26.531 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:26.529+0000 7fd3ac823700 1 -- 172.21.15.137:0/976373052 shutdown_connections 2023-11-15T01:58:26.531 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:26.529+0000 7fd3ac823700 1 --2- 172.21.15.137:0/976373052 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fd39402ff60 0x7fd394032420 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:58:26.532 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:26.529+0000 7fd3ac823700 1 --2- 172.21.15.137:0/976373052 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd3a80f5410 0x7fd3a80fd3a0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:58:26.532 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:26.529+0000 7fd3ac823700 1 -- 172.21.15.137:0/976373052 >> 172.21.15.137:0/976373052 conn(0x7fd3a80f0b70 msgr2=0x7fd3a805fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:58:26.532 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:26.529+0000 7fd3ac823700 1 -- 172.21.15.137:0/976373052 shutdown_connections 2023-11-15T01:58:26.533 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:26.529+0000 7fd3ac823700 1 -- 172.21.15.137:0/976373052 wait complete. 2023-11-15T01:58:26.533 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:58:27.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:58:26 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:58:26.528179+0000 mon.smithi055 (mon.0) 320 : audit [DBG] from='client.? 172.21.15.137:0/976373052' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:58:27.915 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:58:27.915 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:58:28.327 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:58:27 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:58:26.775871+0000 mgr.smithi055.szwngw (mgr.14164) 270 : cluster [DBG] pgmap v256: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:58:29.043 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:29.043+0000 7fea01fdb700 1 -- 172.21.15.137:0/1925597215 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe9fc0f4740 msgr2=0x7fe9fc0f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:58:29.044 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:29.043+0000 7fea01fdb700 1 --2- 172.21.15.137:0/1925597215 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe9fc0f4740 0x7fe9fc0f4b60 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7fe9ec009fb0 tx=0x7fe9ec009b30).stop 2023-11-15T01:58:29.044 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:29.043+0000 7fea01fdb700 1 -- 172.21.15.137:0/1925597215 shutdown_connections 2023-11-15T01:58:29.044 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:29.043+0000 7fea01fdb700 1 --2- 172.21.15.137:0/1925597215 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe9fc0f4740 0x7fe9fc0f4b60 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:58:29.044 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:29.043+0000 7fea01fdb700 1 -- 172.21.15.137:0/1925597215 >> 172.21.15.137:0/1925597215 conn(0x7fe9fc0f0260 msgr2=0x7fe9fc0f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:58:29.045 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:29.043+0000 7fea01fdb700 1 -- 172.21.15.137:0/1925597215 shutdown_connections 2023-11-15T01:58:29.045 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:29.043+0000 7fea01fdb700 1 -- 172.21.15.137:0/1925597215 wait complete. 2023-11-15T01:58:29.045 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:29.043+0000 7fea01fdb700 1 Processor -- start 2023-11-15T01:58:29.047 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:29.043+0000 7fea01fdb700 1 -- start start 2023-11-15T01:58:29.047 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:29.043+0000 7fea01fdb700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe9fc0f4740 0x7fe9fc0f72f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:58:29.047 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:29.043+0000 7fea01fdb700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9fc0f5770 con 0x7fe9fc0f4740 2023-11-15T01:58:29.048 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:29.044+0000 7fe9fb7fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe9fc0f4740 0x7fe9fc0f72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:58:29.048 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:29.044+0000 7fe9fb7fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe9fc0f4740 0x7fe9fc0f72f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:38206/0 (socket says 172.21.15.137:38206) 2023-11-15T01:58:29.048 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:29.044+0000 7fe9fb7fe700 1 -- 172.21.15.137:0/1597034732 learned_addr learned my addr 172.21.15.137:0/1597034732 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:58:29.048 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:29.044+0000 7fe9fb7fe700 1 -- 172.21.15.137:0/1597034732 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9ec005040 con 0x7fe9fc0f4740 2023-11-15T01:58:29.048 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:29.044+0000 7fe9fb7fe700 1 --2- 172.21.15.137:0/1597034732 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe9fc0f4740 0x7fe9fc0f72f0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7fe9ec013510 tx=0x7fe9ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:58:29.049 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:29.044+0000 7fe9e3fff700 1 -- 172.21.15.137:0/1597034732 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe9ec009320 con 0x7fe9fc0f4740 2023-11-15T01:58:29.049 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:29.044+0000 7fea01fdb700 1 -- 172.21.15.137:0/1597034732 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9fc0f5910 con 0x7fe9fc0f4740 2023-11-15T01:58:29.049 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:29.044+0000 7fe9e3fff700 1 -- 172.21.15.137:0/1597034732 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe9ec013f00 con 0x7fe9fc0f4740 2023-11-15T01:58:29.049 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:29.044+0000 7fe9e3fff700 1 -- 172.21.15.137:0/1597034732 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe9ec014d90 con 0x7fe9fc0f4740 2023-11-15T01:58:29.049 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:29.045+0000 7fea01fdb700 1 -- 172.21.15.137:0/1597034732 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9fc0f5da0 con 0x7fe9fc0f4740 2023-11-15T01:58:29.050 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:29.045+0000 7fe9e3fff700 1 -- 172.21.15.137:0/1597034732 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe9ec0095f0 con 0x7fe9fc0f4740 2023-11-15T01:58:29.050 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:29.045+0000 7fea01fdb700 1 -- 172.21.15.137:0/1597034732 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9fc059100 con 0x7fe9fc0f4740 2023-11-15T01:58:29.050 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:29.046+0000 7fe9e3fff700 1 --2- 172.21.15.137:0/1597034732 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fe9e402ff60 0x7fe9e4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:58:29.050 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:29.046+0000 7fe9e3fff700 1 -- 172.21.15.137:0/1597034732 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe9ec040fd0 con 0x7fe9fc0f4740 2023-11-15T01:58:29.050 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:29.046+0000 7fe9faffd700 1 --2- 172.21.15.137:0/1597034732 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fe9e402ff60 0x7fe9e4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:58:29.050 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:29.046+0000 7fe9faffd700 1 --2- 172.21.15.137:0/1597034732 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fe9e402ff60 0x7fe9e4032420 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7fe9f00096a0 tx=0x7fe9f0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:58:29.051 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:29.051+0000 7fe9e3fff700 1 -- 172.21.15.137:0/1597034732 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe9ec014420 con 0x7fe9fc0f4740 2023-11-15T01:58:29.258 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:29.258+0000 7fea01fdb700 1 -- 172.21.15.137:0/1597034732 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe9fc0fbf40 con 0x7fe9fc0f4740 2023-11-15T01:58:29.259 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:29.258+0000 7fe9e3fff700 1 -- 172.21.15.137:0/1597034732 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe9ec01d7f0 con 0x7fe9fc0f4740 2023-11-15T01:58:29.259 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:58:29.259 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:58:29.260 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:29.260+0000 7fea01fdb700 1 -- 172.21.15.137:0/1597034732 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fe9e402ff60 msgr2=0x7fe9e4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:58:29.261 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:29.260+0000 7fea01fdb700 1 --2- 172.21.15.137:0/1597034732 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fe9e402ff60 0x7fe9e4032420 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7fe9f00096a0 tx=0x7fe9f0006b40).stop 2023-11-15T01:58:29.261 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:29.260+0000 7fea01fdb700 1 -- 172.21.15.137:0/1597034732 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe9fc0f4740 msgr2=0x7fe9fc0f72f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:58:29.261 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:29.260+0000 7fea01fdb700 1 --2- 172.21.15.137:0/1597034732 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe9fc0f4740 0x7fe9fc0f72f0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7fe9ec013510 tx=0x7fe9ec00b040).stop 2023-11-15T01:58:29.261 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:29.260+0000 7fea01fdb700 1 -- 172.21.15.137:0/1597034732 shutdown_connections 2023-11-15T01:58:29.261 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:29.260+0000 7fea01fdb700 1 --2- 172.21.15.137:0/1597034732 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fe9e402ff60 0x7fe9e4032420 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:58:29.262 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:29.260+0000 7fea01fdb700 1 --2- 172.21.15.137:0/1597034732 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe9fc0f4740 0x7fe9fc0f72f0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:58:29.262 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:29.260+0000 7fea01fdb700 1 -- 172.21.15.137:0/1597034732 >> 172.21.15.137:0/1597034732 conn(0x7fe9fc0f0260 msgr2=0x7fe9fc0f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:58:29.262 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:29.260+0000 7fea01fdb700 1 -- 172.21.15.137:0/1597034732 shutdown_connections 2023-11-15T01:58:29.262 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:29.260+0000 7fea01fdb700 1 -- 172.21.15.137:0/1597034732 wait complete. 2023-11-15T01:58:29.263 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:58:30.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:58:29 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:58:28.776274+0000 mgr.smithi055.szwngw (mgr.14164) 271 : cluster [DBG] pgmap v257: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:58:30.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:58:29 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:58:29.259396+0000 mon.smithi055 (mon.0) 321 : audit [DBG] from='client.? 172.21.15.137:0/1597034732' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:58:30.652 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:58:30.653 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:58:31.803 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:31.801+0000 7f45ce28f700 1 -- 172.21.15.137:0/3931027392 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f45c80f5410 msgr2=0x7f45c80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:58:31.803 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:31.801+0000 7f45ce28f700 1 --2- 172.21.15.137:0/3931027392 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f45c80f5410 0x7f45c80f5830 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7f45b0009fb0 tx=0x7f45b0009b30).stop 2023-11-15T01:58:31.803 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:31.801+0000 7f45ce28f700 1 -- 172.21.15.137:0/3931027392 shutdown_connections 2023-11-15T01:58:31.804 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:31.801+0000 7f45ce28f700 1 --2- 172.21.15.137:0/3931027392 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f45c80f5410 0x7f45c80f5830 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:58:31.804 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:31.801+0000 7f45ce28f700 1 -- 172.21.15.137:0/3931027392 >> 172.21.15.137:0/3931027392 conn(0x7f45c80f0b70 msgr2=0x7f45c80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:58:31.804 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:31.801+0000 7f45ce28f700 1 -- 172.21.15.137:0/3931027392 shutdown_connections 2023-11-15T01:58:31.804 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:31.801+0000 7f45ce28f700 1 -- 172.21.15.137:0/3931027392 wait complete. 2023-11-15T01:58:31.804 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:31.802+0000 7f45ce28f700 1 Processor -- start 2023-11-15T01:58:31.805 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:31.802+0000 7f45ce28f700 1 -- start start 2023-11-15T01:58:31.805 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:31.802+0000 7f45ce28f700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f45c80f5410 0x7f45c8109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:58:31.805 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:31.802+0000 7f45ce28f700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45c8060240 con 0x7f45c80f5410 2023-11-15T01:58:31.805 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:31.802+0000 7f45c7fff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f45c80f5410 0x7f45c8109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:58:31.805 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:31.802+0000 7f45c7fff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f45c80f5410 0x7f45c8109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:38210/0 (socket says 172.21.15.137:38210) 2023-11-15T01:58:31.805 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:31.802+0000 7f45c7fff700 1 -- 172.21.15.137:0/3914530302 learned_addr learned my addr 172.21.15.137:0/3914530302 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:58:31.806 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:31.803+0000 7f45c7fff700 1 -- 172.21.15.137:0/3914530302 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45b0005040 con 0x7f45c80f5410 2023-11-15T01:58:31.807 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:31.803+0000 7f45c7fff700 1 --2- 172.21.15.137:0/3914530302 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f45c80f5410 0x7f45c8109430 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f45b0014000 tx=0x7f45b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:58:31.808 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:31.803+0000 7f45c4ff9700 1 -- 172.21.15.137:0/3914530302 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f45b00095f0 con 0x7f45c80f5410 2023-11-15T01:58:31.808 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:31.803+0000 7f45ce28f700 1 -- 172.21.15.137:0/3914530302 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45c8060440 con 0x7f45c80f5410 2023-11-15T01:58:31.808 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:31.803+0000 7f45c4ff9700 1 -- 172.21.15.137:0/3914530302 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f45b0007e70 con 0x7f45c80f5410 2023-11-15T01:58:31.808 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:31.803+0000 7f45c4ff9700 1 -- 172.21.15.137:0/3914530302 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f45b0014ea0 con 0x7f45c80f5410 2023-11-15T01:58:31.808 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:31.803+0000 7f45ce28f700 1 -- 172.21.15.137:0/3914530302 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45c8109ae0 con 0x7f45c80f5410 2023-11-15T01:58:31.809 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:31.804+0000 7f45c4ff9700 1 -- 172.21.15.137:0/3914530302 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f45b0025070 con 0x7f45c80f5410 2023-11-15T01:58:31.809 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:31.804+0000 7f45ce28f700 1 -- 172.21.15.137:0/3914530302 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45c8045ad0 con 0x7f45c80f5410 2023-11-15T01:58:31.809 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:31.804+0000 7f45c4ff9700 1 --2- 172.21.15.137:0/3914530302 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f45b402ff60 0x7f45b4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:58:31.809 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:31.804+0000 7f45c4ff9700 1 -- 172.21.15.137:0/3914530302 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f45b0040a00 con 0x7f45c80f5410 2023-11-15T01:58:31.809 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:31.804+0000 7f45c77fe700 1 --2- 172.21.15.137:0/3914530302 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f45b402ff60 0x7f45b4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:58:31.809 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:31.805+0000 7f45c77fe700 1 --2- 172.21.15.137:0/3914530302 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f45b402ff60 0x7f45b4032420 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f45b80096a0 tx=0x7f45b8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:58:31.810 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:31.809+0000 7f45c4ff9700 1 -- 172.21.15.137:0/3914530302 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f45b00079c0 con 0x7f45c80f5410 2023-11-15T01:58:31.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:58:31 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:58:30.776715+0000 mgr.smithi055.szwngw (mgr.14164) 272 : cluster [DBG] pgmap v258: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:58:32.016 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:32.015+0000 7f45ce28f700 1 -- 172.21.15.137:0/3914530302 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f45c80fae60 con 0x7f45c80f5410 2023-11-15T01:58:32.017 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:32.016+0000 7f45c4ff9700 1 -- 172.21.15.137:0/3914530302 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f45b000f300 con 0x7f45c80f5410 2023-11-15T01:58:32.017 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:58:32.018 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:58:32.018 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:32.018+0000 7f45ce28f700 1 -- 172.21.15.137:0/3914530302 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f45b402ff60 msgr2=0x7f45b4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:58:32.019 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:32.018+0000 7f45ce28f700 1 --2- 172.21.15.137:0/3914530302 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f45b402ff60 0x7f45b4032420 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f45b80096a0 tx=0x7f45b8006b40).stop 2023-11-15T01:58:32.019 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:32.018+0000 7f45ce28f700 1 -- 172.21.15.137:0/3914530302 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f45c80f5410 msgr2=0x7f45c8109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:58:32.019 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:32.018+0000 7f45ce28f700 1 --2- 172.21.15.137:0/3914530302 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f45c80f5410 0x7f45c8109430 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f45b0014000 tx=0x7f45b000b040).stop 2023-11-15T01:58:32.019 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:32.018+0000 7f45ce28f700 1 -- 172.21.15.137:0/3914530302 shutdown_connections 2023-11-15T01:58:32.019 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:32.018+0000 7f45ce28f700 1 --2- 172.21.15.137:0/3914530302 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f45b402ff60 0x7f45b4032420 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:58:32.020 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:32.018+0000 7f45ce28f700 1 --2- 172.21.15.137:0/3914530302 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f45c80f5410 0x7f45c8109430 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:58:32.020 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:32.018+0000 7f45ce28f700 1 -- 172.21.15.137:0/3914530302 >> 172.21.15.137:0/3914530302 conn(0x7f45c80f0b70 msgr2=0x7f45c80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:58:32.020 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:32.018+0000 7f45ce28f700 1 -- 172.21.15.137:0/3914530302 shutdown_connections 2023-11-15T01:58:32.020 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:32.018+0000 7f45ce28f700 1 -- 172.21.15.137:0/3914530302 wait complete. 2023-11-15T01:58:32.021 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:58:32.827 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:58:32 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:58:32.017175+0000 mon.smithi055 (mon.0) 322 : audit [DBG] from='client.? 172.21.15.137:0/3914530302' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:58:33.411 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:58:33.412 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:58:33.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:58:33 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:58:32.776937+0000 mgr.smithi055.szwngw (mgr.14164) 273 : cluster [DBG] pgmap v259: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:58:34.574 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:34.573+0000 7f1fe9a8b700 1 -- 172.21.15.137:0/2715979376 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1fe40f5410 msgr2=0x7f1fe40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:58:34.574 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:34.573+0000 7f1fe9a8b700 1 --2- 172.21.15.137:0/2715979376 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1fe40f5410 0x7f1fe40f5830 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7f1fd8009fb0 tx=0x7f1fd8009b30).stop 2023-11-15T01:58:34.575 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:34.573+0000 7f1fe9a8b700 1 -- 172.21.15.137:0/2715979376 shutdown_connections 2023-11-15T01:58:34.575 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:34.573+0000 7f1fe9a8b700 1 --2- 172.21.15.137:0/2715979376 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1fe40f5410 0x7f1fe40f5830 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:58:34.575 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:34.573+0000 7f1fe9a8b700 1 -- 172.21.15.137:0/2715979376 >> 172.21.15.137:0/2715979376 conn(0x7f1fe40f0b70 msgr2=0x7f1fe40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:58:34.575 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:34.574+0000 7f1fe9a8b700 1 -- 172.21.15.137:0/2715979376 shutdown_connections 2023-11-15T01:58:34.575 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:34.574+0000 7f1fe9a8b700 1 -- 172.21.15.137:0/2715979376 wait complete. 2023-11-15T01:58:34.577 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:34.574+0000 7f1fe9a8b700 1 Processor -- start 2023-11-15T01:58:34.578 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:34.574+0000 7f1fe9a8b700 1 -- start start 2023-11-15T01:58:34.578 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:34.574+0000 7f1fe9a8b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1fe40f5410 0x7f1fe4109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:58:34.578 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:34.574+0000 7f1fe9a8b700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe4060240 con 0x7f1fe40f5410 2023-11-15T01:58:34.578 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:34.575+0000 7f1fe37fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1fe40f5410 0x7f1fe4109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:58:34.578 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:34.575+0000 7f1fe37fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1fe40f5410 0x7f1fe4109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:38224/0 (socket says 172.21.15.137:38224) 2023-11-15T01:58:34.579 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:34.575+0000 7f1fe37fe700 1 -- 172.21.15.137:0/4174522158 learned_addr learned my addr 172.21.15.137:0/4174522158 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:58:34.579 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:34.575+0000 7f1fe37fe700 1 -- 172.21.15.137:0/4174522158 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fd8005040 con 0x7f1fe40f5410 2023-11-15T01:58:34.579 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:34.575+0000 7f1fe37fe700 1 --2- 172.21.15.137:0/4174522158 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1fe40f5410 0x7f1fe4109430 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7f1fd8013510 tx=0x7f1fd800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:58:34.579 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:34.575+0000 7f1fcbfff700 1 -- 172.21.15.137:0/4174522158 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1fd8009320 con 0x7f1fe40f5410 2023-11-15T01:58:34.579 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:34.575+0000 7f1fe9a8b700 1 -- 172.21.15.137:0/4174522158 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fe4060440 con 0x7f1fe40f5410 2023-11-15T01:58:34.580 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:34.575+0000 7f1fcbfff700 1 -- 172.21.15.137:0/4174522158 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1fd8013f00 con 0x7f1fe40f5410 2023-11-15T01:58:34.580 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:34.575+0000 7f1fcbfff700 1 -- 172.21.15.137:0/4174522158 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1fd8014d90 con 0x7f1fe40f5410 2023-11-15T01:58:34.580 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:34.575+0000 7f1fe9a8b700 1 -- 172.21.15.137:0/4174522158 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fe4109ae0 con 0x7f1fe40f5410 2023-11-15T01:58:34.580 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:34.576+0000 7f1fe9a8b700 1 -- 172.21.15.137:0/4174522158 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fe4045ad0 con 0x7f1fe40f5410 2023-11-15T01:58:34.580 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:34.576+0000 7f1fcbfff700 1 -- 172.21.15.137:0/4174522158 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1fd80095f0 con 0x7f1fe40f5410 2023-11-15T01:58:34.581 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:34.576+0000 7f1fcbfff700 1 --2- 172.21.15.137:0/4174522158 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f1fcc02ff60 0x7f1fcc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:58:34.581 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:34.576+0000 7f1fcbfff700 1 -- 172.21.15.137:0/4174522158 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1fd8040fc0 con 0x7f1fe40f5410 2023-11-15T01:58:34.581 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:34.577+0000 7f1fe2ffd700 1 --2- 172.21.15.137:0/4174522158 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f1fcc02ff60 0x7f1fcc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:58:34.581 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:34.577+0000 7f1fe2ffd700 1 --2- 172.21.15.137:0/4174522158 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f1fcc02ff60 0x7f1fcc032420 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f1fd40096a0 tx=0x7f1fd4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:58:34.582 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:34.581+0000 7f1fcbfff700 1 -- 172.21.15.137:0/4174522158 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1fd8014420 con 0x7f1fe40f5410 2023-11-15T01:58:34.791 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:34.790+0000 7f1fe9a8b700 1 -- 172.21.15.137:0/4174522158 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1fe40fae60 con 0x7f1fe40f5410 2023-11-15T01:58:34.791 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:34.790+0000 7f1fcbfff700 1 -- 172.21.15.137:0/4174522158 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1fd8010230 con 0x7f1fe40f5410 2023-11-15T01:58:34.792 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:58:34.792 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:58:34.793 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:34.792+0000 7f1fe9a8b700 1 -- 172.21.15.137:0/4174522158 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f1fcc02ff60 msgr2=0x7f1fcc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:58:34.793 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:34.792+0000 7f1fe9a8b700 1 --2- 172.21.15.137:0/4174522158 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f1fcc02ff60 0x7f1fcc032420 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f1fd40096a0 tx=0x7f1fd4006b40).stop 2023-11-15T01:58:34.793 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:34.792+0000 7f1fe9a8b700 1 -- 172.21.15.137:0/4174522158 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1fe40f5410 msgr2=0x7f1fe4109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:58:34.793 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:34.792+0000 7f1fe9a8b700 1 --2- 172.21.15.137:0/4174522158 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1fe40f5410 0x7f1fe4109430 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7f1fd8013510 tx=0x7f1fd800b040).stop 2023-11-15T01:58:34.794 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:34.792+0000 7f1fe9a8b700 1 -- 172.21.15.137:0/4174522158 shutdown_connections 2023-11-15T01:58:34.794 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:34.792+0000 7f1fe9a8b700 1 --2- 172.21.15.137:0/4174522158 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f1fcc02ff60 0x7f1fcc032420 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:58:34.794 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:34.792+0000 7f1fe9a8b700 1 --2- 172.21.15.137:0/4174522158 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1fe40f5410 0x7f1fe4109430 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:58:34.794 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:34.792+0000 7f1fe9a8b700 1 -- 172.21.15.137:0/4174522158 >> 172.21.15.137:0/4174522158 conn(0x7f1fe40f0b70 msgr2=0x7f1fe40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:58:34.794 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:34.792+0000 7f1fe9a8b700 1 -- 172.21.15.137:0/4174522158 shutdown_connections 2023-11-15T01:58:34.795 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:34.792+0000 7f1fe9a8b700 1 -- 172.21.15.137:0/4174522158 wait complete. 2023-11-15T01:58:34.795 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:58:35.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:58:34 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:58:34.791632+0000 mon.smithi055 (mon.0) 323 : audit [DBG] from='client.? 172.21.15.137:0/4174522158' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:58:36.180 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:58:36.181 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:58:36.327 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:58:35 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:58:34.777156+0000 mgr.smithi055.szwngw (mgr.14164) 274 : cluster [DBG] pgmap v260: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:58:37.338 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:37.336+0000 7f232c40f700 1 -- 172.21.15.137:0/1442301955 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f23240f5410 msgr2=0x7f23240f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:58:37.339 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:37.336+0000 7f232c40f700 1 --2- 172.21.15.137:0/1442301955 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f23240f5410 0x7f23240f5830 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7f2318009fb0 tx=0x7f2318009b30).stop 2023-11-15T01:58:37.339 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:37.336+0000 7f232c40f700 1 -- 172.21.15.137:0/1442301955 shutdown_connections 2023-11-15T01:58:37.339 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:37.336+0000 7f232c40f700 1 --2- 172.21.15.137:0/1442301955 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f23240f5410 0x7f23240f5830 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:58:37.340 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:37.336+0000 7f232c40f700 1 -- 172.21.15.137:0/1442301955 >> 172.21.15.137:0/1442301955 conn(0x7f23240f0b70 msgr2=0x7f23240f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:58:37.340 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:37.336+0000 7f232c40f700 1 -- 172.21.15.137:0/1442301955 shutdown_connections 2023-11-15T01:58:37.340 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:37.336+0000 7f232c40f700 1 -- 172.21.15.137:0/1442301955 wait complete. 2023-11-15T01:58:37.340 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:37.337+0000 7f232c40f700 1 Processor -- start 2023-11-15T01:58:37.340 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:37.337+0000 7f232c40f700 1 -- start start 2023-11-15T01:58:37.341 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:37.337+0000 7f232c40f700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f23240f5410 0x7f23241066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:58:37.341 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:37.337+0000 7f232c40f700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2324106bd0 con 0x7f23240f5410 2023-11-15T01:58:37.341 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:37.337+0000 7f232a1ab700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f23240f5410 0x7f23241066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:58:37.341 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:37.337+0000 7f232a1ab700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f23240f5410 0x7f23241066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:38240/0 (socket says 172.21.15.137:38240) 2023-11-15T01:58:37.341 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:37.337+0000 7f232a1ab700 1 -- 172.21.15.137:0/2311168040 learned_addr learned my addr 172.21.15.137:0/2311168040 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:58:37.342 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:37.337+0000 7f232a1ab700 1 -- 172.21.15.137:0/2311168040 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2318005040 con 0x7f23240f5410 2023-11-15T01:58:37.342 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:37.338+0000 7f232a1ab700 1 --2- 172.21.15.137:0/2311168040 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f23240f5410 0x7f23241066e0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7f2318013510 tx=0x7f231800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:58:37.342 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:37.338+0000 7f2316ffd700 1 -- 172.21.15.137:0/2311168040 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2318009320 con 0x7f23240f5410 2023-11-15T01:58:37.342 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:37.338+0000 7f232c40f700 1 -- 172.21.15.137:0/2311168040 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2324106dd0 con 0x7f23240f5410 2023-11-15T01:58:37.343 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:37.338+0000 7f2316ffd700 1 -- 172.21.15.137:0/2311168040 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2318013f00 con 0x7f23240f5410 2023-11-15T01:58:37.343 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:37.338+0000 7f2316ffd700 1 -- 172.21.15.137:0/2311168040 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2318014d90 con 0x7f23240f5410 2023-11-15T01:58:37.343 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:37.338+0000 7f232c40f700 1 -- 172.21.15.137:0/2311168040 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2324107260 con 0x7f23240f5410 2023-11-15T01:58:37.344 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:37.339+0000 7f232c40f700 1 -- 172.21.15.137:0/2311168040 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2324045ad0 con 0x7f23240f5410 2023-11-15T01:58:37.344 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:37.339+0000 7f2316ffd700 1 -- 172.21.15.137:0/2311168040 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f23180095f0 con 0x7f23240f5410 2023-11-15T01:58:37.345 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:37.339+0000 7f2316ffd700 1 --2- 172.21.15.137:0/2311168040 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f231002ff60 0x7f2310032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:58:37.345 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:37.339+0000 7f2316ffd700 1 -- 172.21.15.137:0/2311168040 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2318041000 con 0x7f23240f5410 2023-11-15T01:58:37.345 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:37.339+0000 7f23299aa700 1 --2- 172.21.15.137:0/2311168040 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f231002ff60 0x7f2310032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:58:37.345 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:37.339+0000 7f23299aa700 1 --2- 172.21.15.137:0/2311168040 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f231002ff60 0x7f2310032420 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f23200096a0 tx=0x7f2320006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:58:37.345 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:37.343+0000 7f2316ffd700 1 -- 172.21.15.137:0/2311168040 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2318014420 con 0x7f23240f5410 2023-11-15T01:58:37.547 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:37.546+0000 7f232c40f700 1 -- 172.21.15.137:0/2311168040 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2324108110 con 0x7f23240f5410 2023-11-15T01:58:37.548 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:37.547+0000 7f2316ffd700 1 -- 172.21.15.137:0/2311168040 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f231801d7f0 con 0x7f23240f5410 2023-11-15T01:58:37.548 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:58:37.549 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:58:37.549 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:37.548+0000 7f232c40f700 1 -- 172.21.15.137:0/2311168040 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f231002ff60 msgr2=0x7f2310032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:58:37.550 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:37.548+0000 7f232c40f700 1 --2- 172.21.15.137:0/2311168040 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f231002ff60 0x7f2310032420 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f23200096a0 tx=0x7f2320006b40).stop 2023-11-15T01:58:37.550 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:37.548+0000 7f232c40f700 1 -- 172.21.15.137:0/2311168040 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f23240f5410 msgr2=0x7f23241066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:58:37.550 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:37.548+0000 7f232c40f700 1 --2- 172.21.15.137:0/2311168040 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f23240f5410 0x7f23241066e0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7f2318013510 tx=0x7f231800b040).stop 2023-11-15T01:58:37.550 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:37.548+0000 7f232c40f700 1 -- 172.21.15.137:0/2311168040 shutdown_connections 2023-11-15T01:58:37.551 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:37.548+0000 7f232c40f700 1 --2- 172.21.15.137:0/2311168040 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f231002ff60 0x7f2310032420 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:58:37.551 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:37.548+0000 7f232c40f700 1 --2- 172.21.15.137:0/2311168040 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f23240f5410 0x7f23241066e0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:58:37.551 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:37.548+0000 7f232c40f700 1 -- 172.21.15.137:0/2311168040 >> 172.21.15.137:0/2311168040 conn(0x7f23240f0b70 msgr2=0x7f232405fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:58:37.551 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:37.548+0000 7f232c40f700 1 -- 172.21.15.137:0/2311168040 shutdown_connections 2023-11-15T01:58:37.552 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:37.548+0000 7f232c40f700 1 -- 172.21.15.137:0/2311168040 wait complete. 2023-11-15T01:58:37.552 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:58:38.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:58:37 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:58:36.777519+0000 mgr.smithi055.szwngw (mgr.14164) 275 : cluster [DBG] pgmap v261: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:58:38.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:58:37 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:58:37.547874+0000 mon.smithi055 (mon.0) 324 : audit [DBG] from='client.? 172.21.15.137:0/2311168040' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:58:38.953 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:58:38.954 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:58:40.102 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:40.100+0000 7f0acddce700 1 -- 172.21.15.137:0/3012405004 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0ac80f5410 msgr2=0x7f0ac80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:58:40.102 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:40.100+0000 7f0acddce700 1 --2- 172.21.15.137:0/3012405004 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0ac80f5410 0x7f0ac80f5830 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7f0ab0009fb0 tx=0x7f0ab0009b30).stop 2023-11-15T01:58:40.103 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:40.100+0000 7f0acddce700 1 -- 172.21.15.137:0/3012405004 shutdown_connections 2023-11-15T01:58:40.103 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:40.100+0000 7f0acddce700 1 --2- 172.21.15.137:0/3012405004 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0ac80f5410 0x7f0ac80f5830 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:58:40.103 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:40.100+0000 7f0acddce700 1 -- 172.21.15.137:0/3012405004 >> 172.21.15.137:0/3012405004 conn(0x7f0ac80f0b70 msgr2=0x7f0ac80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:58:40.103 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:40.100+0000 7f0acddce700 1 -- 172.21.15.137:0/3012405004 shutdown_connections 2023-11-15T01:58:40.104 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:40.100+0000 7f0acddce700 1 -- 172.21.15.137:0/3012405004 wait complete. 2023-11-15T01:58:40.104 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:40.100+0000 7f0acddce700 1 Processor -- start 2023-11-15T01:58:40.104 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:40.100+0000 7f0acddce700 1 -- start start 2023-11-15T01:58:40.104 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:40.100+0000 7f0acddce700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0ac80f5410 0x7f0ac80fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:58:40.105 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:40.100+0000 7f0acddce700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ac80fb820 con 0x7f0ac80f5410 2023-11-15T01:58:40.105 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:40.101+0000 7f0ac77fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0ac80f5410 0x7f0ac80fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:58:40.105 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:40.101+0000 7f0ac77fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0ac80f5410 0x7f0ac80fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:60900/0 (socket says 172.21.15.137:60900) 2023-11-15T01:58:40.105 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:40.101+0000 7f0ac77fe700 1 -- 172.21.15.137:0/2926020335 learned_addr learned my addr 172.21.15.137:0/2926020335 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:58:40.105 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:40.101+0000 7f0ac77fe700 1 -- 172.21.15.137:0/2926020335 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ab0005040 con 0x7f0ac80f5410 2023-11-15T01:58:40.106 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:40.101+0000 7f0ac77fe700 1 --2- 172.21.15.137:0/2926020335 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0ac80f5410 0x7f0ac80fd3a0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7f0ab0013510 tx=0x7f0ab000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:58:40.106 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:40.101+0000 7f0abffff700 1 -- 172.21.15.137:0/2926020335 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0ab0009320 con 0x7f0ac80f5410 2023-11-15T01:58:40.106 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:40.101+0000 7f0acddce700 1 -- 172.21.15.137:0/2926020335 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ac80fb9c0 con 0x7f0ac80f5410 2023-11-15T01:58:40.107 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:40.101+0000 7f0abffff700 1 -- 172.21.15.137:0/2926020335 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0ab0013f00 con 0x7f0ac80f5410 2023-11-15T01:58:40.107 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:40.101+0000 7f0abffff700 1 -- 172.21.15.137:0/2926020335 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0ab0014d90 con 0x7f0ac80f5410 2023-11-15T01:58:40.107 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:40.101+0000 7f0acddce700 1 -- 172.21.15.137:0/2926020335 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ac80fbe50 con 0x7f0ac80f5410 2023-11-15T01:58:40.108 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:40.102+0000 7f0abffff700 1 -- 172.21.15.137:0/2926020335 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0ab00095f0 con 0x7f0ac80f5410 2023-11-15T01:58:40.108 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:40.102+0000 7f0acddce700 1 -- 172.21.15.137:0/2926020335 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ac8059100 con 0x7f0ac80f5410 2023-11-15T01:58:40.109 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:40.102+0000 7f0abffff700 1 --2- 172.21.15.137:0/2926020335 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f0ab402ff60 0x7f0ab4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:58:40.109 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:40.102+0000 7f0abffff700 1 -- 172.21.15.137:0/2926020335 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0ab0041000 con 0x7f0ac80f5410 2023-11-15T01:58:40.109 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:40.103+0000 7f0ac6ffd700 1 --2- 172.21.15.137:0/2926020335 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f0ab402ff60 0x7f0ab4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:58:40.109 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:40.103+0000 7f0ac6ffd700 1 --2- 172.21.15.137:0/2926020335 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f0ab402ff60 0x7f0ab4032420 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f0ab80096a0 tx=0x7f0ab8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:58:40.110 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:40.107+0000 7f0abffff700 1 -- 172.21.15.137:0/2926020335 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0ab0014420 con 0x7f0ac80f5410 2023-11-15T01:58:40.313 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:40.312+0000 7f0acddce700 1 -- 172.21.15.137:0/2926020335 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0ac80fae60 con 0x7f0ac80f5410 2023-11-15T01:58:40.314 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:40.313+0000 7f0abffff700 1 -- 172.21.15.137:0/2926020335 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0ab0010230 con 0x7f0ac80f5410 2023-11-15T01:58:40.314 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:58:40.314 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:58:40.316 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:40.315+0000 7f0acddce700 1 -- 172.21.15.137:0/2926020335 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f0ab402ff60 msgr2=0x7f0ab4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:58:40.316 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:40.315+0000 7f0acddce700 1 --2- 172.21.15.137:0/2926020335 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f0ab402ff60 0x7f0ab4032420 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f0ab80096a0 tx=0x7f0ab8006b40).stop 2023-11-15T01:58:40.316 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:40.315+0000 7f0acddce700 1 -- 172.21.15.137:0/2926020335 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0ac80f5410 msgr2=0x7f0ac80fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:58:40.316 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:40.315+0000 7f0acddce700 1 --2- 172.21.15.137:0/2926020335 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0ac80f5410 0x7f0ac80fd3a0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7f0ab0013510 tx=0x7f0ab000b040).stop 2023-11-15T01:58:40.317 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:40.315+0000 7f0acddce700 1 -- 172.21.15.137:0/2926020335 shutdown_connections 2023-11-15T01:58:40.317 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:40.315+0000 7f0acddce700 1 --2- 172.21.15.137:0/2926020335 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7f0ab402ff60 0x7f0ab4032420 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:58:40.317 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:40.315+0000 7f0acddce700 1 --2- 172.21.15.137:0/2926020335 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0ac80f5410 0x7f0ac80fd3a0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:58:40.317 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:40.315+0000 7f0acddce700 1 -- 172.21.15.137:0/2926020335 >> 172.21.15.137:0/2926020335 conn(0x7f0ac80f0b70 msgr2=0x7f0ac805fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:58:40.317 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:40.315+0000 7f0acddce700 1 -- 172.21.15.137:0/2926020335 shutdown_connections 2023-11-15T01:58:40.317 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:40.315+0000 7f0acddce700 1 -- 172.21.15.137:0/2926020335 wait complete. 2023-11-15T01:58:40.318 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:58:40.327 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:58:39 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:58:38.777850+0000 mgr.smithi055.szwngw (mgr.14164) 276 : cluster [DBG] pgmap v262: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:58:41.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:58:40 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: audit 2023-11-15T01:58:40.314081+0000 mon.smithi055 (mon.0) 325 : audit [DBG] from='client.? 172.21.15.137:0/2926020335' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T01:58:41.678 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T01:58:41.679 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T01:58:42.328 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:58:41 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: cluster 2023-11-15T01:58:40.778150+0000 mgr.smithi055.szwngw (mgr.14164) 277 : cluster [DBG] pgmap v263: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T01:58:42.851 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:42.850+0000 7fbb3e45f700 1 -- 172.21.15.137:0/3066996234 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbb380f6480 msgr2=0x7fbb380f68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:58:42.852 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:42.850+0000 7fbb3e45f700 1 --2- 172.21.15.137:0/3066996234 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbb380f6480 0x7fbb380f68a0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 rx=0x7fbb28009fb0 tx=0x7fbb28009b30).stop 2023-11-15T01:58:42.852 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:42.850+0000 7fbb3e45f700 1 -- 172.21.15.137:0/3066996234 shutdown_connections 2023-11-15T01:58:42.852 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:42.850+0000 7fbb3e45f700 1 --2- 172.21.15.137:0/3066996234 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbb380f6480 0x7fbb380f68a0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:58:42.852 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:42.850+0000 7fbb3e45f700 1 -- 172.21.15.137:0/3066996234 >> 172.21.15.137:0/3066996234 conn(0x7fbb380f1c00 msgr2=0x7fbb380f4020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:58:42.852 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:42.850+0000 7fbb3e45f700 1 -- 172.21.15.137:0/3066996234 shutdown_connections 2023-11-15T01:58:42.852 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:42.850+0000 7fbb3e45f700 1 -- 172.21.15.137:0/3066996234 wait complete. 2023-11-15T01:58:42.853 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:42.851+0000 7fbb3e45f700 1 Processor -- start 2023-11-15T01:58:42.855 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:42.851+0000 7fbb3e45f700 1 -- start start 2023-11-15T01:58:42.855 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:42.851+0000 7fbb3e45f700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbb380f6480 0x7fbb381066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:58:42.855 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:42.851+0000 7fbb3e45f700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb38106bc0 con 0x7fbb380f6480 2023-11-15T01:58:42.855 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:42.852+0000 7fbb37fff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbb380f6480 0x7fbb381066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:58:42.855 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:42.852+0000 7fbb37fff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbb380f6480 0x7fbb381066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.137:60928/0 (socket says 172.21.15.137:60928) 2023-11-15T01:58:42.856 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:42.852+0000 7fbb37fff700 1 -- 172.21.15.137:0/2686212098 learned_addr learned my addr 172.21.15.137:0/2686212098 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T01:58:42.856 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:42.852+0000 7fbb37fff700 1 -- 172.21.15.137:0/2686212098 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb28005040 con 0x7fbb380f6480 2023-11-15T01:58:42.856 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:42.852+0000 7fbb37fff700 1 --2- 172.21.15.137:0/2686212098 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbb380f6480 0x7fbb381066d0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 rx=0x7fbb28013510 tx=0x7fbb2800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:58:42.856 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:42.852+0000 7fbb34ff9700 1 -- 172.21.15.137:0/2686212098 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbb28009320 con 0x7fbb380f6480 2023-11-15T01:58:42.856 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:42.852+0000 7fbb3e45f700 1 -- 172.21.15.137:0/2686212098 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb38106dc0 con 0x7fbb380f6480 2023-11-15T01:58:42.857 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:42.852+0000 7fbb34ff9700 1 -- 172.21.15.137:0/2686212098 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbb28013f00 con 0x7fbb380f6480 2023-11-15T01:58:42.857 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:42.852+0000 7fbb34ff9700 1 -- 172.21.15.137:0/2686212098 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbb28014d90 con 0x7fbb380f6480 2023-11-15T01:58:42.857 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:42.852+0000 7fbb3e45f700 1 -- 172.21.15.137:0/2686212098 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb38107250 con 0x7fbb380f6480 2023-11-15T01:58:42.857 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:42.853+0000 7fbb34ff9700 1 -- 172.21.15.137:0/2686212098 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbb280095f0 con 0x7fbb380f6480 2023-11-15T01:58:42.857 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:42.853+0000 7fbb3e45f700 1 -- 172.21.15.137:0/2686212098 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb38045ad0 con 0x7fbb380f6480 2023-11-15T01:58:42.858 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:42.853+0000 7fbb34ff9700 1 --2- 172.21.15.137:0/2686212098 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fbb2402ff60 0x7fbb24032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T01:58:42.858 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:42.853+0000 7fbb34ff9700 1 -- 172.21.15.137:0/2686212098 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbb28041090 con 0x7fbb380f6480 2023-11-15T01:58:42.858 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:42.853+0000 7fbb377fe700 1 --2- 172.21.15.137:0/2686212098 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fbb2402ff60 0x7fbb24032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T01:58:42.858 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:42.854+0000 7fbb377fe700 1 --2- 172.21.15.137:0/2686212098 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fbb2402ff60 0x7fbb24032420 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7fbb200096a0 tx=0x7fbb20006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T01:58:42.859 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:42.859+0000 7fbb34ff9700 1 -- 172.21.15.137:0/2686212098 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbb28014420 con 0x7fbb380f6480 2023-11-15T01:58:43.066 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:43.065+0000 7fbb3e45f700 1 -- 172.21.15.137:0/2686212098 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbb38108100 con 0x7fbb380f6480 2023-11-15T01:58:43.066 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:43.066+0000 7fbb34ff9700 1 -- 172.21.15.137:0/2686212098 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbb2801d7f0 con 0x7fbb380f6480 2023-11-15T01:58:43.067 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T01:58:43.067 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"0cb4fc0c-8359-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T01:48:45.055593Z","created":"2023-11-15T01:48:45.055593Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T01:58:43.068 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:43.067+0000 7fbb3e45f700 1 -- 172.21.15.137:0/2686212098 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fbb2402ff60 msgr2=0x7fbb24032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:58:43.068 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:43.067+0000 7fbb3e45f700 1 --2- 172.21.15.137:0/2686212098 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fbb2402ff60 0x7fbb24032420 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7fbb200096a0 tx=0x7fbb20006b40).stop 2023-11-15T01:58:43.068 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:43.067+0000 7fbb3e45f700 1 -- 172.21.15.137:0/2686212098 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbb380f6480 msgr2=0x7fbb381066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T01:58:43.069 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:43.067+0000 7fbb3e45f700 1 --2- 172.21.15.137:0/2686212098 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbb380f6480 0x7fbb381066d0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 rx=0x7fbb28013510 tx=0x7fbb2800b040).stop 2023-11-15T01:58:43.069 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:43.067+0000 7fbb3e45f700 1 -- 172.21.15.137:0/2686212098 shutdown_connections 2023-11-15T01:58:43.069 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:43.067+0000 7fbb3e45f700 1 --2- 172.21.15.137:0/2686212098 >> [v2:172.21.15.55:6800/157802929,v1:172.21.15.55:6801/157802929] conn(0x7fbb2402ff60 0x7fbb24032420 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:58:43.069 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:43.067+0000 7fbb3e45f700 1 --2- 172.21.15.137:0/2686212098 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbb380f6480 0x7fbb381066d0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T01:58:43.069 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:43.067+0000 7fbb3e45f700 1 -- 172.21.15.137:0/2686212098 >> 172.21.15.137:0/2686212098 conn(0x7fbb380f1c00 msgr2=0x7fbb380f2720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T01:58:43.070 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:43.067+0000 7fbb3e45f700 1 -- 172.21.15.137:0/2686212098 shutdown_connections 2023-11-15T01:58:43.070 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T01:58:43.067+0000 7fbb3e45f700 1 -- 172.21.15.137:0/2686212098 wait complete. 2023-11-15T01:58:43.070 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T01:58:43.461 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds 2023-11-15T01:58:43.462 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2023-11-15T01:58:43.463 DEBUG:teuthology.orchestra.run.smithi055:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-11-15T01:58:43.488 DEBUG:teuthology.orchestra.run.smithi137:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-11-15T01:58:43.507 INFO:tasks.cephadm:Stopping all daemons... 2023-11-15T01:58:43.508 INFO:tasks.cephadm.mon.smithi055:Stopping mon.smithi055... 2023-11-15T01:58:43.508 DEBUG:teuthology.orchestra.run.smithi055:> sudo systemctl stop ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7@mon.smithi055 2023-11-15T01:58:43.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:58:43 smithi055 systemd[1]: Stopping Ceph mon.smithi055 for 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7... 2023-11-15T01:58:43.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:58:43 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:58:43.644+0000 7fe889274700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi055 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2023-11-15T01:58:43.829 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Nov 15 01:58:43 smithi055 ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7-mon.smithi055[141575]: debug 2023-11-15T01:58:43.644+0000 7fe889274700 -1 mon.smithi055@0(leader) e1 *** Got Signal Terminated *** 2023-11-15T01:58:44.135 DEBUG:teuthology.orchestra.run.smithi055:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7@mon.smithi055.service' 2023-11-15T01:58:44.188 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-15T01:58:44.189 INFO:tasks.cephadm.mon.smithi055:Stopped mon.smithi055 2023-11-15T01:58:44.189 INFO:tasks.cephadm.mon.smithi137:Stopping mon.smithi137... 2023-11-15T01:58:44.190 DEBUG:teuthology.orchestra.run.smithi137:> sudo systemctl stop ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7@mon.smithi137 2023-11-15T01:58:44.224 INFO:teuthology.orchestra.run.smithi137.stderr:Failed to stop ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7@mon.smithi137.service: Unit ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7@mon.smithi137.service not loaded. 2023-11-15T01:58:44.226 DEBUG:teuthology.orchestra.run:got remote process result: 5 2023-11-15T01:58:44.227 ERROR:tasks.cephadm:Failed to stop "ceph.mon.smithi137" Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 96, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi137 with status 5: 'sudo systemctl stop ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7@mon.smithi137' 2023-11-15T01:58:44.227 INFO:tasks.cephadm:Archiving crash dumps... 2023-11-15T01:58:44.230 DEBUG:teuthology.misc:Transferring archived files from smithi055:/var/lib/ceph/0cb4fc0c-8359-11ee-8dbe-212e2dc638e7/crash to /home/teuthworker/archive/yuriw-2023-11-14_20:31:57-fs-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458065/remote/ubuntu@smithi055.front.sepia.ceph.com/crash 2023-11-15T01:58:44.233 DEBUG:teuthology.orchestra.run.smithi055:> sudo tar c -f - -C /var/lib/ceph/0cb4fc0c-8359-11ee-8dbe-212e2dc638e7/crash -- . 2023-11-15T01:58:44.285 DEBUG:teuthology.misc:Transferring archived files from smithi137:/var/lib/ceph/0cb4fc0c-8359-11ee-8dbe-212e2dc638e7/crash to /home/teuthworker/archive/yuriw-2023-11-14_20:31:57-fs-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458065/remote/ubuntu@smithi137.front.sepia.ceph.com/crash 2023-11-15T01:58:44.286 DEBUG:teuthology.orchestra.run.smithi137:> sudo tar c -f - -C /var/lib/ceph/0cb4fc0c-8359-11ee-8dbe-212e2dc638e7/crash -- . 2023-11-15T01:58:44.325 INFO:teuthology.orchestra.run.smithi137.stderr:tar: /var/lib/ceph/0cb4fc0c-8359-11ee-8dbe-212e2dc638e7/crash: Cannot open: No such file or directory 2023-11-15T01:58:44.325 INFO:teuthology.orchestra.run.smithi137.stderr:tar: Error is not recoverable: exiting now 2023-11-15T01:58:44.326 INFO:tasks.cephadm:Checking cluster log for badness... 2023-11-15T01:58:44.327 DEBUG:teuthology.orchestra.run.smithi055:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/0cb4fc0c-8359-11ee-8dbe-212e2dc638e7/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2023-11-15T01:58:44.366 INFO:teuthology.orchestra.run.smithi055.stderr:grep: /var/log/ceph/0cb4fc0c-8359-11ee-8dbe-212e2dc638e7/ceph.log: No such file or directory 2023-11-15T01:58:44.368 INFO:tasks.cephadm:Compressing logs... 2023-11-15T01:58:44.369 DEBUG:teuthology.orchestra.run.smithi055:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-11-15T01:58:44.411 DEBUG:teuthology.orchestra.run.smithi137:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-11-15T01:58:44.439 INFO:teuthology.orchestra.run.smithi137.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-11-15T01:58:44.440 INFO:teuthology.orchestra.run.smithi137.stderr:: No such file or directory 2023-11-15T01:58:44.441 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/cephadm.log: 96.2% -- replaced with /var/log/ceph/cephadm.log.gz 2023-11-15T01:58:44.443 INFO:teuthology.orchestra.run.smithi137.stderr: 2023-11-15T01:58:44.443 INFO:teuthology.orchestra.run.smithi137.stderr:real 0m0.014s 2023-11-15T01:58:44.443 INFO:teuthology.orchestra.run.smithi137.stderr:user 0m0.013s 2023-11-15T01:58:44.443 INFO:teuthology.orchestra.run.smithi137.stderr:sys 0m0.009s 2023-11-15T01:58:44.445 INFO:teuthology.orchestra.run.smithi055.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/0cb4fc0c-8359-11ee-8dbe-212e2dc638e7/ceph-volume.log 2023-11-15T01:58:44.445 INFO:teuthology.orchestra.run.smithi055.stderr:: No such file or directory 2023-11-15T01:58:44.445 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/0cb4fc0c-8359-11ee-8dbe-212e2dc638e7/ceph-mgr.smithi055.szwngw.log 2023-11-15T01:58:44.445 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/0cb4fc0c-8359-11ee-8dbe-212e2dc638e7/ceph-mon.smithi055.log 2023-11-15T01:58:44.445 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/0cb4fc0c-8359-11ee-8dbe-212e2dc638e7/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-11-15T01:58:44.446 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/0cb4fc0c-8359-11ee-8dbe-212e2dc638e7/ceph-mgr.smithi055.szwngw.log: 88.4% -- replaced with /var/log/ceph/0cb4fc0c-8359-11ee-8dbe-212e2dc638e7/ceph-volume.log.gz 2023-11-15T01:58:44.450 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/0cb4fc0c-8359-11ee-8dbe-212e2dc638e7/ceph-mon.smithi055.log: /var/log/ceph/cephadm.log: 91.7% -- replaced with /var/log/ceph/cephadm.log.gz 2023-11-15T01:58:44.526 INFO:teuthology.orchestra.run.smithi055.stderr:gzip: /var/log/ceph/0cb4fc0c-8359-11ee-8dbe-212e2dc638e7/ceph-mgr.smithi055.szwngw.log: file size changed while zipping 2023-11-15T01:58:44.526 INFO:teuthology.orchestra.run.smithi055.stderr: 92.6% -- replaced with /var/log/ceph/0cb4fc0c-8359-11ee-8dbe-212e2dc638e7/ceph-mgr.smithi055.szwngw.log.gz 2023-11-15T01:58:44.597 INFO:teuthology.orchestra.run.smithi055.stderr: 90.6% -- replaced with /var/log/ceph/0cb4fc0c-8359-11ee-8dbe-212e2dc638e7/ceph-mon.smithi055.log.gz 2023-11-15T01:58:44.599 INFO:teuthology.orchestra.run.smithi055.stderr: 2023-11-15T01:58:44.599 INFO:teuthology.orchestra.run.smithi055.stderr:real 0m0.169s 2023-11-15T01:58:44.599 INFO:teuthology.orchestra.run.smithi055.stderr:user 0m0.251s 2023-11-15T01:58:44.599 INFO:teuthology.orchestra.run.smithi055.stderr:sys 0m0.018s 2023-11-15T01:58:44.600 INFO:tasks.cephadm:Archiving logs... 2023-11-15T01:58:44.600 DEBUG:teuthology.misc:Transferring archived files from smithi055:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-11-14_20:31:57-fs-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458065/remote/ubuntu@smithi055.front.sepia.ceph.com/log 2023-11-15T01:58:44.602 DEBUG:teuthology.orchestra.run.smithi055:> sudo tar c -f - -C /var/log/ceph -- . 2023-11-15T01:58:44.718 DEBUG:teuthology.misc:Transferring archived files from smithi137:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-11-14_20:31:57-fs-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458065/remote/ubuntu@smithi137.front.sepia.ceph.com/log 2023-11-15T01:58:44.719 DEBUG:teuthology.orchestra.run.smithi137:> sudo tar c -f - -C /var/log/ceph -- . 2023-11-15T01:58:44.750 INFO:tasks.cephadm:Removing cluster... 2023-11-15T01:58:44.751 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 --force 2023-11-15T02:00:15.299 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 0cb4fc0c-8359-11ee-8dbe-212e2dc638e7 --force 2023-11-15T02:00:15.608 INFO:tasks.cephadm:Removing cephadm ... 2023-11-15T02:00:15.609 DEBUG:teuthology.orchestra.run.smithi055:> rm -rf /home/ubuntu/cephtest/cephadm 2023-11-15T02:00:15.635 DEBUG:teuthology.orchestra.run.smithi137:> rm -rf /home/ubuntu/cephtest/cephadm 2023-11-15T02:00:15.656 INFO:tasks.cephadm:Teardown complete 2023-11-15T02:00:15.658 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 96, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi137 with status 5: 'sudo systemctl stop ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7@mon.smithi137' 2023-11-15T02:00:15.761 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=fbc41a2c262e45ffaaffa0838dde2f16 Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 96, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi137 with status 5: 'sudo systemctl stop ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7@mon.smithi137' 2023-11-15T02:00:15.767 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2023-11-15T02:00:15.800 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-11-15T02:00:15.840 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-11-15T02:00:15.841 DEBUG:teuthology.orchestra.run.smithi055:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-11-15T02:00:15.867 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-11-15T02:00:15.950 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-11-15T02:00:15.962 INFO:teuthology.task.clock:Checking final clock skew... 2023-11-15T02:00:15.963 DEBUG:teuthology.orchestra.run.smithi055:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-15T02:00:15.966 DEBUG:teuthology.orchestra.run.smithi137:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-15T02:00:15.983 INFO:teuthology.orchestra.run.smithi137.stderr:bash: ntpq: command not found 2023-11-15T02:00:15.985 INFO:teuthology.orchestra.run.smithi055.stderr:bash: ntpq: command not found 2023-11-15T02:00:15.986 INFO:teuthology.orchestra.run.smithi137.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-15T02:00:15.987 INFO:teuthology.orchestra.run.smithi137.stdout:=============================================================================== 2023-11-15T02:00:15.987 INFO:teuthology.orchestra.run.smithi137.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 47 -14us[ +12us] +/- 92ms 2023-11-15T02:00:15.987 INFO:teuthology.orchestra.run.smithi137.stdout:^* hv02.front.sepia.ceph.com 3 6 377 47 -202us[ -175us] +/- 59ms 2023-11-15T02:00:15.987 INFO:teuthology.orchestra.run.smithi137.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 47 +233us[ +233us] +/- 65ms 2023-11-15T02:00:15.988 INFO:teuthology.orchestra.run.smithi137.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-11-15T02:00:15.990 INFO:teuthology.orchestra.run.smithi055.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-15T02:00:15.990 INFO:teuthology.orchestra.run.smithi055.stdout:=============================================================================== 2023-11-15T02:00:15.990 INFO:teuthology.orchestra.run.smithi055.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 48 +11us[ +11us] +/- 92ms 2023-11-15T02:00:15.991 INFO:teuthology.orchestra.run.smithi055.stdout:^* hv02.front.sepia.ceph.com 3 6 377 49 -183us[ -159us] +/- 59ms 2023-11-15T02:00:15.991 INFO:teuthology.orchestra.run.smithi055.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 47 +267us[ +267us] +/- 65ms 2023-11-15T02:00:15.991 INFO:teuthology.orchestra.run.smithi055.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-11-15T02:00:15.992 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-11-15T02:00:16.027 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-11-15T02:00:16.028 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-11-15T02:00:16.077 DEBUG:teuthology.orchestra.run.smithi055:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-11-15T02:00:16.182 DEBUG:teuthology.orchestra.run.smithi137:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-11-15T02:00:16.272 DEBUG:teuthology.orchestra.run.smithi055:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-15T02:00:16.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-15T02:00:16.311 DEBUG:teuthology.orchestra.run.smithi137:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-15T02:00:16.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-15T02:00:16.343 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-11-15T02:00:16.354 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-11-15T02:00:16.385 INFO:teuthology.task.internal:Duration was 3155.036865 seconds 2023-11-15T02:00:16.386 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-11-15T02:00:16.441 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-11-15T02:00:16.442 DEBUG:teuthology.orchestra.run.smithi055:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-11-15T02:00:16.444 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-11-15T02:00:16.490 INFO:teuthology.orchestra.run.smithi137.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-15T02:00:16.499 INFO:teuthology.orchestra.run.smithi055.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-15T02:00:16.777 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-11-15T02:00:16.778 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi055.front.sepia.ceph.com 2023-11-15T02:00:16.778 DEBUG:teuthology.orchestra.run.smithi055:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-11-15T02:00:16.808 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi137.front.sepia.ceph.com 2023-11-15T02:00:16.809 DEBUG:teuthology.orchestra.run.smithi137:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-11-15T02:00:16.832 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-11-15T02:00:16.832 DEBUG:teuthology.orchestra.run.smithi055:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-15T02:00:16.851 DEBUG:teuthology.orchestra.run.smithi137:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-15T02:00:16.916 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-11-15T02:00:16.917 DEBUG:teuthology.orchestra.run.smithi055:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-15T02:00:16.923 DEBUG:teuthology.orchestra.run.smithi137:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-15T02:00:17.028 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-11-15T02:00:17.038 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-11-15T02:00:17.039 DEBUG:teuthology.orchestra.run.smithi055:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-11-15T02:00:17.070 DEBUG:teuthology.orchestra.run.smithi137:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-11-15T02:00:17.097 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-11-15T02:00:17.111 DEBUG:teuthology.orchestra.run.smithi055:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-11-15T02:00:17.114 DEBUG:teuthology.orchestra.run.smithi137:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-11-15T02:00:17.144 INFO:teuthology.orchestra.run.smithi055.stdout:kernel.core_pattern = core 2023-11-15T02:00:17.164 INFO:teuthology.orchestra.run.smithi137.stdout:kernel.core_pattern = core 2023-11-15T02:00:17.189 DEBUG:teuthology.orchestra.run.smithi055:> test -e /home/ubuntu/cephtest/archive/coredump 2023-11-15T02:00:17.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-15T02:00:17.246 DEBUG:teuthology.orchestra.run.smithi137:> test -e /home/ubuntu/cephtest/archive/coredump 2023-11-15T02:00:17.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-15T02:00:17.264 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-11-15T02:00:17.277 INFO:teuthology.task.internal:Transferring archived files... 2023-11-15T02:00:17.278 DEBUG:teuthology.misc:Transferring archived files from smithi055:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-11-14_20:31:57-fs-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458065/remote/smithi055 2023-11-15T02:00:17.280 DEBUG:teuthology.orchestra.run.smithi055:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-11-15T02:00:17.338 DEBUG:teuthology.misc:Transferring archived files from smithi137:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-11-14_20:31:57-fs-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458065/remote/smithi137 2023-11-15T02:00:17.339 DEBUG:teuthology.orchestra.run.smithi137:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-11-15T02:00:17.378 INFO:teuthology.task.internal:Removing archive directory... 2023-11-15T02:00:17.379 DEBUG:teuthology.orchestra.run.smithi055:> rm -rf -- /home/ubuntu/cephtest/archive 2023-11-15T02:00:17.382 DEBUG:teuthology.orchestra.run.smithi137:> rm -rf -- /home/ubuntu/cephtest/archive 2023-11-15T02:00:17.430 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-11-15T02:00:17.442 INFO:teuthology.task.internal:Not uploading archives. 2023-11-15T02:00:17.443 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-11-15T02:00:17.470 INFO:teuthology.task.internal:Tidying up after the test... 2023-11-15T02:00:17.471 DEBUG:teuthology.orchestra.run.smithi055:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-11-15T02:00:17.474 DEBUG:teuthology.orchestra.run.smithi137:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-11-15T02:00:17.491 INFO:teuthology.orchestra.run.smithi137.stdout: 525356 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 15 02:00 /home/ubuntu/cephtest 2023-11-15T02:00:17.494 INFO:teuthology.orchestra.run.smithi055.stdout: 393224 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 15 02:00 /home/ubuntu/cephtest 2023-11-15T02:00:17.495 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-11-15T02:00:17.523 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-11-15T02:00:17.620 INFO:teuthology.nuke:Checking targets against current locks 2023-11-15T02:00:17.648 DEBUG:teuthology.nuke:shortname: smithi055 2023-11-15T02:00:17.649 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-15T02:00:17.674 DEBUG:teuthology.nuke:shortname: smithi137 2023-11-15T02:00:17.675 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-15T02:00:17.679 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi055.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-14_20:31:57-fs-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458065', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-15 00:37:03.376529', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFCxUHNBYaiwQjldCEjzagkMP60k+HLdxkSQGUDsAFu627DaiMoEbdnt9Z2pS3+2DNyC7wQptSUKwW9Jh71uKSk='} 2023-11-15T02:00:17.701 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi137.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-14_20:31:57-fs-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458065', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-15 00:37:03.374892', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFlLiM0EFGm1UTU66Mp+CYSdELpNyOuOJ19ajyibr4CZtpACk50se3zF6/qrlzSGY3wrTgLwgoGnO6gOn23R/zE='} 2023-11-15T02:00:17.741 INFO:teuthology.orchestra.console.smithi055:Power off 2023-11-15T02:00:17.742 DEBUG:teuthology.orchestra.console.smithi055:pexpect command: ipmitool -H smithi055.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-15T02:00:17.766 DEBUG:teuthology.orchestra.console.smithi055:power off output: Chassis Power Control: Down/Off 2023-11-15T02:00:17.767 DEBUG:teuthology.orchestra.console.smithi055:pexpect command: ipmitool -H smithi055.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-15T02:00:17.784 INFO:teuthology.orchestra.console.smithi137:Power off 2023-11-15T02:00:17.785 DEBUG:teuthology.orchestra.console.smithi137:pexpect command: ipmitool -H smithi137.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-15T02:00:17.794 DEBUG:teuthology.orchestra.console.smithi055:check power output: Chassis Power is on 2023-11-15T02:00:17.809 DEBUG:teuthology.orchestra.console.smithi137:power off output: Chassis Power Control: Down/Off 2023-11-15T02:00:17.809 DEBUG:teuthology.orchestra.console.smithi137:pexpect command: ipmitool -H smithi137.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-15T02:00:17.833 DEBUG:teuthology.orchestra.console.smithi137:check power output: Chassis Power is on 2023-11-15T02:00:21.795 DEBUG:teuthology.orchestra.console.smithi055:pexpect command: ipmitool -H smithi055.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-15T02:00:21.835 DEBUG:teuthology.orchestra.console.smithi137:pexpect command: ipmitool -H smithi137.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-15T02:00:21.906 DEBUG:teuthology.orchestra.console.smithi055:check power output: Chassis Power is on 2023-11-15T02:00:21.950 DEBUG:teuthology.orchestra.console.smithi137:check power output: Chassis Power is on 2023-11-15T02:00:25.910 DEBUG:teuthology.orchestra.console.smithi055:pexpect command: ipmitool -H smithi055.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-15T02:00:25.951 DEBUG:teuthology.orchestra.console.smithi137:pexpect command: ipmitool -H smithi137.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-15T02:00:26.021 DEBUG:teuthology.orchestra.console.smithi055:check power output: Chassis Power is off 2023-11-15T02:00:26.064 DEBUG:teuthology.orchestra.console.smithi137:check power output: Chassis Power is on 2023-11-15T02:00:26.123 INFO:teuthology.orchestra.console.smithi055:Power off completed 2023-11-15T02:00:30.066 DEBUG:teuthology.orchestra.console.smithi137:pexpect command: ipmitool -H smithi137.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-15T02:00:30.176 DEBUG:teuthology.orchestra.console.smithi137:check power output: Chassis Power is off 2023-11-15T02:00:30.277 INFO:teuthology.orchestra.console.smithi137:Power off completed 2023-11-15T02:00:30.396 INFO:teuthology.run:Summary data: description: fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} duration: 3155.0368649959564 failure_reason: 'Command failed on smithi137 with status 5: ''sudo systemctl stop ceph-0cb4fc0c-8359-11ee-8dbe-212e2dc638e7@mon.smithi137''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=fbc41a2c262e45ffaaffa0838dde2f16 status: fail success: false 2023-11-15T02:00:30.396 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-15T02:00:30.493 INFO:teuthology.run:FAIL